Confluence Retirement

Due to the feedback from stakeholders and our commitment to not adversely impact USGS science activities that Confluence supports, we are extending the migration deadline to January 2023.

In an effort to consolidate USGS hosted Wikis, myUSGS’ Confluence service is targeted for retirement. The official USGS Wiki and collaboration space is now SharePoint. Please migrate existing spaces and content to the SharePoint platform and remove it from Confluence at your earliest convenience. If you need any additional information or have any concerns about this change, please contact myusgs@usgs.gov. Thank you for your prompt attention to this matter.

GIS Seminar, September 21

Raster Formats

Data source vs MXD layer vs layer file

  • In a map document, a layer is a pointer to data with instructions for ArcMap about how to display the data (the layer properties).
  • More than one layer can point to the same data. Different properties are allowed for each layer.
  • The layer properties you configure are saved in the mxd or can be exported to a separate Layer File (.lyr extension)
  • The Layer File does not contain the data, only the properties and is only usable by ArcMap.
  • The Layer File has to be able to find the data when added to another map. Use relative paths.
    • Check Map Document Properties > ‘Store relative pathnames to data sources’ checkbox
    • Or, use Save to Layer File tool
  • Not necessary, but save the .lyr file in the same folder as the data source when possible so that they stay together

Raster layers and Image functions

  • Raster layers can be used as inputs to an image function, an on-the-fly calculated raster based on the input
  • Alternative to writing a raster out to disk based on a calculation
  • Example is putting a hillshade function on top of a dem
    • Open Image Analysis window

  • Select raster as input to function and click the Add Function button

  • Right-click on the name of the raster, select Insert Function and choose the function

 

  • Fill out the parameters and close the Function Template Editor.
  • There is a new raster layer in your map document that you can treat as any other raster layer, including as input to another raster process.
  • Change function parameters by going to the Function tab in the layer properties dialog and double-clicking on the function name to open the Raster Function Properties dialog.


Raster basics

  • A raster is an ordered matrix/array/grid of values, with instructions to a program for how to display the values:

Example pseudo file, header and values

This raster is 15 columns x 5 rows

Start displaying these values in the upper left of your display:

4,4,4,4,4,7,7,7,7,7,3,3,4,4,4,4,4,4,4,4,4,7,7,7,3,4,4,4,5,5,4,4,4,4,4,7,7,7,4,4,4,5,5,5,5,4,4,4,4,7,7,7,4,4,4,4,3,3,3,4,4,4,4,7,7,7,7,7,3,3,3,3,4,4,4

Result:

4,4,4,4,4,7,7,7,7,7,3,3,4,4,4

4,4,4,4,4,4,7,7,7,3,4,4,4,5,5

4,4,4,4,4,7,7,7,4,4,4,5,5,5,5

4,4,4,4,7,7,7,4,4,4,4,3,3,3,4

4,4,4,7,7,7,7,7,3,3,3,3,4,4,4

  • Values can be elevation, wind speed, land cover category, gravity measurement, color (RGB stacked rasters) etc.
  • When displayed on a screen, the values are represented by a color within a pixel which has a width and a height.

cell width and height

  • A non-GIS image viewer will keep track of values using image coordinates

Image result for raster rows columns

  • Image coordinates vs spatial reference coordinate
  • In image space, the width and height are defined by the screen resolution
  • In coordinate space, the width and height are defined by additional georeferencing information


Georeferencing

  • ‘Sidecar’ or ‘buddy’ file: world file (*.tfw, *.jpw, *.bpw, etc). Text file you can open in a regular text editor.

50

- cell size in x-dimension

0

- rotation about y-axis

-0

- rotation about x-axis

-50

- cell size in y-dimension

666907.78621063067

- x-coordinate of the center of the upper left pixel

6795674.9494232498

- y-coordinate of the center of the upper left pixel

 

  • Embedded: eg., ASCII raster header

ncols         407

nrows         404

xllcorner     666882.78621063  (can also specify xllcenter)

yllcorner     6775499.9494232

cellsize      50

NODATA_value  -9999

-9999 -9999 -9999 -9999 -9999 -9999 97 97 90 89 95 87 53 56 55 54 57 58 55 57 56 57 58 59 56 59 55 56 58 54 55 59 60 56 58 56 56 56 62 58 52 51 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999

 

  • Note that neither the world file nor the ASCII raster headers have any information about the coordinate system. You need a .prj file, auxiliary file (.aux.xml) or knowledge of the correct SR.
  • .prj file:

Projection    UTM

Zone           5

Datum         NAD83

Spheroid      GRS80

Units         METERS

Zunits        NO

Parameters   

 

  • Auxiliary file (.aux.xml) has SpatialReference XML element:

      < SpatialReference xsi:type="typens:ProjectedCoordinateSystem">    <WKT>PROJCS["NAD_1983_Alaska_Albers",GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137.0,298.257222101]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Albers"],PARAMETER["False_Easting",0.0],PARAMETER["False_Northing",0.0],PARAMETER["Central_Meridian",-154.0],PARAMETER["Standard_Parallel_1",55.0],PARAMETER["Standard_Parallel_2",65.0],PARAMETER["Latitude_Of_Origin",50.0],UNIT["Meter",1.0],AUTHORITY["EPSG",3338]]</WKT>

 

  • Georeferencing can be embedded in the header of a few binary formats, eg, TIFF (which makes it a geotiff), JPEG 2000, ECW, MrSID.
  • Check properties of raster in ArcCatalog or use gdalinfo command to see if a raster has georeferencing associated with it:

>gdalinfo ascii_raster.asc

Driver: AAIGrid/Arc/Info ASCII Grid

Files: ascii_raster.asc

       ascii_raster.prj

Size is 407, 404

Coordinate System is:

PROJCS["UTM Zone 5, Northern Hemisphere",

    GEOGCS["NAD83",

        DATUM["North_American_Datum_1983",

            SPHEROID["GRS 1980",6378137,298.257222101,

                AUTHORITY["EPSG","7019"]],

            TOWGS84[0,0,0,0,0,0,0],

            AUTHORITY["EPSG","6269"]],

        PRIMEM["Greenwich",0,

            AUTHORITY["EPSG","8901"]],

        UNIT["degree",0.0174532925199433,

            AUTHORITY["EPSG","9122"]],

        AUTHORITY["EPSG","4269"]],

    PROJECTION["Transverse_Mercator"],

    PARAMETER["latitude_of_origin",0],

    PARAMETER["central_meridian",-153],

    PARAMETER["scale_factor",0.9996],

    PARAMETER["false_easting",500000],

    PARAMETER["false_northing",0],

    UNIT["METERS",1]]

Origin = (666882.786210629970000,6795699.949423200500000)

Pixel Size = (50.000000000000000,-50.000000000000000)

Corner Coordinates:

Upper Left  (  666882.786, 6795699.949) (149d53'17.59"W, 61d15'35.26"N)

Lower Left  (  666882.786, 6775499.949) (149d54'21.69"W, 61d 4'43.36"N)

Upper Right (  687232.786, 6795699.949) (149d30'34.38"W, 61d15' 2.06"N)

Lower Right (  687232.786, 6775499.949) (149d31'46.24"W, 61d 4'10.40"N)

Center      (  677057.786, 6785599.949) (149d42'29.97"W, 61d 9'53.25"N)

Band 1 Block=407x1 Type=Int32, ColorInterp=Undefined

  NoData Value=-9999


Georeferencing demo

  • ‘Update georeferencing’ – creates world file, .aux.xml, and internal tags, if applicable.
  • Update or rectify?
    • Transformations

Polynomial transformations

  • Affine (1 st order polynomial)  - update is fine
  • 2 nd , 3rd order or spline, eg., any rubber sheeting – rectify or always provide .aux.xml.
  • The most honest way to release a georeferenced raster is probably to release the original, un-georeferenced raster as well as the .aux.xml file which will contain the transformation methods and the control links used rather than a rectified raster. From a rectified raster, one cannot go back to the untransformed original to assess the accuracy of the georeferencing.
  • In general, avoid any transformation other than affine. Exception is when you have no reason to believe that the non-georeferenced image must maintain it’s aspect ratio.
    • For example, a scan of a field sheet that is made up of taped together sections of a paper map. Paper can shrink and warp over time and there are probably gaps in the taped seams. In this case, you need to provide as many georeferencing links as possible and you can use a rubber sheet transformation


Pixel/bit depth

  • Determines the range of possible values in a cell.
  • Resize, if possible, for your range

unsigned 1 bit = 0 to 1  
unsigned 2 bit = 0 to 4  
unsigned 4 bit = 0 to 16  
unsigned 8 bit = 0 to 255  
signed 8 bit = -128 to 127  
unsigned 16 bit = 0 to 65535  
signed 16 bit = -32768 to 32767  
unsigned 32 bit = 0 to 4294967295  
signed 32 bit = -2147483648 to 2147483647
floating point 32 bit = -3.402823466e+38 to 3.402823466e+38     The default for DEMs

unsigned 64 bit = 0 to 18446744073709551616

 

  • Nominal size of raster is rows * columns * bands * bytes. An 8-bit RGB image measuring 5000 x 5000 is

5,000 (height) x 5,000 (width) x 3 (bands) x 1 (byte) = 75,000,000 bytes = 75 MB

  • Size of a floating point 32 bit DEM – 30 MB
  • The same DEM reduced to 16 unsigned – 14.5 MB


Compression

  • Lossy vs lossless

Image result for lossy compression raster

 

Image result for lossy compression example

  • Lossy is fine for background images but usually not suitable for data
  • Lossless compression does not usually result in much compression unless there are large areas of NoData or the same value.

Compression

Lossless/lossy

Notes

LZ77

lossless

Used in PNG

PackBits

lossless

Apple format

LZW

lossless

Used in GIF

JPEG

lossy

Choose amount of compression

JPEG 2000

either

 

 

  • TIFF files can use any of these methods or none.
    • An uncompressed tiff – 190 MB
    • The same tiff compressed with LZW – 167 MB
    • The same tiff compressed with JPEG – 8.5 MB
    • From ArcMap raster layer, choose Data > Export Data from right-click menu
    • When TIFF is selected as output Format, Compression Type dropdown has all available options.
  • Compressed files take up less space on disk, but are uncompressed in RAM when used. The greater the compression, the longer the decoding takes.
  • In general, for background imagery – TIFF with JPEG compression is good compromise between file size, speed, and accuracy


Resolution

  • 1987, Waldo Tobler, UCSB cartographer:
    • “.. divide the denominator of the map scale by 1,000 t o get the detectable size in meters. The resolution is one half of this amount.”

Resolutions Table


Pyramids

  • Downsampled versions of the original raster that speed up panning and zooming.
  • At ArcGIS 10, pyramids are stored in .ovr file adjacent to the raster
  • Can also be internal in JPEG 2000, ECW, MrSID, and TIFF
  • GDAL overview = ESRI pyramid

Example of pyramid compression


Formats

GDAL – Geospatial Data Abstraction Library  http://www.gdal.org/

  • Open source library for raster and vector data formats
  • Drivers (read/write access) for 142 raster formats (some have 3 rd party dependencies)
    • ESRI – 70 formats
    • Global Mapper – 65 formats
    • QGIS – all GDAL supported formats
    • In practice, we are mostly talking about just a handful; TIFF, JPEG, PNG, ESRI GDB, GRID,IMG, ASCII, NetCDF, NITF
  • Raster utilities
    • Available as Windows DOS, linux, unix console/terminal commands
    • Python bindings (https://pypi.python.org/pypi/GDAL)
    • R bindings (https://cran.r-project.org/web/packages/rgdal/index.html)
    • GDAL utilities get installed with QGIS, GRASS, OSGeo (which includes a nice console shell so you don’t have to worry about updating your PATH variable), FWTools, a few other open-source GIS packages
    • QGIS provides GUIs for most of the GDAL utilities.


A few common GIS Raster formats

Format

Ownership

Notes

TIFF

Adobe

Ubiquitous; flexible; internal georeferencing, tiles, and pyramids

JPEG/JPEG 2000

Joint Photographic Experts Group

JPG – 3 band only, no internal georeferencing.

JP2 – single  or multiband, internal georeferencing

PNG

Open, World Wide Web Consortium (W3C)

Internal channel for transparency, very common internet interchange format, used for tiles in web-served rasters.

FGDBR – ESRI file gdb raster

ESRI

Useful if there is a need to store raster data in the same container as associated vector data.

ESRI binary GRID

ESRI

Similar to ArcInfo coverage, a set of files grouped in two directories; grid name and info

ASCII

Created by ESRI but functionally open

Completely open, text-based BUT no compression leading to big files.

NetCDF

Open, created by Univ. Corp for Atmospheric Research (UCAR)

Multi-variable and multi-dimensional (eg., time, height) environmental data.

A couple others

 

 

GeoPackage raster

Open Geospatial Consortium

SQL-Lite based database raster stored in png or jpg tiles.

GDAL Virtual Raster (VRT)

Open

XML-based specification for creating raster catalogs, eg., multiple, tiled rasters can be viewed and analyzed as a single raster without loading all of the tiles at once.


GDAL Examples

  • Internal tiles and pyramids in geotiff
    • SPOTchip_0.tif - 2.6 GB
    • Step 1 – build internal tiles and compress with JPEG compression

gdal_translate -co TILED=YES -co COMPRESS=JPEG SPOTchip_0.tif SPOTchip_1.tif

 

  • Step 2 – build internal pyramids. Create 8 levels to open in ArcGIS

gdaladdo -r average --config COMPRESS_OVERVIEW JPEG \

--config PHOTOMETRIC_OVERVIEW YCBCR \

--config INTERLEAVE_OVERVIEW PIXEL \

SPOTchip_2.tif 2 4 8 16 32 64 128 256

  • SPOTchip_2.tif – 462 MB

 

  • Virtual Raster (VRT)
    • Build one virtual raster from a collection of tiles, eg., a directory of rasters or a list of paths to rasters – RGB_items_1perline.txt
    • Build VRT from list

gdalbuildvrt -input_file RGB_items_1perline.txt SPOT5.SDMI.AK.ORTHO.RGB.vrt

 

  • Build overviews
  • SPOT5.SDMI.AK.ORTHO.RGB.vrt
    • 4,475 rasters
    • 2.08 TB of data
    • 8.3 GB of overviews
    • Opens in ArcGIS and QGIS