Confluence Retirement

Due to the feedback from stakeholders and our commitment to not adversely impact USGS science activities that Confluence supports, we are extending the migration deadline to January 2023.

In an effort to consolidate USGS hosted Wikis, myUSGS’ Confluence service is targeted for retirement. The official USGS Wiki and collaboration space is now SharePoint. Please migrate existing spaces and content to the SharePoint platform and remove it from Confluence at your earliest convenience. If you need any additional information or have any concerns about this change, please contact myusgs@usgs.gov. Thank you for your prompt attention to this matter.

GIS Seminar, July 20

GIS Vector Formats

ArcMap Layer File vs data source

  • In a map document, a layer is a pointer to data with instructions for ArcMap about how to display the data (the layer properties).
  • More than one layer can point to the same data. Different properties are allowed for each layer.
  • The layer properties you edit are saved in the mxd or can be exported to a separate Layer File (.lyr extension)
  • The Layer File does not contain the data, only the properties.
  • The Layer File has to be able to find the data when added to another map. Relative vs absolute paths.
    • Check Map Document Properties > Store relative pathnames to data sources checkbox
    • Or, use Save to Layer File tool


ESRI formats – old to new

ArcINFO coverage

  • Older ESRI binary format. Development stopped around 1999.
  • Well-described through reverse engineering but still proprietary.
  • Stored topology – spatial relationships between features, notably lines coincident with polygon boundaries. Normalized coordinate storage.

  Coverage feature classes

  • Cannot be edited in ArcMap versions beyond 8.3. Cannot be viewed in ArcGIS Pro.
  • Shared via the e00 interchange format which is ASCII (though if compressed it appears to be binary)
  • QGIS  and Global Mapper can read e00 files directly
  • Use ArcGIS tool ‘Import from e00’ to convert to coverage and then convert to another ESRI format

Shapefile

  • Fully open format described by ESRI. Lots of options for reading and writing.
  • Main limitations:
    • Lacks Unicode support which limits the use of non-English characters
    • Text fields limited to 255 characters
    • Field names limited to 10 characters; after import you might see “LONGFIELD1”, “LONGFIELD2”, etc.
    • Numeric values (not coordinates) stored as characters, not binary – rounding problems
    • Date field must be date or time, not both
    • Cannot store NULL values; instead, you might see -1.7976931348623158e+308 , 0, or “ ”
    • No XY tolerance (minimum distance before coordinates are considered equal) which affects spatial comparison operations and editing.
    • No explicit topology.
  • Use shapefiles for simple features (no topology) and attributes.
  • To make them more usable, save a .lyr file.
  • However, the shapefile IS a Library of Congress preferred format for storing spatial data.


ESRI Geodatabase (GDB)

  • Spatial relational database in three versions:
    • Personal – built on top of a Microsoft Access database
    • File – an ESRI proprietary system of files
    • SDE (spatial data engine - built on top of an enterprise RDBMS ( Oracle, Microsoft SQL Server, PostgreSQL, etc.)
  • Can store various forms of:
    • Data: complicated geometries, non-spatial tables, rasters, mosaic datasets, schematic datasets, terrain datasets, toolboxes, etc
    • And data controls: feature datasets, relationship classes, feature-linked annotation, subtypes, domains, topology rules, merge rules, etc.
  • File GDB is the recommended choice for single-editor desktop use. Support for personal GDB’s will end as ESRI moves to 64 bit software (ArcGIS Pro cannot read personal GDBs).
  • Personal GDB’s are limited to 2GB in size whereas file GDB’s can be terabytes in size.
  • SDE is for large multi-editor databases where versioning is important.


Non-ESRI formats

Excel

  • Excel spreadsheets can be used to store XY locations and associated attributes.
  • Check formatting of columns. Mixed values - numbers and text - are imported as text.
  • Make sure you know the coordinate system of the XY values.
  • You won’t be able to edit the table inside of ArcMap. It will have to be converted to an ESRI format.
    1. Create XY event layer in ArcMap and Export to ESRI format  OR
    2. In ArcCatalog, right-click table > Create Feature Class from XY table.

CSV

  • Mostly the same concerns as Excel.

KML

  • Keyhole Markup Language. XML-based. OGC standard since 2008.
  • Stores multiple geometries, attributes, symbology, rasters, and pointers to streaming services.
  • Opens in Google Earth, Global Mapper, QGIS. Editable?
  • In ArcGIS, use a tool to convert to another format.

WKT – Well Known Text

  • Open Geospatial Consortium standard for text-based description of geometries.
  • Writable from ArcGIS, not readable without coding.
  • Easy to write ESRI SHAPE field geometries as WKT > export to csv.
    • Calculate field expression !shape.wkt!
  • QGIS opens comma delimited files with WKT strings.

JSON/GeoJSON/TopoJSON

  • JSON is Javascript Object Notation
  • Text-based open format
  • GeoJSON is an extension of JSON for encoding spatial data
    • Editable in QGIS
    • All coordinates in decimal degrees WGS84
  • TopoJSON is an extension of GeoJSON for encoding shared topology in a way similar to the e00 format
    • Writing to TopoJSON is not part of ArcGIS or QGIS yet. QGIS can open TopoJSON feature classes.
    • MapShaper website

GeoPackage

  • Open binary standard for data interchange base on a SQLite database. Seems to be gaining traction.
  • Can create in ArcGIS and then load/copy data. Table operations such as APPEND, INSERT, DELETE, etc. but you can’t edit geometries or attributes in an edit session
  • Fully editable in QGIS (AFAIK)