GDP Spatial Data Infrastructure Documentation
Disclaimer: This documentation is a work in progress, treat it as such and please send me feedback on it if you need additions or think changes are in order. firstname.lastname@example.org
User interface utility services that respond synchronously are at http://cida.usgs.gov/climate/gdp/utility/
Processing services that respond asynchronously are at http://cida.usgs.gov/climate/gdp/process/
The UI focused on downscaled climate data is available at http://cida.usgs.gov/climate/gdp/
The USGS Geo Data Portal (GDP) project provides scientists and environmental resource managers access to downscaled climate projections and other data resources that are otherwise difficult to access and manipulate. An example user interface has been developed to demonstrate an example implementation of the GDP project web-service software and standards-based spatial data infrastructure (SDI). A user of this example GDP interface can supply their area of interest as a pre-existing GIS shapefile with one to many unique polygons or by drawing a single polygon using an interactive web-map. A user can select from available GDP project web-service processing algorithms, which include raw data subsetting and area-weighted statistics summarization. Processing algorithm options such as dataset of interest, time period of interest and output file formatting must be specified. In order to allow processing of very large datasets, GDP project web-services perform asynchronously. This allows a user to supply an email address and be notified upon process completion.
The GDP spatial data infrastructure consists of centralized and decentralized web-service components, some are flexible and customizable, others strictly defined and rigid. In the broadest terms, the infrastructure will facilitate and inform rapid deployment of thin web-clients to work with a wide variety of scientific data for which adequate tools are either unavailable or prohibitively complex for non-IT specialists to utilize. The framework is designed to utilize open source and commonly available proprietary server technologies with minimal setup cost and training requirements.
Spatial Area and Temporal Period of Interest
Users spatial analysis zones are specified using the Open Geospatial Consortium (OGC) Web Feature Services (WFS) which serve the OGC simple features profile of the Geography Markup Language (GML). Temporal specification employs ISO standard date-time formatting. ArcGIS server, GeoServer, and MapServer are existing WFS servers compatible with the GDP core processes. GDP clients will generate a WFS "GetFeature" request which the GDP core processes can execute to access the users analysis zones.
Spatial Area of Interest
The GDP processing services rely on the Web Feature Service (WFS) standard to specify the users spatial area/s of interest.
- ArcGIS server, GeoServer, and MapServer all offer web feature service functionality.
- Clients may implement an upload service or geometry creation service so an end user without access to server resources may specify their own geometry.
- Client applications can use the WFS standard to get dataset-metadata needed to generate a complete WFS getFeatures request.
- A fully formed getFeatures request is required and will be treated as a remotely accessible data source by the GDP core processing engine.
Details of the CIDA reference implementation of File upload and metadata gathering functionality are explained in the following pages.
Adding a Shapefile as a GeoServer WFS EndPoint
Getting a List of Available FeatureTypes (Analysis Zone Sets)
Getting a List of Available Attributes Given a FeatureType
Getting a List of Attribute Values Given FeatureType and Attribute
Temporal Period of Interest
The GDP core processing engine will accept any ISO 8601 date/time analysis starting and ending time. See the core processing documentation section for a description of exceptions and behavior.
Dataset to be analyzed
Server resources accessible by the GDP include the Opensource Program for a Network Data Access Protocol (OPeNDAP) and the OGC Web Coverage Services (WCS). As development continues, the generalized feature and time series standards WFS, Sensor Observation Service (SOS) , and possibly others will be used to handle time series associated with arbitrary mapped features. GDP clients provide data service urls with parameters to completely specify the component of the dataset to be accessed. The full parameter set will vary depending on data access standard.
In order to access internal dataset metadata, some data sources require software that is not compatible with rapidly deployable thin client implementations. To satisfy this requirement of the GDP DIF, access to metadata like available grids, data types, time ranges, and spatial domains are facilitated by WPS services to keep client implementations as simple as possible.
OPeNDAP data source metadata access
GDP Process WPS Documentation
Process output specification and GDP core services interaction uses the OGC Web Processing Service standard.
GDP WPS Services Using GET
GDP WPS services can be encoded using HTTP GET. There are the normal limitations in terms of the length of the URL encoded request. HTTP POST is the recommended method of accessing the services, especially for asynchronous processing.
THREDDS Best Practices
THREDDS Data Server
Unidata laid out some best practices that should be followed when setting up and using a THREDDS server. This page was derived from lessons learned at Unidata's thredds training in late 2010 and the GDP team's THREDDS Data Server experience.
Dataset Prearation Guidelines
Dataset Preparation Guidelines
These guidelines summarize many considerations for publishing large gridded time series data for the Geo Data Portal.