Confluence Retirement

Due to the feedback from stakeholders and our commitment to not adversely impact USGS science activities that Confluence supports, we are extending the migration deadline to January 2023.

In an effort to consolidate USGS hosted Wikis, myUSGS’ Confluence service is targeted for retirement. The official USGS Wiki and collaboration space is now SharePoint. Please migrate existing spaces and content to the SharePoint platform and remove it from Confluence at your earliest convenience. If you need any additional information or have any concerns about this change, please contact myusgs@usgs.gov. Thank you for your prompt attention to this matter.
Skip to end of metadata
Go to start of metadata

Appendix A FY10 Goal 1 – ArcGIS/ArcMap Access to Corporate DatabasesIntroduction and Problem

USGS in cooperation with USEPA has developed a water-quality data exchange that organizes and serves geo-referenced monitoring location information for 125 thousand stream sites and 1.3 million groundwater wells and over 4.5 million water-quality samples collected over the last 130 years. These samples yield over 100 million results for over 1100 water-quality constituents. In FY2010 resources provided by the USGS Office of Water Quality, the National Water Quality Monitoring Network and USEPA will support development of a shared federal water-quality data catalog and registry as well as a unified web portal that will provide analysts with a single search and download interface for water-quality data. The FY 2010 Data Integration Development Project intends to leverage this activity by providing additional geospatial search and data access mechanisms to this USGS corporate data asset. MRDATA--USGS’ corporate repository for geology, geochemistry, geophysics, and mineral resource data--already has in place many GIS data services that will be combined into this effort. In addition to these services, the GIS Community of Practice will supervise development of ArcGIS desktop plugins. These plugins, coupled with the services provided, will allow USGS GIS users to evaluate acceptable options for data delivery and visualization.

In addition, there is a need to begin establishing best practices around the development of web services for data access and delivery. Within Goal 1, we will identify and bring together OGC compliant WMS implementations within a viewer application environment. We will demonstrate that data provisioned through WMS implementations for The National Map, Water data, Biology data, and Geology data can be discovered and integrated to enable a foundation for Bureau science. 

Relevance and Benefits

Well over a third of all USGS employees have access to or use enterprise GIS software. These scientists commonly and frequently overlay and integrate USGS collected or managed datasets to conduct their investigations within this GIS. Yet many corporate data systems within USGS do not provide ready and efficient access within a GIS environment. Goal 1 intends to deliver GIS access for two major USGS corporate information systems by implementing two data delivery services. The first service is founded on Open Geospatial Consortium standards that also will allow integration with both the National Map and a federal water-quality data portal. Coupled with the service tier, Goal 1 will also include the development of ArcGIS client tools (plugins) to increase the usability of the services. Cost savings and systems maintenance efficiency are gained by developing a common and singular data service that meets multiple objectives.

Objectives
  1. Provide a suite of open standards geospatial services1 on a summarization of National Water Information System (NWIS) monitoring sites being prepared under the USGS Water-Quality Web Services Project (http://cida.usgs.gov/qwwebservices.html) to facilitate access by ArcGIS and other tools capable of accessing these formats.
  2. Research and document scalability of the ArcGIS/ArcMAP toolset in accessing and making full use of queries against WFS or other open standard methods to determine feasibility in allowing GIS users direct access through these methods.
  3. Document best available methods for querying corporate data services with geographic area, data type, and content criteria (e.g., specific water quality measurements at stream gage sites within an area of interest collected during a specified time period) and provide training and consulting in these methods through appropriate venues.
  4. Incorporate new National Map base data layer services in existing mapping applications (various water quality applications, ScienceBase Catalog search and visualization tool, BASISMap service of project footprints) and document case studies as a demonstration of using these corporate data assets.
  5. Provide appropriately formatted metadata via Geospatial One-Stop or appropriate catalog mechanism to enable discovery and access of NWIS and MRDATA OGC spatial services through the new National Map viewer application (Palanterra). Document case study of this exposure of USGS corporate data and present in appropriate venues.
User Stories

User Story #1: ArcMap user requests via ArcMap Catalog (1) a layer from MRDATA and (2) a layer from NWIS. User searches for water-quality data based on both site characteristics (within a geographic area) and data inventory (constituents measured and when). After sites are displayed within the GIS, data is downloaded into the ArcMap environment. 
User Story #2: National Map user can request within the new National Map Viewer water-quality monitoring locations, geology data from MRDATA, and biologic data from NBII’s GAP analysis project.
User Story #3: Within the water-quality data delivery portal, user queries for water-quality data from both USGS and USEPA and displays monitoring locations over the eight base layers from the National Map for both geographic reference and context.

Approach

This goal is structured into five consecutive components and will be led by staff at CIDA in cooperation with Geography and the GIO.
Corporate databases served via OGC standards
Within this component, OGC specifications relevant to cataloging, mapping and serving water-quality and minerals data will be investigated. Specifically, Web Mapping Service (WMS) and Web Feature Service (WFS) will be developed against the water-quality database. These services already exist against MRDATA and The National Map to some extent. Documentation and instructions for all services from these corporate databases will be exposed through data.usgs.gov and the Data Services Wiki (an online open form at data.usgs.gov for documenting Web services capabilities in USGS) among other outlets.
Research best available methods of accessing corporate data via ArcGIS tools
Initial indications through conversation with many ArcMap power users suggests that the current approach with the ESRI toolset of downloading and caching data through a WFS connection introduces a number of scalability and perception issues. Small databases with a low number of features will return query results relatively quickly. Larger databases with many thousands of records and features will at least appear to operate very slowly due to the process of downloading the entire data set to run queries against locally.
This dynamic introduces a scalability problem in that it forces processing and storage capacity to be handled by all individual client systems using the connection that may prove a problem for large corporate datasets like NWIS. To address these issues, the following approach will be pursued:
ArcGIS Client Tool Development
This component will focus on making the developed services easier to access and consume by ArcGIS Client (ArcMap) users. The tools will be distributed as plugins for USGS desktops, and will be evaluated for usefulness as a part of the overall task.
Approach Assessment Documentation
This component will be an assessment of coupling web services with client tools. The assessment will be written in consultation with GIS experts and testers. Critical to any future efforts is an approach assessment by the USGS GIS Community. While the conceptual workflow (Web Services coupled with client tools) is sound, it will only be the GIS Community that can make the decision whether the Goal objectives have been met.
Corporate databases served via ESRI Spatial Data Engine
This component is reserved pending research and testing of OGC standards within the ArcGIS environment. Additional resources are required for this task should it become necessary. ESRI Spatial Data Engine (SDE) is a client-server architecture that centralizes geoprocessing on a middle-tier server (placed between the RDBMS and the GIS client). If this component is required, a metadata schema will be developed against the water-quality and MRDATA databases, and the SDE server will be configured and tested.

Big Picture Implication

The introduction and maintenance of a specialized technology platform to support optimized access by ArcGIS clients will involve significant, long-term resource commitment at various levels of the organization – data owners and the enterprise at large. This project aims to push the limits on what the toolset can do with open standards data, provide recommendations if necessary for communication with ESRI on technology needs, and document the implications of an ESRI-specific technology implementation if necessary.

National Map Integration

This component will focus on two aspects of National Map integration:

  • Use National Map Web services (WMS, tile cache services) in the water-quality data portal and ScienceBase Catalog search applications as a demonstration case study documented in the Data Services Wiki
  • Registering metadata about new WMS and WFS services created for water-quality data and existing services from other corporate databases (e.g., MRDATA) with Geospatial One-Stop (GOS) with the appropriate attribution to enable their discovery and access through the National Map Viewer application

The National Map services have been available in a testing mode for some time with several experiments already conducted into their use. Incorporation into existing viewer applications will involve a relatively minor set of tasks. Documentation as case studies will serve to educate many other application developers on the utility of these base data layers.
The National Map viewer uses a particular construct within FGDC metadata to address a long-standing problem with a given metadata record being “disconnected” from the data it documents. Metadata records for NWIS water-quality Web services and MRDATA resources will either be created or updated to reflect this component, registered in the GOS catalog, and tested for discovery and access through the National Map Viewer. The details of this process will also be documented in the Data Services Wiki, within GOS documentation, and elsewhere as appropriate to help other data providers in this process.

1 Open Geospatial Consortium (OGC) Web Map Services (WMS) and Web Feature Services (WFS)

  • No labels