Skip to end of metadata
Go to start of metadata

Data Services Toolkit (DS-Toolkit) a.k.a. Tech Stack Working Group

The wiki page listed above should be consulted for the most recent developments of this group. The following text reflects thoughts at the time of the workshop, which continue to evolve. In general, this element builds on the FY2010 work for the Data Uploader and the concepts discussed heavily in the workshop about a group of data hosting and serving technologies operating in different parts of the USGS. The hope is that all these developments can be coordinated to form a "Scientific Data Network." At the very least, this element seeks to develop a community of practice around the technology and design of such a network.

Scientists may opt to use a centralized incarnation of the Data Uploader or download and deploy a local version of this platform (this version is sometimes referred to as a "droppable appliance") within their project. In addition, the design of this platform and the community discussions around it will help to provide a template and guidance for projects that need to develop their own information management system ("appliance") and will hopefully result in a project-specialized system that can still participate in the greater Scientific Data Network.

An emphasis that has grown since the workshop that is worth mentioning here is that this community seeks to develop understanding and recommendations on how such appliances can be engineered to effectively exploit the many corporate/enterprise data services, such as those being pushed forward by the National Map and the National Water Information System. While the Data Uploader has consciously attempted to build around open-source software (in addition to open standards), many of these corporate services do leverage important proprietary components of the Agency's corporate computer model, such as ESRI ArcGIS Server.

Possible Leadership Team Identified at the Workshop - Blodgett, Viger, Dadisman, Kern, Gunther, Skinner, Hope/Tricomi, Greenlee

Major Elements

  • Develop the Tech Stack Working Group Technology Stack Working Group (TSWG)
  • Engage the developers of major corporate components, including the Data Uploader, TNM services, and NWIS services with this group.
  • DS-Toolkit engineered, documented, and established for nodes targeted on current hosting needs from the Science Strategy mission areas and specific priorities. These will include, at a minimum, a node for the Coastal and Marine Geology Program (focused toward Coastal and Marine Spatial Planning), nodes for the Climate Effects Network (Upper Colorado River Basin, Greater Platte River Basin, Yukon River Basin), and node(s) for the National Climate Change and Wildlife Science Center.
  • Coordinated Catalog Services (probably based on the Open Geospatial Consortium Catalog Service for Web (OGC-CSW) standard) and a central harvest of all nodes in the DS-Toolkit network to support broad search and discovery.
  • Application of the DS-Toolkit to one or more projects/data hosting needs identified for the nodes and targeted to a major science question. Data services from the DS-Toolkit should feed directly into usage from the DI-Toolkit. This application of the technology stack will be released in a USGS series report or submitted to an outside journal to both document the approach and demonstrate its applicability.

Continuation of FY2010 Work

  • Work will build on the approach piloted by the Data Uploader group to create a "droppable virtual machine (VM)" with a suite of supported technologies that will advertise/register hosted data resources with a shared metadata cataloging service.
  • The workshop teams responsible for this element discussed an approach that would embrace both open-source methods and technologies along with the elements of the ESRI technology stack that serve open standards. This builds on discoveries made about the relative efficacy of these two approaches.

Original Workshop Bullets

  1. Develop a Technology Stack Working Group
    1. Unified Access Framework for Gridded and Vector Data:  WMS, WCS, WFS, and OPeNDAP+CF - Rich Signell , Blodgett, Viger
    2. Discoverability and Interoperability of Web-based Data and Processing Services - Blodgett, Viger, Dadisman, Kern, Gunther, Skinner,
    3. Catalog Services Plan, including architecture, as well as search and harvesting functions. With metadata standards - Steve Richard
    4. Web Application Integration Framework (Summary Notes Presented) - Matt Tricomi, Greg Smoczyk, Hollister
    5. Moved from DI Toolkit to Tech Stack Create USGS Corporate Services Integrated Roadmap for serving TNM, GloVIS, MRDATA, NWIS (streamflow and water quality), watershed boundaries at National scope with Statistical Capability (TNM, WBD, NWIS, MRDATA, GloVIS)
    6. Data/Application Publishing - Web Services Publishing Best Practices (TNM, WSWG) (Moving from DM to Tech Stack since about publishing tech services and to allow NGP to participate in one group - Discussed at 12/7 TS meeting)