Confluence Retirement

Due to the feedback from stakeholders and our commitment to not adversely impact USGS science activities that Confluence supports, we are extending the migration deadline to January 2023.

In an effort to consolidate USGS hosted Wikis, myUSGS’ Confluence service is targeted for retirement. The official USGS Wiki and collaboration space is now SharePoint. Please migrate existing spaces and content to the SharePoint platform and remove it from Confluence at your earliest convenience. If you need any additional information or have any concerns about this change, please contact myusgs@usgs.gov. Thank you for your prompt attention to this matter.

USGS

PROPOSAL

NWIS Web Services Snapshot for ArcGIS

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

United States Geological Survey

Austin , Texas  

September 30, 2011

 


NWIS Web Services Snapshot for ArcGIS

Overview

Retrieving a “snapshot” of National Water Information System (NWIS) data into an ArcGIS geodatabase was conceived in 2002 to reduce the steps required to retrieve and analyze NWIS data. The snapshot process was also designed to serve as a learning tool for new or infrequent NWIS users as well as provide direct documentation of the numerous codes commonly found in NWIS data output. More than 22 water science centers have actively utilized the snapshot process for projects, NWIS database administration, basic data analysis, and map production.  In addition, at least four well-funded USGS water availability programs have utilized the snapshot to facilitate NWIS data management and analysis (High Plains, Columbia River Plateau , Mississippi Embayment, and the Great Basin ). 

 

New NWIS data distribution methods such as web services and web mapping are evolving, and with them, the NWIS snapshot. Past snapshot development focused on output directly from an NWIS database installation, typically only available to USGS scientists in Water Science Centers.  Now, snapshot development focuses on retrieving output from web services. Produced in 2010 and 2011 to leverage USGS investment in streaming data from web services, the NWIS Web Services Snapshot for ArcGIS represents the next generation of data retrieval and management ( Figure 1 ). The newest Snapshot allows instant access to NWIS data from four different web services through ArcGIS, software available to all USGS scientists. The Snapshot database design enables efficient data compilation and preparation which is fundamental and pre-requisite to achieving the USGS Science Strategy vision of integrated ecosystem science based on integrated data.

 

The NWIS Web Services Geodatabase Snapshot improves retrieval and management of data within a well-defined database ( Figure 2 ). This data was previously only accessible by database experts. A proposed next phase of Snapshot development is now required to keep pace with evolving science needs and state-of-the-art technologies. Now that NWIS data can be quickly and easily retrieved and effectively managed, we propose to update, enhance, support and maintain the software. There will be an increased emphasis on training, outreach and partnership building to raise awareness about the Snapshot tool. In addition to the software package, deliverables include training and information sessions at USGS centers and partner institutions, updated tutorials based on answering real science questions, a public user community website, and published, open source code. By raising awareness and putting the software and source code in as many hands as possible, more data integration, visualization and analysis tools will emerge that build on the output from this project and, ultimately, facilitate integrated ecosystem science.

 

 

Figure 1 . Overview of the NWIS Web Services Snapshot for ArcGIS.

 

Figure 2 . NWIS Web Services Snapshot enables ArcGIS users to retrieve and manage NWIS data. The next phase of work will include publishing the Snapshot code. Feedback will continue to be provided to web services developers regarding application development needs and end user requirements.


Benefit to FSP/Scientists/Mission Areas

Produced to leverage USGS investment in streaming data from web services, the NWIS Web Services Snapshot represents the next generation of data retrieval and management. The newest Snapshot tool allows instant access to NWIS data from four different web services through ArcGIS, software available to all USGS scientists in all mission areas. Increased data retrieval efficiency reduces the steps required to retrieve and compile water data from multiple sites from what can be more than 30 steps to just a few clicks. As an end-user education tool, it promotes use of NWIS data from both web services and the NWIS database, which increases the production of scientific research and analysis that uses NWIS data. The Snapshot database design enables efficient data compilation and preparation which is fundamental and pre-requisite to achieving the USGS Science Strategy vision of integrated ecosystem science based on integrated data.

In kind funding and work leveraged

  • Leveraging NWIS Web Services
  • Extending previous work by David McCulloch, Brian Reece, and Texas Water Science Center CDI-supported Snapshot software
  • Leveraging ArcGIS Desktop enterprise license
  • In kind contributions from Core Science Informatics, Energy and Minerals, Fort Collins Science Center, CIDA, NWISWeb, and ten USGS science centers
  • In kind contributions from external reviewers at partner institutions

Partnerships

Partnerships with:

  • Core Science Informatics
  • NWIS Web Services
  • Energy and Minerals
  • USGS Enterprise GIS (EGIS)
  • USGS Science Centers (CIDA, FORT)


Work plan

 

Task 1 : Support and maintenance

  • Maintain the application code. Code will be updated as needed based on issues reported by end users, changes in ArcGIS software, and changes in web services.
  • Maintain, update and moderate a user community support website ( http://tx.cr.usgs.gov/snapshot ). The website, established in FY11, provides access to code, documentation, and testing results and end user ideas related to the application and web services.
  • Maintain an issue tracking database.
  • Streamline automated issue reporting . Currently, users may opt to submit error or issue reports when an exception is raised during execution of the application. These reports are currently stored as individual text files.  The reports will be aggregated in a single database for tracking. 
  • Provide feedback to NWIS web services developers. Communicate software developer and end-user requirements to web services development teams.
  • Update technical documentation as needed. Technical documentation includes the following information:
    • How to install and use the software
    • How to obtain user support, provide user feedback, and interact with the broader community of snapshot users
    • Technical specifications
    • Known functionality issues
    • Potential data policy issues relative to snapshot usage and distribution
    • Decision-making process logic for how issues are handled in coding
    • Planning for future updates and maintenance
    • What is and is not included in web services data compared to a direct NWIS database query

Task 2: Training, outreach, and awareness

  • Conduct training and information sessions . Hands-on training sessions and informational overview sessions will be presented in-person at selected USGS centers and via WebEx. An overview will also be presented for the USGS Executive Leadership Team.
  • Review, update, and expand existing training documentation (“user stories”). Produce web-based videos to accompany “user story” tutorials and migrate ArcGIS 9.3 tutorials to ArcGIS 10.0. Add indexing and improve search function in web documentation. Expand tutorials that demonstrate how to connect to or import the geodatabase data in other applications such as Excel, R, and Matlab.
  • Build partnerships through training and education to raise awareness about the Snapshot tool and web services. Presentations at partner institutions such as other federal agencies and universities will raise awareness about the tool and promote the development of future web services harvesting and data integration tools based on the Snapshot code and concept.

Task 3 : Software enhancements and updates

  • Gather evolving end-user requirements. This will be achieved by continuing to interview USGS scientists who need to access and utilize NWIS data. This cross-section will represent a variety of NWIS data expertise levels, including scientists who currently cannot easily create customized data queries. This task will be integrated with the training, education and outreach tasks.
  • Enhance the application based on testing results, user feedback and requirements. There were several changes that were suggested and deemed beneficial for the overall user experience and data analysis:
    • Create a ‘Cancel’ option. This will allow users to bail out of running a web services request.
    • Refine the water quality query builder. A “common English” predictive query builder has successfully been implemented and the ~18,000 parameter codes reduced to less than 5,000. To further support end users in selecting needed water quality parameters from 5,000 choices, the following refinements will be made:
      1. implement search by medium code (for example, soil or water);
      2. enable user to save and load query expressions; and
      3. provide a means to select ready-made query expressions based on common parameter sets (for example, parameter sets commonly queried for NAWQA studies).
    • Improve the query builder for unit and daily values . Right now users can select any combination of parameter and statistics code in the Daily Values tab. However, some combinations will not yield results from the web service. To assist users, auto-matching of parameter and statistics codes will be implemented. Users can also only retrieve data for one unit value parameter and one daily value parameter at a time. Multiple parameter selection will be implemented to allow simultaneous retrieval of complementary parameters such as dissolved oxygen, temperature, and streamflow.
    • Add site retrieval by parameter code and period of record (web service functionality permitting ).
  • Update the geodatabase design. Changes in web services will necessitate changes in the geodatabase schema and attributes as more data such as that for groundwater and the instantaneous data archives become available.  In addition, a Snapshot-generated html report summarizing number of samples and period of record per site will be made more readily accessible for analysis by converting it to a geodatabase table.

Task 4: Testing and revision

  • Test major revisions of the ArcGIS 10 Add-In and geodatabase. This will include testing on PC and Macintosh platforms.
  • Update tester feedback form and testing instructions.
  • Recruit testers. In anticipation of the Snapshot Add-In becoming publicly available, testers at partner institutions may be asked to help test the software.
  • Summarize testing results.
  • Prioritize and execute revisions based on testing results. Components of the software packages (code, interface, geodatabase and documentation) will be revised based on both automated and manual issue reporting results. Revisions will be ranked according to severity, frequency of occurrence, and end user requirements.

Task 5: Publish the software and website.

  • Investigate publication of the source code under an open-source licensing agreement.
  • Migrate the USGS internal Snapshot website from the internal to the public domain.

 

Communication

  • Participate on weekly and monthly CDI conference calls.
  • Monthly E-mail updates to CDI.
  • Other E-mail, phone, and WebEx conference calls as needed.

 


Deliverables

  1. ArcGIS 10 Add-In software package.*
  2. Training and information sessions.
  3. Feedback to web services developers.
  4. A public-facing user community website.
  5. Training and technical documentation.*
  6. Publicly-available source code.*
  7. Issue tracking database.*
  8. Software testing results.*

 

* To be made available on the user community website.

 

Timeline

 

 

Budget

 

The total estimated project cost is $119,513.