Confluence Retirement

Due to the feedback from stakeholders and our commitment to not adversely impact USGS science activities that Confluence supports, we are extending the migration deadline to January 2023.

In an effort to consolidate USGS hosted Wikis, myUSGS’ Confluence service is targeted for retirement. The official USGS Wiki and collaboration space is now SharePoint. Please migrate existing spaces and content to the SharePoint platform and remove it from Confluence at your earliest convenience. If you need any additional information or have any concerns about this change, please contact myusgs@usgs.gov. Thank you for your prompt attention to this matter.

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The summary will repeat but provide more details about the information provided on the cover sheet including the following:

Introduction and background

At the first 2009 CDI workshop, scientists stated one of the top three goals was to make it very simple to access corporate databases. In 2009, scientists and users made it clear to CDI - Make these setup mashup steps go away - 1-click.  Our current viewing and app portfolio across USGS, DOI, and generally industry-wide does not meet this need. We've had many problems to solve such as we have 100s of viewer instances in 10s of flavors, there still isn’t a good standard for this, and we have yet to establish business case for programs and developers to move towards new approach. 

The National Geospatial Program took lead and developed a capability to save the map session mashup setup or "context" in one-click in a standard map context then open it in web viewer APIs and popular thick clients like ArcMap and Google Earth. This project attempts to develop a core service that other viewers can call to save map  sessions, build session read and view Libraries to open in popular viewer flavors, and work on getting a single standard via coordinating with OGC, ESRI/OWF, and NGA. NGP has worked over 3 CDI years collaborating with CIDA, WiM, NEIC, CSAS, and EGSC on advancing this concept to work not just in The National Map applications and services, but the any web services and applications that can support saving as multiple standards and opening with a focus on being able to use this in any web viewer application in USGS. 

This year's proposal is to focus initial deployment of current SaveAs and OpenIn services and support continuous improvement on standardizing the context, hooking into OpenLayers and ESRI APIs.

  • CDI SSF Category : Computational tools and services
  • Project Title : Deploy Save As and Open In USGS Wide

Project Leaders or Principal Investigators

NameOrganization AffiliationMailing AddressPhone NumberEmailRole
Rob DollisonUSGS NGPUSGS HQ Reston, VA703-648-5724rdollison@usgs.govNGP Delivery Services Lead
Glenn GuempelUSGS NGPUSGS HQ Reston, VA gguempel@usgs.govNGP Standards Architect

All Personnel Involved

NameOrganization AffiliationMailing AddressPhone NumberEmailRole
Rick BrownUSGS NGP NGTOCRolla, MO  NGTOC Project Manager
Dave HughesUSGS NGP NGTOCDenver, CO  NGTOC Engineering Projects Manager
Robert DjurasajUSGS NGP NGTOCDenver, CO  NGTOC Developer Contractor

Aaron Dandy

USGS NGP NGTOCDenver, CO  NGTOC Developer Contractor
Jim McAndrewsUSGS NGP NGTOCDenver, CO  NGTOC Developer Contractor
Tim KernUSGS CSAS SciencebaseFt Collins, CO  CSAS Sciencebase Developer Lead
Natalie LatysheUSGS CSASDenver, CO  CSAS Sciencebase Project Manager
Sky BristolUSGS AESIRDenver, CO  CSAS AESIR Research lead/Sciencebase SME
Mike FrameUSGS CSASOak Ridge, TN  CSAS Hosting/Engineering Lead
Tim MancusoUSGS CSASDenver, CO  CSAS Hosting/Engineering Manager Contractor
David BlodgettUSGS CIDAMadison, WI  CIDA Architect/Developer
Ivan SuftinUSGS CIDAMadison, WI  CIDA Developer
John AguinaldoUSGS ESGCReston, VA  ESGC Developer Contractor

Collaborating Organizations

NameOrganization AffiliationMailing AddressPhone NumberEmailRole
MultipleESRI   Technical Architects Points of Contacts
MultipleOpenGeo   Technical Architects Points of Contacts
David WeslohOGC/NGASt Louis, MO  OGC OWS SWG Chair
Justin HaasNGA Palanterra X3Fort Belvoir, VA  Palanterra X3 Product Manager

Other Relevant Personnel or points of contact

NameOrganization AffiliationMailing AddressPhone NumberEmailRole
Dave Soller    NCGMP Sr. Leader Sponsor
Nate Booth    CIDA Sr. Leader Sponsor
Kevin HopePaul Wiese  703-648-4543khope@usgspmwiese@usgs.govNGP Sr. Leader Sponsor
Matt TricomiUSGSDenver, CO720-244-3063mtricomi@usgs.govArchitect Contractor
Greg SmoczykUSGS NEICGolden, CO  CDI SME Focus on ArcMap Integration
Gary LatzkeUSGS WiMMadison, WI  WiM Projects Manager
Jon BaierUSGS WiMMadison, WI  CDI SME Focus on ESRI Flex API Integration
Cassandra LadinoUSGS EGSCReston, VA  EGSC Projects Manager

Detailed description of geographic/geologic/ecosystem/habitat/taxonomic/other context of the project and its importance or value if applicable.

This project enables simplifying map service integration. The USGS provides many national, regional and local datasets for download, streaming interaction such as WFS/WCS, and analysis. Ultimately, most datasets are presented for visualization in "viewers" with basic navigation and interaction for inspection and even lightweight WebGIS like web service functions, annotations, etc. Many viewers – different APIs, clients, purposes, and niche functions – are invested in at USGS and DOI and the whole Federal Government. This project has the capability to save map contexts of the mashup a user makes in a viewer. Any context requiring Spatial Visualization (i.e. Geographic, Geologic, Water, Ecosystem, Habitat, Topographic, etc.)

Scope

Scientists, Analysts, etc. use many varying science and map tools with viewing capabilities to be used while working different parts of a problem. They’ll move from one tool to another: Finding - Use TNM to download foundational data, Discovering - Sciencebase to find thematic data, Processing - CIDA GDP for processing data, Analyzing - ArcMap for in-depth study, etc. For example, taking a map session mashup they make in web views and move it easily into their advanced thick clients like ArcMap and Google Earth. It could take a scientist, planner, user, etc. a half-hour or more to create these unique views - zoom in, re-order services, change opacity, add services, turn on layers, etc. - and to have to recreate that in their environment, which could be one of the large reasons why services are not taking off - its too hard.  Meaning, each time they switch tools, they need to re-setup their session or “mash-up” which is not their priority and is a time-waster or nuisance

The solution is not "1 viewer" or "1 viewer API". We are stuck in a multiple viewer environment, we could recommend a few APIs, and restrict others at best. The problem with this is that when someone goes to a new viewer, they don't always have the same backdrop basemaps, common list of overlays, and most viewers don't have capability to add services into the viewer, except thick clients and The National Map. Essentially there is no way for a user to jump easily between viewers.

The output of this project is to implement a web service that any viewer API can call that allows a user to save the last view of the multiple data map services. By having this capability become USGS/DOI wide, or even integrated late into the Geospatial platform, this should look to increase web service use by making easier to use between different web viewers and thick clients. 

The goals this year by August 31, 2013 are to:

TaskEnd Product/ServiceOrganization
Geospatial - Save As FrameworkLive Application Service with initial Transforms to Px3, ESRI, KML, and later, support more formats as source and target formats, such as new OGC betaNGTOC/OGC
Architecture, Design, CoordinationOGC OWS Context Feedback incorporated into standardNGP/OGC
Viewer Library API, SaveAs, and OpenIn Migration to SB/CSASInterim and Mid-term hosting decision made/planned - Explore Enterprise rollout discussions/decisions (communicating, training, hosting, investing, inventory) and moving towards program/developer adoption education, communication, and working with programs on where fits in prioritiesCSAS/NGTOC
Open In OpenLayers Framework (Read Px3 JSON, ESRI JSONBeta OpenLayers submitted to SciencebaseCSAS/CIDA
Save to ArcMap (10 days)Add transform to MXDNGTOC
Open in Flex (Convert Px3 JSON to ESRI JSON)OGC ESRI/REST Context Feedback incorporated into standard

NCGMP

The CDI Funding required would be for 30% of the effort for the non-NGTOC/NGP items. More details on the technical approach, level of effort commitment by organization, and quarterly timeline are provided below.

Plan

FY13 SaveAs-OpenIn Scope and Plan for OGC Beta Test

Technical Approach

  • User Stories - Support these user stories
  • Geospatial - Save As FrameworkDevelop an application service that can take the output of a session context from a viewer in a specific standard, then transform and export it to multiple formats. It was first developed directly into The National Map Palanterra x3 Viewer based on ESRI JS supporting multiple target formats (KML, Px3 JSON, and PDF/Images). It has since been refactored as a standalone application. Now, the focus is on going live with existing transforms, finding a hosting location, complete remaining transforms, and explore new context standards with OGC via testbed participation and evaluation of 2 standards: New OGC OWS XML conversion to JSON, and the ESRI JSON adoption to OGC. One thing that shouldnt go unmentioned is that opening a context file not located on the same domain requires additional cross-domain and script/SQL injection validation controls on top of schema validation. This "OpenIn" functionality also was developed as part of the framework which its code, for javascript libraries, an be leveraged for use in other viewers.
  • Architecture, Design, Coordination -  In 2010, the team architected a Target Application ArchitectureTarget Info Flow, and overall Architecture Diagram that is based on a common context and standalone application web service. The role is to support coordinating design requirements and promoting this concept of operations across the multiple Viewer APIs, security requirements, and use of the web service. As well, help progress the OGC standards towards one that can be industry adopted and adopted across USGS/DOI in JSON to be used by popular Viewer APIs.
  • Viewer Library API, SaveAs, and OpenIn Migration to SB/CSAS -  Since 2011, its been discussed with the Sciencebase team to be able to save, catalog and then open contexts in different viewers. In 2013, we are proposing at minimum to evaluate what is needed to get the application web service in the same Sciencebase hosting environment to go live with save as in permanently, and decide the length of time that it may need to continue in interim reside on NGP servers.
  • Open In OpenLayers Framework (Read Px3 JSON, ESRI JSON -  In 2011, we developed the capability to read ESRI REST Web services in Openlayers. That code was re-used in 2012, and taken a step further to be called via the Px3 context standard. Now, the approach is to complete the context integration in Openlayers with all standard function calls as well as offer up this new framework to be used and maintained by the ScienceBase Team. As well, we should look to continue to coordinate with another 2012 CDI project that researched Openlayers HTML5 library integration. Example of work still to be done: 
    Read in some missing session input (Basemap Buttons, layers, zoom), Debug some ESRI REST Service compatibility, Incorporate with ERSG OpenLayers work, and discover ways to make this part of standard OpenLayers Library
  • Save to ArcMap (10 days) -   Early on focused on enabling the initial problem through manual solutions (i.e. an MXD with common service palette), and in 2011, NGP and NEIC researched with ESRI Architects multiple methods of how to move from web context into an ArcMap MXD file. Through discussions, and updates from ESRI in 2012, we now have the code that works in ArcGIS Server 10.1 that will create an MXD file for the user to save via ArcPy libraries. When a AGS 10.1 environment is made available, we are suggesting to go live with this transform target format.  The Save As Function needs to simply call a 15 line python script to output an MXD file, but to do it needs to be running 10.1 server side, but the resulting MXD can be opened in ArcMap 9.x and up. ArcMap still slow & “bloated” to open many services due to how they make lyr files, but it works – which may be an eGIS type enhancement request (make services usable in ArcMap).
  • Open in Flex (Convert Px3 JSON to ESRI JSON) -  In 2011, ESRI came out with a REST Standard offered to the Open Web Framework, and later offered it to OGC for consideration. Since 2011, ESRI has made all Viewer APIs, and ArcGIS.com completely integrated with this format. We would like to coordinate as part of an OGC sponsored testbed a second standard JSON context format and tracking the ESRI/REST OGC SWG progress towards offering a 2nd alternative option. As well, this task would produce the capability to call the Save As function and transform the REST/ESRI JSON to other targets. Likely this will require more code refactoring by NGTOC to the final architecture factoring levels originally required 

Active Discussion Threads

  • Discussion on JSON and Security implementation - Context Security - Good threads on how we handle cross-domain JSON calls (client side or server side processing is the question)
  • Standard JSON Context - Context Standard - OGC has drafted the new OWS context in JSON encoding - now onto seeing the use cases and requirements come through.

Background Architecture

Project Experience

TaskGroupAcronym$ TargetGovt LeadDeveloper/SMEDeveloper LocationExperience
Geospatial - Save As FrameworkNational Geospatial Technical Operations CenterNGTOCNGTOCDavid Hughes / Rick Brown / Rob DollisonNGTOC Team led by Rick BrownDenver Bldg 810
  • Dollison - Has lead CDI SaveAs project since 2010
  • Hughes - Managed CDI Development of SaveAs since 2011
  • Brown - Deep Experience in Sciencebase and TNM
Architecture, Design, CoordinationNational Geospatial ProgramNGPNGPGlenn GuempelMatt Tricomi (Contractor)Denver Bldg 810
  • Tricomi - Architected CDI SaveAs project since 2009
  • Guempel - OGC lead for NGP and more than decade experience in working with OGC
Viewer Library API, SaveAs, and OpenIn Migration to SB/CSASCore Science Analytics and SynthesisCSASCSASTim Kern / Natalie LatysheTim Kern / Tim Mancuso and coordinating heavilly with NGTOCDenver/Fort
  • ScienceBase Lead Team
Open In OpenLayers Framework (Read Px3 JSON, ESRI JSONCore Science Analytics and SynthesisCSASCIDA/CSAS/EGSCNatalie Latyshe / David BlodgettIvan Suftin / Tim Kern and input from John Aguinaldo and the Openlayers HTML5 library effortsFort/Madison
  • Blodgett/Suftin - Lead and Developer in 2012
  • Aguinaldo/Kern - Lead on OpenLayers projects in EGSC and Sciencebase
Save to ArcMap (10 days)National Geospatial Technical Operations CenterNGTOCNGTOCDavid Hughes / Rick BrownNGTOC Team led by Rick BrownDenver Bldg 810
  • Same SaveAs team noted above
  • In 2012, tranitioned NEIC research on SaveAs MXD
Open in Flex (Convert Px3 JSON to ESRI JSON)National Cooperative Geologic Mapping ProgramNCGMPNCGMPChris GarrityChris Garrity/ (Input from WiM Jon Baier)Reston
  • Garritty - Deep experience in Flex Viewer Widget development

We will not be submitting bios, profiles, or CVs for this project.

Commitment to Effort

TaskEnd Product/ServiceLOE (Days)%Funding SourceAcronym
Geospatial - Save As FrameworkLive enterprise Application Service with initial Transforms to Px3, ESRI, KML which will be code maintained by NGP and hosted mid-term by CSAS9046%NGP and OGC FundingNGTOC
Architecture, Design, CoordinationOGC OWS Context Feedback incorporated into standard2010%NGP and OGC FundingNGP
Viewer Library API, SaveAs, and OpenIn Migration to SB/CSASInterim and Mid-term hosting decision made/planned158%NGP and CSAS FundingCSAS
Open In OpenLayers Framework (Read Px3 JSON, ESRI JSONBeta OpenLayers submitted for incorporation as part of Sciencebase code base3015%CDI FundingCSAS
Save to ArcMap (10 days)Add transform to MXD as part of Save As application service105%NGP and OGC FundingNGTOC
Open in Flex (Convert Px3 JSON to ESRI JSON)OGC ESRI/REST Context Feedback incorporated into standard which would become one of the primary ESRI Flex libraries maintained by CSS.3015%CDI FundingNCGMP

Budget

Please provide a detailed budget for the project using Table 3. Include at least the following categories of information, separating the CDI funds from the in-kind match:

  1. Salaries and Wages (see also Budgetary Considerations)
  2. Fringe Benefits
  3. Field Expenses
  4. Other Direct Cost Line Items
  5. Total Direct Charges
  6. Indirect Charges (Overhead)

...