Confluence Retirement

In an effort to consolidate USGS hosted Wikis, the myUSGS Confluence service is targeted for retirement on January 28, 2022. The official USGS Wiki and collaboration space is now SharePoint. Please migrate existing spaces and content to the SharePoint platform and remove it from Confluence at your earliest convenience. If you need any additional information or have any concerns about this change, please contact myusgs@usgs.gov. Thank you for your prompt attention to this matter.
Skip to end of metadata
Go to start of metadata

Cover Sheet Title: Deploy Map Save As and Open In Application Service USGS Wide

CDI SSF Category : Computational tools and services

Applicants/Principal Investigator(s) : Glenn Guempel, USGS NGP Standards Architect; Rob Dollison, USGS NGP Delivery Services Lead

Abstract - At the first 2009 CDI workshop, scientists stated one of the top three goals was to make it very simple to access corporate databases. For example, taking a map session mashup they make in web views and move it easily into their advanced thick clients like ArcMap and Google Earth. It could take a scientist, planner, user, etc. a half-hour or more to create these unique views - zoom in, re-order services, change opacity, add services, turn on layers, etc. - and to have to recreate that in their environment, which could be one of the large reasons why services are not taking off - its too hard. The National Geospatial Program took lead and developed a capability to save the map session mashup setup or "context" in one-click in a standard map context then open it in web viewer APIs and popular thick clients like ArcMap and Google Earth. NGP has worked over 3 CDI years collaborating with CIDA, WiM, NEIC, CSAS, and EGSC on advancing this concept to work not just in The National Map applications and services, but the any web services and applications that can support saving as multiple standards and opening with a focus on being able to use this in any web viewer application in USGS. This year's proposal is to focus initial deployment of current SaveAs and OpenIn services and support continuous improvement on standardizing the context, hooking into OpenLayers and ESRI APIs.

%Funding Source
31%Total funding amount requested from CDI
69%Total in-kind funding (NGP, OGC, CSAS)
100%Total Funding Needed
Datasets

All Open-Standard Web Services available in USGS Web Viewers (i.e. TNM, NHSS, HDDS, Atlas, Water, MRDATA, NGMDB, CRC, MARIS, Multiple CSAS datasets

Geographic/geologic/ecosystem/habitat/taxonomic/other context

Any context requiring Spatial Visualization (i.e. Geographic, Geologic, Water, Ecosystem, Habitat, Topogrpahic, etc.)

Type of Product(s) Generated
  • Application Web Services - SaveAs Web Service, OpenIn Application Services, The National Map ESRI JS API Viewer Integration, OpenLayers Viewer Integration, ESRI Flex Viewer Integration
  • Map Context Formats - Px3 Map Context, OWS Map Context JSON, ESRI/OWF JSON Map Context, OGC KML, ArcMap MXD

Summary

The summary will repeat but provide more details about the information provided on the cover sheet including the following:

Introduction and background

At the first 2009 CDI workshop, scientists stated one of the top three goals was to make it very simple to access corporate databases. In 2009, scientists and users made it clear to CDI - Make these setup mashup steps go away - 1-click.  Our current viewing and app portfolio across USGS, DOI, and generally industry-wide does not meet this need. We've had many problems to solve such as we have 100s of viewer instances in 10s of flavors, there still isn’t a good standard for this, and we have yet to establish business case for programs and developers to move towards new approach. 

The National Geospatial Program took lead and developed a capability to save the map session mashup setup or "context" in one-click in a standard map context then open it in web viewer APIs and popular thick clients like ArcMap and Google Earth. This project attempts to develop a core service that other viewers can call to save map  sessions, build session read and view Libraries to open in popular viewer flavors, and work on getting a single standard via coordinating with OGC, ESRI/OWF, and NGA. NGP has worked over 3 CDI years collaborating with CIDA, WiM, NEIC, CSAS, and EGSC on advancing this concept to work not just in The National Map applications and services, but the any web services and applications that can support saving as multiple standards and opening with a focus on being able to use this in any web viewer application in USGS. 

This year's proposal is to focus initial deployment of current SaveAs and OpenIn services and support continuous improvement on standardizing the context, hooking into OpenLayers and ESRI APIs.

  • CDI SSF Category : Computational tools and services
  • Project Title : Deploy Save As and Open In USGS Wide

Project Leaders or Principal Investigators

NameOrganization AffiliationMailing AddressPhone NumberEmailRole
Rob DollisonUSGS NGPUSGS HQ Reston, VA703-648-5724rdollison@usgs.govNGP Delivery Services Lead
Glenn GuempelUSGS NGPUSGS HQ Reston, VA gguempel@usgs.govNGP Standards Architect

All Personnel Involved

NameOrganization AffiliationMailing AddressPhone NumberEmailRole
Rick BrownUSGS NGP NGTOCRolla, MO  NGTOC Project Manager
Dave HughesUSGS NGP NGTOCDenver, CO  NGTOC Engineering Projects Manager
Robert DjurasajUSGS NGP NGTOCDenver, CO  NGTOC Developer Contractor

Aaron Dandy

USGS NGP NGTOCDenver, CO  NGTOC Developer Contractor
Jim McAndrewsUSGS NGP NGTOCDenver, CO  NGTOC Developer Contractor
Tim KernUSGS CSAS SciencebaseFt Collins, CO  CSAS Sciencebase Developer Lead
Natalie LatysheUSGS CSASDenver, CO  CSAS Sciencebase Project Manager
Sky BristolUSGS AESIRDenver, CO  CSAS AESIR Research lead/Sciencebase SME
Mike FrameUSGS CSASOak Ridge, TN  CSAS Hosting/Engineering Lead
Tim MancusoUSGS CSASDenver, CO  CSAS Hosting/Engineering Manager Contractor
David BlodgettUSGS CIDAMadison, WI  CIDA Architect/Developer
Ivan SuftinUSGS CIDAMadison, WI  CIDA Developer
John AguinaldoUSGS ESGCReston, VA  ESGC Developer Contractor

Collaborating Organizations

NameOrganization AffiliationMailing AddressPhone NumberEmailRole
MultipleESRI   Technical Architects Points of Contacts
MultipleOpenGeo   Technical Architects Points of Contacts
David WeslohOGC/NGASt Louis, MO  OGC OWS SWG Chair
Justin HaasNGA Palanterra X3Fort Belvoir, VA  Palanterra X3 Product Manager

Other Relevant Personnel or points of contact

NameOrganization AffiliationMailing AddressPhone NumberEmailRole
Dave Soller    NCGMP Sr. Leader Sponsor
Nate Booth    CIDA Sr. Leader Sponsor
Paul Wiese  703-648-4543pmwiese@usgs.govNGP Sr. Leader Sponsor
Matt TricomiUSGSDenver, CO720-244-3063mtricomi@usgs.govArchitect Contractor
Greg SmoczykUSGS NEICGolden, CO  CDI SME Focus on ArcMap Integration
Gary LatzkeUSGS WiMMadison, WI  WiM Projects Manager
Jon BaierUSGS WiMMadison, WI  CDI SME Focus on ESRI Flex API Integration
Cassandra LadinoUSGS EGSCReston, VA  EGSC Projects Manager

Detailed description of geographic/geologic/ecosystem/habitat/taxonomic/other context of the project and its importance or value if applicable.

This project enables simplifying map service integration. The USGS provides many national, regional and local datasets for download, streaming interaction such as WFS/WCS, and analysis. Ultimately, most datasets are presented for visualization in "viewers" with basic navigation and interaction for inspection and even lightweight WebGIS like web service functions, annotations, etc. Many viewers – different APIs, clients, purposes, and niche functions – are invested in at USGS and DOI and the whole Federal Government. This project has the capability to save map contexts of the mashup a user makes in a viewer. Any context requiring Spatial Visualization (i.e. Geographic, Geologic, Water, Ecosystem, Habitat, Topographic, etc.)

Scope

Scientists, Analysts, etc. use many varying science and map tools with viewing capabilities to be used while working different parts of a problem. They’ll move from one tool to another: Finding - Use TNM to download foundational data, Discovering - Sciencebase to find thematic data, Processing - CIDA GDP for processing data, Analyzing - ArcMap for in-depth study, etc. For example, taking a map session mashup they make in web views and move it easily into their advanced thick clients like ArcMap and Google Earth. It could take a scientist, planner, user, etc. a half-hour or more to create these unique views - zoom in, re-order services, change opacity, add services, turn on layers, etc. - and to have to recreate that in their environment, which could be one of the large reasons why services are not taking off - its too hard.  Meaning, each time they switch tools, they need to re-setup their session or “mash-up” which is not their priority and is a time-waster or nuisance

The solution is not "1 viewer" or "1 viewer API". We are stuck in a multiple viewer environment, we could recommend a few APIs, and restrict others at best. The problem with this is that when someone goes to a new viewer, they don't always have the same backdrop basemaps, common list of overlays, and most viewers don't have capability to add services into the viewer, except thick clients and The National Map. Essentially there is no way for a user to jump easily between viewers.

The output of this project is to implement a web service that any viewer API can call that allows a user to save the last view of the multiple data map services. By having this capability become USGS/DOI wide, or even integrated late into the Geospatial platform, this should look to increase web service use by making easier to use between different web viewers and thick clients. 

The goals this year by August 31, 2013 are to:

TaskEnd Product/ServiceOrganization
Geospatial - Save As FrameworkLive Application Service with initial Transforms to Px3, ESRI, KML, and later, support more formats as source and target formats, such as new OGC betaNGTOC/OGC
Architecture, Design, CoordinationOGC OWS Context Feedback incorporated into standardNGP/OGC
Viewer Library API, SaveAs, and OpenIn Migration to SB/CSASInterim and Mid-term hosting decision made/planned - Explore Enterprise rollout discussions/decisions (communicating, training, hosting, investing, inventory) and moving towards program/developer adoption education, communication, and working with programs on where fits in prioritiesCSAS/NGTOC
Open In OpenLayers Framework (Read Px3 JSON, ESRI JSONBeta OpenLayers submitted to SciencebaseCSAS/CIDA
Save to ArcMap (10 days)Add transform to MXDNGTOC
Open in Flex (Convert Px3 JSON to ESRI JSON)OGC ESRI/REST Context Feedback incorporated into standard

NCGMP

The CDI Funding required would be for 30% of the effort for the non-NGTOC/NGP items. More details on the technical approach, level of effort commitment by organization, and quarterly timeline are provided below.

Plan

FY13 SaveAs-OpenIn Scope and Plan for OGC Beta Test

Technical Approach

  • User Stories - Support these user stories
  • Geospatial - Save As FrameworkDevelop an application service that can take the output of a session context from a viewer in a specific standard, then transform and export it to multiple formats. It was first developed directly into The National Map Palanterra x3 Viewer based on ESRI JS supporting multiple target formats (KML, Px3 JSON, and PDF/Images). It has since been refactored as a standalone application. Now, the focus is on going live with existing transforms, finding a hosting location, complete remaining transforms, and explore new context standards with OGC via testbed participation and evaluation of 2 standards: New OGC OWS XML conversion to JSON, and the ESRI JSON adoption to OGC. One thing that shouldnt go unmentioned is that opening a context file not located on the same domain requires additional cross-domain and script/SQL injection validation controls on top of schema validation. This "OpenIn" functionality also was developed as part of the framework which its code, for javascript libraries, an be leveraged for use in other viewers.
  • Architecture, Design, Coordination -  In 2010, the team architected a Target Application ArchitectureTarget Info Flow, and overall Architecture Diagram that is based on a common context and standalone application web service. The role is to support coordinating design requirements and promoting this concept of operations across the multiple Viewer APIs, security requirements, and use of the web service. As well, help progress the OGC standards towards one that can be industry adopted and adopted across USGS/DOI in JSON to be used by popular Viewer APIs.
  • Viewer Library API, SaveAs, and OpenIn Migration to SB/CSAS -  Since 2011, its been discussed with the Sciencebase team to be able to save, catalog and then open contexts in different viewers. In 2013, we are proposing at minimum to evaluate what is needed to get the application web service in the same Sciencebase hosting environment to go live with save as in permanently, and decide the length of time that it may need to continue in interim reside on NGP servers.
  • Open In OpenLayers Framework (Read Px3 JSON, ESRI JSON -  In 2011, we developed the capability to read ESRI REST Web services in Openlayers. That code was re-used in 2012, and taken a step further to be called via the Px3 context standard. Now, the approach is to complete the context integration in Openlayers with all standard function calls as well as offer up this new framework to be used and maintained by the ScienceBase Team. As well, we should look to continue to coordinate with another 2012 CDI project that researched Openlayers HTML5 library integration. Example of work still to be done: 
    Read in some missing session input (Basemap Buttons, layers, zoom), Debug some ESRI REST Service compatibility, Incorporate with ERSG OpenLayers work, and discover ways to make this part of standard OpenLayers Library
  • Save to ArcMap (10 days) -   Early on focused on enabling the initial problem through manual solutions (i.e. an MXD with common service palette), and in 2011, NGP and NEIC researched with ESRI Architects multiple methods of how to move from web context into an ArcMap MXD file. Through discussions, and updates from ESRI in 2012, we now have the code that works in ArcGIS Server 10.1 that will create an MXD file for the user to save via ArcPy libraries. When a AGS 10.1 environment is made available, we are suggesting to go live with this transform target format.  The Save As Function needs to simply call a 15 line python script to output an MXD file, but to do it needs to be running 10.1 server side, but the resulting MXD can be opened in ArcMap 9.x and up. ArcMap still slow & “bloated” to open many services due to how they make lyr files, but it works – which may be an eGIS type enhancement request (make services usable in ArcMap).
  • Open in Flex (Convert Px3 JSON to ESRI JSON) -  In 2011, ESRI came out with a REST Standard offered to the Open Web Framework, and later offered it to OGC for consideration. Since 2011, ESRI has made all Viewer APIs, and ArcGIS.com completely integrated with this format. We would like to coordinate as part of an OGC sponsored testbed a second standard JSON context format and tracking the ESRI/REST OGC SWG progress towards offering a 2nd alternative option. As well, this task would produce the capability to call the Save As function and transform the REST/ESRI JSON to other targets. Likely this will require more code refactoring by NGTOC to the final architecture factoring levels originally required 

Active Discussion Threads

  • Discussion on JSON and Security implementation - Context Security - Good threads on how we handle cross-domain JSON calls (client side or server side processing is the question)
  • Standard JSON Context - Context Standard - OGC has drafted the new OWS context in JSON encoding - now onto seeing the use cases and requirements come through.

Background Architecture

Project Experience

TaskGroupAcronym$ TargetGovt LeadDeveloper/SMEDeveloper LocationExperience
Geospatial - Save As FrameworkNational Geospatial Technical Operations CenterNGTOCNGTOCDavid Hughes / Rick Brown / Rob DollisonNGTOC Team led by Rick BrownDenver Bldg 810
  • Dollison - Has lead CDI SaveAs project since 2010
  • Hughes - Managed CDI Development of SaveAs since 2011
  • Brown - Deep Experience in Sciencebase and TNM
Architecture, Design, CoordinationNational Geospatial ProgramNGPNGPGlenn GuempelMatt Tricomi (Contractor)Denver Bldg 810
  • Tricomi - Architected CDI SaveAs project since 2009
  • Guempel - OGC lead for NGP and more than decade experience in working with OGC
Viewer Library API, SaveAs, and OpenIn Migration to SB/CSASCore Science Analytics and SynthesisCSASCSASTim Kern / Natalie LatysheTim Kern / Tim Mancuso and coordinating heavilly with NGTOCDenver/Fort
  • ScienceBase Lead Team
Open In OpenLayers Framework (Read Px3 JSON, ESRI JSONCore Science Analytics and SynthesisCSASCIDA/CSAS/EGSCNatalie Latyshe / David BlodgettIvan Suftin / Tim Kern and input from John Aguinaldo and the Openlayers HTML5 library effortsFort/Madison
  • Blodgett/Suftin - Lead and Developer in 2012
  • Aguinaldo/Kern - Lead on OpenLayers projects in EGSC and Sciencebase
Save to ArcMap (10 days)National Geospatial Technical Operations CenterNGTOCNGTOCDavid Hughes / Rick BrownNGTOC Team led by Rick BrownDenver Bldg 810
  • Same SaveAs team noted above
  • In 2012, tranitioned NEIC research on SaveAs MXD
Open in Flex (Convert Px3 JSON to ESRI JSON)National Cooperative Geologic Mapping ProgramNCGMPNCGMPChris GarrityChris Garrity/ (Input from WiM Jon Baier)Reston
  • Garritty - Deep experience in Flex Viewer Widget development

We will not be submitting bios, profiles, or CVs for this project.

Commitment to Effort

TaskEnd Product/ServiceLOE (Days)%Funding SourceAcronym
Geospatial - Save As FrameworkLive enterprise Application Service with initial Transforms to Px3, ESRI, KML which will be code maintained by NGP and hosted mid-term by CSAS9046%NGP and OGC FundingNGTOC
Architecture, Design, CoordinationOGC OWS Context Feedback incorporated into standard2010%NGP and OGC FundingNGP
Viewer Library API, SaveAs, and OpenIn Migration to SB/CSASInterim and Mid-term hosting decision made/planned158%NGP and CSAS FundingCSAS
Open In OpenLayers Framework (Read Px3 JSON, ESRI JSONBeta OpenLayers submitted for incorporation as part of Sciencebase code base3015%CDI FundingCSAS
Save to ArcMap (10 days)Add transform to MXD as part of Save As application service105%NGP and OGC FundingNGTOC
Open in Flex (Convert Px3 JSON to ESRI JSON)OGC ESRI/REST Context Feedback incorporated into standard which would become one of the primary ESRI Flex libraries maintained by CSS.3015%CDI FundingNCGMP

Budget

Please provide a detailed budget for the project using Table 3. Include at least the following categories of information, separating the CDI funds from the in-kind match:

  1. Salaries and Wages (see also Budgetary Considerations)
  2. Fringe Benefits
  3. Field Expenses
  4. Other Direct Cost Line Items
  5. Total Direct Charges
  6. Indirect Charges (Overhead)

1. SALARIES (inc. number of hours and hourly rate):

Included in official Proposal document

2. FRINGE BENEFITS:

Included in official Proposal document

3. TRAVEL EXPENSES*:

Included in official Proposal document

4. OTHER DIRECT COSTS: (itemize)

Included in official Proposal document

*Noting the Bureau’s goal of reducing travel by 30% in FY13.

Timeline

Milestone

Geospatial - Save As Framework

Architecture, Design, Coordination

Viewer Library API, SaveAs, and OpenIn Migration to SB/CSAS

Open In OpenLayers Framework (Read Px3 JSON, ESRI JSON

Save to ArcMap (10 days)

Open in Flex (Convert Px3 JSON to ESRI JSON)

Q1

Plan OGC Testbed with NGP EA

Test ArcMap

Initial Deploy on NGP HW

Re-organize all past 4 years of SaveAs/OpenIn Content

Facilitate Transition of concepts/designs to New Teams and Design

Scope out FY12 scope issues

Integrate into SB plans

 Scope out FY12 scope issues

Integrate into SB plans

Complete ArcPy Libraries

Initial Deploy on NGP HW

 Scope out FY12 scope issues

 

Integrate into SB plans

Q2

Complete KML Transform

NGTOC begin working w/ OGC SWG on getting OWS JSON working in OGC Testbed (OGC Funding?)

NGP EA investigate REST SWG for context overlap and changes to ESRI OWF Context with OWS Context

Confirm if direction for Context Standard should be OWS or REST OWF changes JSON

Test hosting ESRI JS Libraries

Test hosting ESRI JS with Px3 Libraries

Test hosting SaveAs Libraries

Test hosting OpenIn Libraries

Complete OpenIn OL tasks

Complete OL Library transitioning from CIDA Work

Align communications to get other CSS OL libraries using such

 

 Coordinate SB CSAS Scope needsNGMDB Develop SaveAs hooks for CSS Wide Flex Library 
Q3

Complete TestBed and OWS JSON Feedback to OGC

NGTOC Build OWS Reader with NGA Px3 Team (CDI Funding)

NGTOC Write SaveAs w/ Source as OWS (currently is Px3)

Coordinate Hosting Stack Transition to CSAS/Bldg 810 Stack

Support OGC Communications

Document exploring REST JSON config changes to OWF JSON and further faciltate

Test hosting OL Libraries

Test hosting ESRI Flex Libraries

 

 

Complete SaveAs OL tasks 

CSAS coordinate with NGP and write OWS Reader with CIDA team for OL API

Transition to CSAS 

NGMDB Complete libraries for Flex to be hosted CSS Wide

Explore Writing OWS Reader for ESRI Framework (currently reading ESRI/OWF JSON)

Q4Transition deployment from NGP to CSASSupport Migration to CSAS for hostingDeploy all above Libraries in CSAS for CSS wide useDeploy OpenIn and SaveAs in SB CSAS  Transition deployment from NGP to CSAS NGMDB Migrate Libraries to CSAS to hosting

Appendices

Past Projects leadings up to this proposal

Benefits and Reasons

Examples

Context profile background:

In the early 2000s, OGC sponsored the Web Map Context Profile XML standard. The idea was have a standard configuration file with the initial view setup for basemaps, URLs for legends, metadata and web map, which maps were turned on by default, what other overlay WMSs are available in a pick list, what is the default title, order, transparency and grouping. If you made a map, turned layers on/off, changed order, transparency, zoomed, pan, you could then save your new "mashup" as its called now as an OGC Context Profile for later on that site, or if another site supported reading that standard, could just drop it into another viewer. Also, this is not just a viewer reuse, but also analytical processing re-use.

Great concept, but didn't take off. Only Geosptial One-Stop on a large scale produced the file. Since then, all viewers support the user to make their own context like OpenLayers, ESRI JS or Flex API, Google Maps, Bing. And most also are trending towards JSON (Javascript Object Notation) as the context profile language of choice. But most developers are choosing their own context profile standard. This is very typical for developers to do without a standard to refer to.

In 2008, NGA Palanterra x3 went this direction and nicely documented their configuration JSON. USGS The National Map followed suit using that and extending with download framework. ESRI as a corporation came out with a 229-page paper for the REST specification, noting a configuration schema, and was released defaultly with Flex, ESRI JS API, and ArcGIS.com. It is similar to the Px3 implementation, but there are differences. USGS asked OpenGeo Openlayers if they would be up for a JSON context profile standard, and they said they were interested in collaborating as did ESRI ArcGIS / Rest Spec Team.

In 2011, OGC has advanced the context even further by creating map and processing context - more powerful than the original. This is up for standard acceptance in September 2011. *Check out the OWS Context Schema Experiment

In 2012, OGC completed a draft standards in XML and accepted to move to adding a JSON format support in 2013. 

  • No labels