FY13 Proposal: Deploy Save As and Open In USGS Wide

Cover Sheet Title: Deploy Map Save As and Open In Application Service USGS Wide

CDI SSF Category : Computational tools and services

Applicants/Principal Investigator(s) : Glenn Guempel, USGS NGP Standards Architect; Rob Dollison, USGS NGP Delivery Services Lead

Abstract - At the first 2009 CDI workshop, scientists stated one of the top three goals was to make it very simple to access corporate databases. For example, taking a map session mashup they make in web views and move it easily into their advanced thick clients like ArcMap and Google Earth. It could take a scientist, planner, user, etc. a half-hour or more to create these unique views - zoom in, re-order services, change opacity, add services, turn on layers, etc. - and to have to recreate that in their environment, which could be one of the large reasons why services are not taking off - its too hard. The National Geospatial Program took lead and developed a capability to save the map session mashup setup or "context" in one-click in a standard map context then open it in web viewer APIs and popular thick clients like ArcMap and Google Earth. NGP has worked over 3 CDI years collaborating with CIDA, WiM, NEIC, CSAS, and EGSC on advancing this concept to work not just in The National Map applications and services, but the any web services and applications that can support saving as multiple standards and opening with a focus on being able to use this in any web viewer application in USGS. This year's proposal is to focus initial deployment of current SaveAs and OpenIn services and support continuous improvement on standardizing the context, hooking into OpenLayers and ESRI APIs.

Summary

The summary will repeat but provide more details about the information provided on the cover sheet including the following:

Introduction and background

At the first 2009 CDI workshop, scientists stated one of the top three goals was to make it very simple to access corporate databases. In 2009, scientists and users made it clear to CDI - Make these setup mashup steps go away - 1-click.  Our current viewing and app portfolio across USGS, DOI, and generally industry-wide does not meet this need. We've had many problems to solve such as we have 100s of viewer instances in 10s of flavors, there still isn’t a good standard for this, and we have yet to establish business case for programs and developers to move towards new approach. 

This year's proposal is to focus initial deployment of current SaveAs and OpenIn services and support continuous improvement on standardizing the context, hooking into OpenLayers and ESRI APIs.

• CDI SSF Category : Computational tools and services
• Project Title : Deploy Save As and Open In USGS Wide

Project Leaders or Principal Investigators

All Personnel Involved

Collaborating Organizations

Other Relevant Personnel or points of contact

Detailed description of geographic/geologic/ecosystem/habitat/taxonomic/other context of the project and its importance or value if applicable.

This project enables simplifying map service integration. The USGS provides many national, regional and local datasets for download, streaming interaction such as WFS/WCS, and analysis. Ultimately, most datasets are presented for visualization in "viewers" with basic navigation and interaction for inspection and even lightweight WebGIS like web service functions, annotations, etc. Many viewers – different APIs, clients, purposes, and niche functions – are invested in at USGS and DOI and the whole Federal Government. This project has the capability to save map contexts of the mashup a user makes in a viewer. Any context requiring Spatial Visualization (i.e. Geographic, Geologic, Water, Ecosystem, Habitat, Topographic, etc.)

Scope

Scientists, Analysts, etc. use many varying science and map tools with viewing capabilities to be used while working different parts of a problem. They’ll move from one tool to another: Finding - Use TNM to download foundational data, Discovering - Sciencebase to find thematic data, Processing - CIDA GDP for processing data, Analyzing - ArcMap for in-depth study, etc. For example, taking a map session mashup they make in web views and move it easily into their advanced thick clients like ArcMap and Google Earth. It could take a scientist, planner, user, etc. a half-hour or more to create these unique views - zoom in, re-order services, change opacity, add services, turn on layers, etc. - and to have to recreate that in their environment, which could be one of the large reasons why services are not taking off - its too hard.  Meaning, each time they switch tools, they need to re-setup their session or “mash-up” which is not their priority and is a time-waster or nuisance

The solution is not "1 viewer" or "1 viewer API". We are stuck in a multiple viewer environment, we could recommend a few APIs, and restrict others at best. The problem with this is that when someone goes to a new viewer, they don't always have the same backdrop basemaps, common list of overlays, and most viewers don't have capability to add services into the viewer, except thick clients and The National Map. Essentially there is no way for a user to jump easily between viewers. The output of this project is to implement a web service that any viewer API can call that allows a user to save the last view of the multiple data map services. By having this capability become USGS/DOI wide, or even integrated late into the Geospatial platform, this should look to increase web service use by making easier to use between different web viewers and thick clients.

The goals this year by August 31, 2013 are to:

The CDI Funding required would be for 30% of the effort for the non-NGTOC/NGP items. More details on the technical approach, level of effort commitment by organization, and quarterly timeline are provided below.

Plan

FY13 SaveAs-OpenIn Scope and Plan for OGC Beta Test

Technical Approach

• User Stories - Support these user stories
• Geospatial - Save As Framework - Develop an application service that can take the output of a session context from a viewer in a specific standard, then transform and export it to multiple formats. It was first developed directly into The National Map Palanterra x3 Viewer based on ESRI JS supporting multiple target formats (KML, Px3 JSON, and PDF/Images). It has since been refactored as a standalone application. Now, the focus is on going live with existing transforms, finding a hosting location, complete remaining transforms, and explore new context standards with OGC via testbed participation and evaluation of 2 standards: New OGC OWS XML conversion to JSON, and the ESRI JSON adoption to OGC. One thing that shouldnt go unmentioned is that opening a context file not located on the same domain requires additional cross-domain and script/SQL injection validation controls on top of schema validation. This "OpenIn" functionality also was developed as part of the framework which its code, for javascript libraries, an be leveraged for use in other viewers.
• Architecture, Design, Coordination -  In 2010, the team architected a Target Application Architecture, Target Info Flow, and overall Architecture Diagram that is based on a common context and standalone application web service. The role is to support coordinating design requirements and promoting this concept of operations across the multiple Viewer APIs, security requirements, and use of the web service. As well, help progress the OGC standards towards one that can be industry adopted and adopted across USGS/DOI in JSON to be used by popular Viewer APIs.
• Viewer Library API, SaveAs, and OpenIn Migration to SB/CSAS -  Since 2011, its been discussed with the Sciencebase team to be able to save, catalog and then open contexts in different viewers. In 2013, we are proposing at minimum to evaluate what is needed to get the application web service in the same Sciencebase hosting environment to go live with save as in permanently, and decide the length of time that it may need to continue in interim reside on NGP servers.
• Open In OpenLayers Framework (Read Px3 JSON, ESRI JSON -  In 2011, we developed the capability to read ESRI REST Web services in Openlayers. That code was re-used in 2012, and taken a step further to be called via the Px3 context standard. Now, the approach is to complete the context integration in Openlayers with all standard function calls as well as offer up this new framework to be used and maintained by the ScienceBase Team. As well, we should look to continue to coordinate with another 2012 CDI project that researched Openlayers HTML5 library integration. Example of work still to be done: 
  Read in some missing session input (Basemap Buttons, layers, zoom), Debug some ESRI REST Service compatibility, Incorporate with ERSG OpenLayers work, and discover ways to make this part of standard OpenLayers Library
• Save to ArcMap (10 days) -   Early on focused on enabling the initial problem through manual solutions (i.e. an MXD with common service palette), and in 2011, NGP and NEIC researched with ESRI Architects multiple methods of how to move from web context into an ArcMap MXD file. Through discussions, and updates from ESRI in 2012, we now have the code that works in ArcGIS Server 10.1 that will create an MXD file for the user to save via ArcPy libraries. When a AGS 10.1 environment is made available, we are suggesting to go live with this transform target format.  The Save As Function needs to simply call a 15 line python script to output an MXD file, but to do it needs to be running 10.1 server side, but the resulting MXD can be opened in ArcMap 9.x and up. ArcMap still slow & “bloated” to open many services due to how they make lyr files, but it works – which may be an eGIS type enhancement request (make services usable in ArcMap).
• Open in Flex (Convert Px3 JSON to ESRI JSON) -  In 2011, ESRI came out with a REST Standard offered to the Open Web Framework, and later offered it to OGC for consideration. Since 2011, ESRI has made all Viewer APIs, and ArcGIS.com completely integrated with this format. We would like to coordinate as part of an OGC sponsored testbed a second standard JSON context format and tracking the ESRI/REST OGC SWG progress towards offering a 2nd alternative option. As well, this task would produce the capability to call the Save As function and transform the REST/ESRI JSON to other targets. Likely this will require more code refactoring by NGTOC to the final architecture factoring levels originally required 

Active Discussion Threads Discussion on JSON and Security implementation - Context Security - Good threads on how we handle cross-domain JSON calls (client side or server side processing is the question) Standard JSON Context - Context Standard - OGC has drafted the new OWS context in JSON encoding - now onto seeing the use cases and requirements come through. Background Architecture Target Application Architecture Target Info Flow Architecture Diagram

Project Experience

We will not be submitting bios, profiles, or CVs for this project.

Commitment to Effort

Budget

Please provide a detailed budget for the project using Table 3. Include at least the following categories of information, separating the CDI funds from the in-kind match:

1. Salaries and Wages (see also Budgetary Considerations)
2. Fringe Benefits
3. Field Expenses
4. Other Direct Cost Line Items
5. Total Direct Charges
6. Indirect Charges (Overhead)

1. SALARIES (inc. number of hours and hourly rate):

Included in official Proposal document

2. FRINGE BENEFITS:

Included in official Proposal document

3. TRAVEL EXPENSES*:

Included in official Proposal document

4. OTHER DIRECT COSTS: (itemize)