USGS has 100s of web applications and thick clients that were built to help explore the 100s of datasets and analytic scenarios at USGS. These viewers use many technologies and solve varying use cases from basic navigation, to feature investigation, data analytics, to geoprocessing. In 2009, scientists asked the USGS Community for Data Integration to make easier to port mashup work done in one environment to another viewer - initially, with a visualization focus. The USGS National Geospatial Program, which manages The National Map viewer application is currently powered by the NGA Palanterra x3 project. Palanterra x3 has been context based since 2009 using a custom JSON schema. Palanterra stated in July to USGS they will likely be moving to support the ESRI REST JSON context schema by end of calendar year.
That being said, in support of the USGS Community for Data Integration, The National Map is working on functionality to allow a user to save a context, and re-read in mashup sessions. This functionality will be its own application, which means the 100s of USGS Viewers could call this Save As function and allow the context to be opened in many viewer flavors that supports different context formats. The USGS Community for Data Integration already has a testbed project moving forward that is working on developing readers of the standard context for Palanterra X3, Google Maps, OpenLayers, the multiple ESRI APIs, etc. that would help grow the approach for developing viewers at USGS that allow for opening mashups created in other viewers.
The save context functionality the USGS is working on would like to support saving a mashup as KML, ESRI JSON and OWS and as well save snapshots as PDF, JPG, PNG. KML can be saved for opening basic mashup in KML clients like Google Maps, Google Earth, OpenLayers, The National Map Viewer, and NGA Palanterra x3. The ESRI flavors to support the scientists requests to easily open in the Arc products, of which USGS has 11,000 licenses. For OWS, we are asking the OGC to consider guiding the OWS XML/Atom to be also “published” as a transformation with specified rules into the hierarchical structure JSON format that is non-lossy. This lightweight option would be of benefit to viewers like Palanterra x3 which powers NGA, USGS, FBI, others and would be trivial to deploy once the Atom schema is known.
The main reasons for requesting JSON transformation support of the ATOM/XML for OWS are:
• JSON is much less verbose, less bytes which is important in service world
• JSON is native supported in all browsers, and thus no additional parsers needed. XML isn’t a datatype in browers as it must be parsed, though in Flex XML is a datatype. There are many good articles comparing XML versus JSON performance on the web, and though most parsing APIs can be tweaked to be just or near as fast in XML, most APIs are faster in JSON. Also, since JSON can be embedded as a javascript which means a configuration, when allowed via crossdomain.xml, can be called across approved site, where XML cannot without addtional code modules.
• Mixing content in JSON can be easier than XML world, and the current OWS will need several extensions to support more richer attributes that NGA Palanterra and USGS The National Map have lready extended – for instance, we have services in OGC and non-OGC flavors, transparency and service order settings, mixed in CSW settings, added a download configuration framework, processing context, etc. Much along the OWS lines, but not quite ready to move all that over to the new OWS.
• JSON makes for easier integration in commercial space as it appears to be favored for web apps
Of note, USGS completely recognizes that for the heavier communication and transaction side, XML/SOAP/WS- has its benefits in the enterprise, but the leaner, read-only lighter web applications like “viewers” tend to be easier to work with for user and developer end in JSON. In a nutshell, its not a open and shut situation on when to use XML/WS- vs REST/JSON , but the current Geo JS APIs do better in JSON due to speed, cross-site scripting can be circumvented using script tags for JSON reference (HTML and XML cannot), BUT XML has better controls/metadata and great in thicker clients with true parsing or TRUE Web apps, not the lightweight simple apps.
A team from CDI focused on making the process of creating, saving, and sharing mashups (aggregations of TNM web services for visualization) easier. Of particular focus is to ensure that these mashups are readily usable in widely deployed client tools (such as ArcMap and Google Earth). The progression has been:
Project work can be seen at: https://my.usgs.gov/confluence/display/cdi/Web+Application+Integration+Framework
This team is being pioneeered by USGS NGP (Rob Dollison, Matt Tricomi, Kevin Hope, Glenn Guempel),
Look at having a base web context (definition?) that can share between any USGS Viewer.
Project Topic | Tasks | Resources Required | Major Outcomes | Total Funding Needed | Temporary Abstract/Planning Text (Dump from Etherpad Brain Dump session) |
Expand TNM Save As/Open In to USGS Wide | Vet the TNM Concept Discover issues for expanding Explore JSON Context Scope Explore Standardization Process Explore how have 1 Save As/Open In App service that USGS can use leveraging TNM efforts | Representatives from other USGS areas interested in Mashup Travel $ may be needed | USGS Wide approach to Mashup Same Context File Demos of Open Mashup in other Viewers Preferrably OpenLayers, Flex, Silverlight and JS API Mashup reader developed for reuse | Travel $ resources from other programs for devel time may be needed | See Save As/Open In project in CDI Wiki under Tech Stack |
A web map context is a description of the initial view, basemaps, overlays, and other user-configured characteristics of a web viewer session. A context can organize things like WMS, WMTS, KML, RSS, and ESRI REST services. OGC has a Web Map Context (OWC) specification, which defines XML as the encoding scheme. This specification has not been widely adopted, at least in part because XML does not work well in Javascript APIs. The goal of saving a web map context is allowing users to create a mashup in one viewer that can be saved and opened another viewer. This gets us out of the Web 1.0 "go to my site" world, allowing mashup sharing.
The project, based in ongoing work by the USGS National Geospatial Program, is an attempt to define a common JSON encoding for web map contexts that could be used across many viewers - OpenLayers, Google Maps, ESRI JavaScript/Flex, and Palanterrax3, among others. Defining a profile of the OWC that allows for encoding in JSON format would be the ultimate goal, but not needed for this beta project. Developing consensus on a JSON map context is the primary goal.
The initial focus on this project has been focused on ESRI-based clients because these are an important part of the USGS corporate computer model. USGS scientists have requested that taking web-browser created views of The National Map be easier and faster to replicate in ArcMap. Specifically, after a scientist makes a mashup using the TNM Viewer (web-based<--Palanterra viewer?), it should be easier to download the mashup with a single mouse-click to the user's local machine, and use that in ArcMap. Additionally, saving a TNM Viewer mashup for use in Google Earth has been a focus. TNM has been using a map context model and JSON encoding scheme defined by the Palanterra x3 platform as a starting point.
Few things going on:
So the initial project goals were to to make it easy to save a context as JSON, as KML, as MXD. If the standard JSON changes, that is not a major issue. The JSON TNM is using predates the ESR REST Geoservices Web context in the ESRI REST geoservices web API. You can see the Px3 JSON at: http://viewer.nationalmap.gov/viewer/config/default/default.r0.json
The Px3 or ESRI context supports calling ESRI REST map services, IMS, WMS, KML, RSS and WMTS, but not WFS or WCS, but typically these are not used in map contexts in most situations.
The OGC context supports WMS,WCS,WFS, KML, GeoRSS, but not ESRI REST Map Services. Wouldn't be a big deal to make this an extension to support.
(as captured on OGC Twiki from 2/14 meeting 'OGC Login required to access Twiki - same documents copied below')
Configuring_Config_USGS_TNM.json.pdf: Data Dictionary on Px3 config and TNM example
config.json: Example of the Px3 JSON context as notes with NGA capture
http://viewer.nationalmap.gov/viewer/config/default/default.r0.json TNM's Configuration
Comparing_Px3_Config_USGS_TNM.json-to-OGC-OWS-draft-spec.pdf: Nov/Dec '12 Thread - USGS/NGA Px3 comparison of OWS context to px3 config
GeoServices_Context_Example.docx: This file I attached has a few examples of the context file that arcgis.com uses, which is the geoservices api base for sharing map mashups - which is the part that is "hidden" or at least not clear in the geoservices api - which hints at, to Rosinger's question, there may be overlap between the OWS Context effort and the GeoServices? eval. and REST OGC groups PS don't be alarmed by the page length - the first 3 pages are the instructions and 2 simple examples. The bulk of pages is a complex example that we made for adding all the USGS National Map overlays and some EPA facilities data
From Marten Hogeweg:
This is the WMC opener I just mentioned for ArcGIS Desktop.
https://github.com/Esri/geoportal-server/tree/master/components/desktop/WMCOpener
it is open source under Apache 2.0 license.
OGC is working as part of OWS-7 Testbed on an ATOM based XML. It will be very robust and complete. The concept of this being used as JSON though, is still relatively new for OGC, and in early phases of consideration. There might be room in OGC to join the discussion of an OWC draft specification that could be expressed in XML or JSON--which is perceived to add the "best of" what a json approach can offer.
If you have access to OGC portal the link is https://portal.opengeospatial.org/twiki/bin/view/OWSContextswg/
OGC accepted USGS proposal to manage 2 encoding: ATOM XML and JSON. They have done a draft on the JSON translation using an XSLT and the standard is being worked. But the field requirements discussion is still ongoing
ESRI Submitted REST API for consideration. Glenn Guempel has attached the OGC GeoServices REST Candidate Standard Version 1.0 in case you have not read what ESRI submitted. Doug is correct this is a hot topic in OGC. Doug noted:
The likelihood of this seems to be low because, per Doug Nebert:
Glenn Guempel attached the OGC GeoServices REST Candidate Standard Version 1.0 in case you have not read what ESRI submitted. Doug is correct this is a hot topic in OGC. Also attached are 1. NGA response to the and 2. The GWG Information transfer and Services Architecture focus groups slides from their last meeting where the REST Candidate Standard was discussed.
Essentially, the conversation of when to use ATOM XML or JSON gets close into a REST v SOAP argument which if you search the web on that term, you'll find plenty of articles on it:
Primers
More Info
In a nutshell, its not a open and shut situation on when to use XML/WS- vs REST/JSON , but the current Geo JS APIs do better in JSON due to speed, cross-site scripting can be circumvented using script tags for JSON reference (HTML and XML cannot), BUT XML has better controls/metadata and great in thicker clients with true parsing or TRUE Web apps, not the lightweight simple apps.
Doug Nebert wrote, "JSON is a hierarchical structure that can be transformed from XML/Atom, but is much less verbose. What the members could do is specify the rules for the transform - a Context JSON schema - and it becomes an alternative non-lossy format that can be exchanged. This lightweight option would be of benefit to viewers like Palanterra and would be trivial to deploy once the Atom schema is known.
This response was requested from the NGP EA team of a lead Palanterra x3 developer on why JSON was selected in 2008 when Px3 was re-architected. This is not a formal request. Just a few things from NGA Palanterra x3 team, and not a formal list, and was great that they offered some of there architecture direction.
Detailed response
=================
Since XML must be parsed and isn’t a datatype in browsers. However, JSON is supported in all browsers, Flex, Silverlight, and parsers can easily be written for languages such as Java that don’t have it as a first class citizen. In a browser based world the only language that supports XML well is Flex so far as there is actually a XML datatype.
In analyzing whether JSON is the right technological solution, I think there are a number of factors that lead to it being a better choice than XML:
1.) Browsers or Browser plugins are generally what I feel to be the main audience for this config and JSON is native and can be parsed by these w/o any additional parsers
2.) JSON schemes are less verbose than XML and thus less bytes over the wire, which is important in our service based world
3.) Mixing content in JSON is easier than stitching XML content together
3.) JSON is slightly easier to extend than XML esp for those with a lower technical knowledge base, because since namespaces don’t complicate things
4.) JSON seems to be now favored in the commercial space over XML for web applications (e.g. Twitter and Foursquare dropped support for xml in 2010) and I think the commercial space is always a good indicator of trying to make things leaner, faster, and more cost efficient.
There are tons of arguments on both sides where XML might provide more flexibility and has been favored in desktop and server communication within the enterprise for a long time but JSON is leaner, simpler and in Palanterra x3 developers opinion (not NGA specifically) is easier to work with across programming languages.
=================
Likely this thought pattern was some of the logic into why ArcGIS.com also selected
I would also add that JSON can be embedded as a javascript which means a configuration, when allowed via crossdomain.xml, can be called across approved site, where XML cannot without addtional code modules.
The USGS has 20 slots available in total. There's an optional $50 Wed night reception. Otherwise OGC events are free. Attendees
If you are part of NGP, please coordinate registration through Glenn Guempel (instead of using the link provided above).
Individuals are USGS unless otherwise noted.
Pedro Goncalves, Terradue (PG)
Matt Tricomi, USGS (MT)
David Rosinger, Intergraph (DR)
Dave Wesloh, NGA (DW)
This ad hoc meeting was held to discuss possible solutions for developing a JSON encoding of OWS Context. Specifically called to discuss methods available for converting the ATOM encoding to JSON.
PG – If we use an XSL conversion method there is an issue related to doing representing GeoRSS? in GeoJSON? .
MT – Are we talking about using GeoJSON? instead of JSON?
DW – I think OGC would prefer we use GeoJSON? .
PG – I have tested an Atom with GeoRSS? Context and it draws correctly in Google Maps. GeoJSON? is not completely supported by all the services.
MT – USGS is working on a “save as” function for translating from one encoding to another.
PG – I will have an Atom and JSON encoding ready for our next meeting. Is there a preference in USGS for ATOM or JSON.
MT – I believe NGA has a preference for JSON.
Action: Dave Wesloh to contact Palenterra team to determine if they have a preference for ATOM, JSON or GeoJSON? . Action Closed – see response below.
MT – I have been told a lot of implementers do not like having to use an XML parser for ATOM in a web browser (thin client). NGA (Palenterra) has been working with Esri and they might be looking at implementing the Esri Context instead of OWS Context. (see email sent Feb. 16, 2012 – Geoservices data context)
PG – I have been looking at using a OpenLayers? API to convert ATOM to JSON to GeoJSON? . Look at GeoJSON? .org. My current process is to take ATOM translate to JSON add geometry elements and output as GeoJSON? . Esri uses a different approach.
Action: Dave Wesloh to contact Palenterra team to determine if they have plan to implement the Esri Geoservices REST Context document or the OGC OWS Context document.
Action Closed:
Question: We were wondering if Palenterra has a preference for JSON over the use of GeoJSON? ? Our thoughts were to create an ATOM encoding translate to JSON add the geometry elements and output to GeoJSON? using an OpenLayers? API. Our second question is if the Palenterra team intends to implement the Esri solution or if there was some thought to using the OGC OWS Context solution when it becomes available.
Response (Palenterra): It isn't ESRI-corporate that is doing this (developing a Esri based Context for Palenterra) – it is the GVS/contractor that has done this back in 2008 timeframe. It is just a JSON-based file we use as a configuration file. The Config file only sets the profile with appropriate layers, links, etc based on a user's profile and their association with a group – so very similar to the context, except not XML based.
ESRI has a preference for ESRI-JSON as in the GeoREST? . GeoJSON? is slightly different and JSON is even a little bit different – neither are out of the question. I believe this JSON file is not a GeoJSON? or ESRI-JSON – it is just a JSON that sets preferences and profiles; and doesn't really present any geographic information. This config was started back in 2008 and just carried forward since – it made sense in 2008 as nothing else was really available.
I believe we would be completely open to implement the OGC one – especially if it isn't the current XML based context file.
PG – ATOM is fully supported in Google Maps, Firefox, Bing already. JSON is a good option if you have full control of the entire process (create/use/share) but it’s so flexible this could lead to interoperability issues if the Context documents are not created exactly the same.
DW – How does OpenLayers? fit in this discussion?
PG – OpenLayers? is an open source library. The idea is that we would create the translations from one encoding to the other then add those into the OpenLayers? library and ask for them to test and approve.
DR- If Esri is developing their own version of Context document do you think they’d be willing to develop a translation between the Esri Context and the OWS Context? Sort of like the USGS “save as” option.
DR – So is our approach to use XSLT to translate between ATOM and JSON?
PG – Preference is to use OpenLayers? and build a translation to put in the library.
(as captured on OGC Twiki from 2/14 meeting 'OGC Login required to access Twiki - same documents copied below')
Configuring_Config_USGS_TNM.json.pdf: Data Dictionary on Px3 config and TNM example
config.json: Example of the Px3 JSON context as notes with NGA capture
http://viewer.nationalmap.gov/viewer/config/default/default.r0.json TNM's Configuration
Comparing_Px3_Config_USGS_TNM.json-to-OGC-OWS-draft-spec.pdf: Nov/Dec '12 Thread - USGS/NGA Px3 comparison of OWS context to px3 config