July 8, 2020: Field data collection apps & Knowing what your USGS data are doing
The Community for Data Integration (CDI) meetings are held the 2nd Wednesday of each month from 11:00 a.m. to 12:30 p.m. Eastern Time.
Meeting Recording and Slides
Recordings and slides are available to CDI Members approximately 24 hours after the completion of the meeting.
These are the publicly available meeting resources. Log in to view all the meeting resources. If you would like to become a member of CDI, join at https://listserv.usgs.gov/mailman/listinfo/cdi-all.
Agenda (in Eastern time)
11:00 am Welcome and Opening Announcements - Kevin T. Gallagher and Tim Quinn
11:15 am Collaboration Area Announcements
11:25 am It's July 2020: Do you know what your data are doing? SAS Science Data Management: Contributing to USGS progress in management of scientific data - Viv Hutchison, USGS
11:55 am Field Data Collection using NGA's free and open source Mobile Awareness GEOINT Environment (MAGE) and MapCache Mobile Apps - Ben Foster and Justin Connelly, National Geospatial-Intelligence Agency
12:30 pm Adjourn
Title: It's July 2020: Do you know what your data are doing? SAS Science Data Management: Contributing to USGS progress in management of scientific data
The Science Analytics & Synthesis Science Data Management Branch (SDM) provides Bureau-wide leadership to optimize and share USGS science data management practices and workflows. SDM designs, maintains, and provides customer service for bureau enterprise tools and services that align with the USGS Science Data Lifecycle - to help ensure scientific data are fully described, preserved, and publicly accessible. We build community around the tools and best practices to maximize effectiveness and promote usability. What are your data doing in 2020? We’ll look at ways SDM is providing tools and metrics to find out.
Viv Hutchison is the Science Data Management Branch Chief for the Science Analytics and Synthesis organization in the Core Science Systems Mission Area. Her team promotes science data management best practices, leads the Community for Data Integration, and builds enterprise tools and services, such as ScienceBase, Digital Object Identifier Tool, Science Data Catalog, and the USGS Data Management Website, designed to increase scientists' efficiency in making data publicly accessible. Viv serves on various committees such as the Federal Data Strategy Working Group, the DataCite Board, the USGS Fundamental Science Practices Advisory Council (FSPAC), the FSPAC Data Guidance SubCommittee, and the CDI Data Management Working Group. She holds a Masters of Library and Information Science from the University of Maryland-College Park.
Title: Field Data Collection using NGA's free and open source Mobile Awareness GEOINT Environment (MAGE) and MapCache Mobile Apps
NGA's (National Geospatial-Intelligence Agency's) Mobile Awareness GEOINT Environment (MAGE) is a free and open source mobile and web browser app developed and maintained by NGA to enable field data collection teams to easily capture and share geotagged field observation notes, photos, videos, and audio recordings. MAGE also provides field teams with an option to enable real-time tracking of teammates to enhance security and safety during humanitarian assistance and disaster response events. NGA's MapCache mobile app further enhances MAGE by giving mobile users with the ability to download datasets from any OGC web map or tile service (including the USGS National Map) into a standalone OGC geopackage dataset for use as a custom basemap within MAGE for mission specific geospatial context such as topographic maps and nautical charts that can can also be used for navigation and reference using a GPS enabled mobile device without any network connection.
Discovery Channel Youtube Feature showing FEMA using MAGE in Puerto Rico: https://youtu.be/-mxO4kDlfpI
MAGE R&D Program with NGA, DHS, and the Joint Interagency Task Force South in Key West: https://www.dhs.gov/science-and-technology/news/2018/07/31/snapshot-intergovernmental-cooperation-enhances-comms
USGIF GEOINT Symposium 2019 Presentation on MAGE featuring U.S. Army Civil Affairs and USDA: https://vimeo.com/340078200
NGA's Protected Internet Exchange (PiX): https://www.pixtoday.net (use government email address for account registration) Once you have an account on PiX, send an email to email@example.com and request to be added to the "MAGE USGS CDI" event.
Ben Foster is NGA's GEOINT Services Capabilities Division (TDG) Technical Executive at NGA's Campus West in St. Louis, MO and is responsible for leading NGA's mobile capability product line and architecting technical GEOINT solutions and services. Prior to his current role, he has served as Product Manager for NGA's GEOINT App Store where he was responsible for leading development and operations for NGA's app distribution platform. Ben has also served in multiple NGA assignments as an imagery and geospatial analyst with experience applying a variety of advanced national and commercial GEOINT sensors. Prior to NGA, Ben served in the United States Army as an All-Source Intelligence Analyst. Ben holds a Bachelor of Arts in International Relations and an Associate of Science in Intelligence Operations from the American Military University. He has also received an Executive Certification in Product Management from UC Berkeley. firstname.lastname@example.org
Justin Connelly is a Science and Technology Programmatics Officer from NGA's Research Directorate currently stationed at Naval Air Station Key West and is responsible for managing joint research and development for NGA's MAGE program. Justin also serves as a GEOINT R&D advisor to the Joint Interagency Task Force South (JIATF-South), a joint IC, DoD, and DHS organization responsible for the detection a monitoring of illicit trafficking activity throughout Central and South America. Prior to his current role, Justin was the Program Manager of NGA's crowdsourced mapping program, NSG Open Mapping Enclave (NOME), and has also served in multiple NGA project management and geospatial analysis positions supporting the National Security Agency and DoD Joint Staff. Prior to NGA, Justin served in the United States Army as an All-Source Intelligence Analyst. Justin holds a Bachelor of Science in Anthropology/Geography from Oregon State University. email@example.com,
- API info page: https://my.usgs.gov/confluence/x/RhLlJw
- USGS Model Catalog Documentation: Model Catalog Documentation
- Sign up for updates on the USGS Model Catalog: https://listserv.usgs.gov/mailman/listinfo/cdi-models
- Leslie presented a word cloud of staff profile expertise keywords of CDI coordinators and volunteers, finding 31 profiles and 115 keywords. We're starting a CDI Staff Profile Study to help fill out these profiles.
- Last meeting, we put out a call for more information on APIs. These are the results: https://my.usgs.gov/confluence/x/RhLlJw
- Kevin Gallagher spoke about the upcoming CDI request for proposals, coming up in the fall. See more on funded projects in the latest report: https://doi.org/10.3133/ofr20201062
- A new effort currently in development is the USGS Model Catalog. More information here
- Tim Quinn spoke about EarthMap and Integrated modeling. Join the Integrated Modeling Webinars MS Team here: https://teams.microsoft.com/l/team/19%3a13edd23778df446793821d800ef0e68f%40thread.skype/conversations?groupId=c90313c2-2447-456a-b1c7-ce8c8a8f2f61&tenantId=0693b5ba-4b18-4d7b-9341-f32f400a549
Collaboration Area Announcements
See slides for all collaboration area announcements.
- UAS presentation tomorrow, livestream:
- Open Innovation
To receive Open Innovation Community meeting invitations, Subscribe to the CDI-OI Listserv
Open Innovation for Floods (July 1, 2020) - Flood-related open innovation resources available on wikipage and video coming soon
Tackling the Paperwork Reduction Act (PRA) in the Age of Social Media and Web-based interactive Technology (June 18)
Indigenous Observation Network (ION): Community-Based Water Quality Monitoring Project (June 19)
FedCCS June Meeting “NOAA Citizen Science on Data Quality” and “Eterna OpenVaccine for COVID-19” (June 25)
To receive Federal Crowdsourcing and Citizen Science (FedCCS) Community meetings and recordings, join the listserv by sending an email to: FCPCCSfirstname.lastname@example.org
July 2020 Open Innovation Newsletter (Coming Soon)
It's July 2020: Do you know what your data are doing? SAS Science Data Management: Contributing to USGS progress in management of scientific data - Viv Hutchison, USGS
- Why manage data?
- increases reproducibility and integrity for Earth science data
- important that data is FAIR (findable, accessible, interoperable, and reusable)
- SDM leverages tools, expertise, and encourage community engagement
- ScienceBase data release
- ScienceBase became a trusted digital repository (TDR) in 2017
- The ScienceBase Data Release (SBDR) Tool connects to other tools like DOI Tool, IPDS, and the Science Data Catalog (SDC). The SBDR Tool can be customized to reflect a science center's specific workflow.
- Many journals require that data accompanying an article is made publicly available - ScienceBase is an easy way to do that.
- 92 centers use SB for data release.
- Steady increase in usage; 1,000 releases per year
- So, what are your data doing?
- Linking publications and supporting data
- SDM worked with PubsWarehouse to collect information on known primary publications.
- The related publication's DOI is then added to SB page and DOI Tool
- Useful for knowing how others use your data to show impact.
- Team uses xdd (preiviously GeoDeepDive) to track references to USGS DOIs
- Would like to display these citations on ScienceBase landing pages in the future
- Impact of scientific data
- Data for a Geospatial Fabric data release has been cited by 17 publications
- Data reuse is happening and is having an impact on further science
- A data release on the global distribution of minerals was cited in public policy.
- State of the data report
- Report to determine how mature and FAIR ScienceBase data is
- Random sample of 165 data releases; goal is to document a methodology for data release and see how scalable it can be
- Connecting enterprise tools
- USGS data tools - python wrapper around a set of system APIs. Creates a bridge between systems (DOI Tool, Pubs Warehouse, BASIS plus, Metadata Parser, SBDR).
- SBDR summary dashboard
- Data release metrics by science center, program, region, mission area
- Connecting tools
- IPDS can auto fill info to SBDR tool (connecting tools)
- USGS Model Catalog
- Goals are to increase discovery and awareness of scientific models and link models to related literature, code, data and other resources.
- Catalog will dynamically compile latest information for models
- CDI is gathering input from modelers across the bureau
- Your data are contributing to USGS successes with open and fair data
- Citations to your data are being tracked
- Data are connected between ScienceBase and your publication in Pubs Warehouse
- Your data are accessible to the scientific community!
Field Data Collection using NGA's free and open source Mobile Awareness GEOINT Environment (MAGE) and MapCache Mobile Apps - Ben Foster and Justin Connelly, National Geospatial-Intelligence Agency
- Focusing on open source applications to reach a wide audience.
- GitHub repository here: http://ngageoint.github.io/MAGE/
- See recording for live demonstration.
- The mobile application allows users to upload geo-located video and photo observations.
- The mobile app also allows creation of lines and polygons.
- Information added will be visible to other team members on the app.
- Web application has a similar interface, but with more robust features.
To join the CDI Event on MAGE for govt employees: (1) Request an account from NGA's Protected Internet Exchange (PiX): https://www.pixtoday.net (use government email address for account registration) Once you have an account on PiX, send an email to email@example.com and request to be added to the "MAGE USGS CDI" event.
- Is software also considered in "data releases"? Does ScienceBase include any software and are they open source and connected to GitHub? Would this help meet the Federal Source Code Policy for ensuring 20% of our software at DOI is open source?
- If authors have a little script that is only applicable to a data release, it can go in ScienceBase. If the code is something that needs to be maintained, it's generally released on code.usgs.gov.
- Drew Ignizio: a static file containing code can be attached to a landing page. Anything that requires management (a true software product) will make more sense to put on code.usgs.gov
Viv, will the SB Data Release dashboard show both pending and public data releases by cost center?
- Currently, we're focusing on public data releases. There was some concern about showing in-progress data releases to people outside of that releases' science center, since the dashboard will be open to all. Currently, the dashboard shows the number of in-progress data releases, but only details for public data releases.
would you please confirm if the link populating ScienceBase from IPDS already active or still in development?
- Drew Ignizio and Madison Langseth: Autofill from IPDS' is in place now for the ScienceBase Data Release form. You will need to be on VPN or the USGS network to access this feature.
- I'm curious to know what are the various formats of models that exist at USGS. Are they software, images, etc.? How does the format of these models enable them to be more FAIR?
- Leslie: regarding your model question, currently there are all sorts of model formats, and how people think about model (equations in a pub vs. code in a repo vs. conceptual model). We've started a list of different types that we would like the community to input on: https://my.usgs.gov/confluence/display/cdi/Model+Catalog+Definitions
Is code.usgs.gov connected to code.gov?
- Brandon Serna: [9:57 AM] Serna, Brandon S
Hey Liu, Sophia iirc our released software inventory is going to code.gov - there's an inventory listing here (https://code.usgs.gov/software-release/inventory/raw/master/code.json)
Official Source Code Archive
Justin / Ben NGA - any thoughts on how MAGE would be used or integrated into AGOL?
- MAGE can use any Open Geospatial Consortium (OGC) Web Map Service (WMS) or Tile Map Service (TMS) for basemaps or overlays in the MAGE web browser, android, or iOS mobile apps. For more information about publishing OGC services from ArcGIS Online, please see the following reference: https://doc.arcgis.com/en/arcgis-online/reference/ogc.htm
so, this is analogous to ESRI Collector?
- MAGE provides users with a different experience from other field GIS data collection applications. MAGE is focused on a collaborative team-based situational awareness where users within a MAGE event are sharing their current location as a dynamic map icon. When a MAGE mobile user creates a new geographic feature using a point, line, or polygon with customized data fields and multimedia attachments, all users within that event receive a notification and will see that new feature created on the event map within the mobile app. MAGE users can also view the contact information for other mobile users in the event and can simply tap the user's email or phone number to instantly contact that user from their mobile device during an event. This MAGE user experience and real-time collaboration feature is ideal for disaster response and humanitarian assistance tasks where information is very time sensitive and needs to be accessible to everyone within an organization during an event, crisis, or field data collection task.
What's the URL to the GitHub repo?
Justin/Ben - are there locations where this tool does not function well, such as under deep veg canopy, or canyons?
- Benjamin Foster: It will still operate completely disconnected from any network - and sync to the server as soon as a connection occurs. So no - works globally in any environment.
is it iOS only, or android too?
- Android as well
- MAGE is developed as a native android and native iOS app to give both versions of MAGE a familiar and intuitive user experience for users of each mobile platform.
can we add our own layers to the app before going out?
- Yes, you can add OGC Web Map Services, Tile Map Services, KML overlays, or OGC Geopackage datasets. Map and Tile services require a network connection on the device using the MAGE Mobile App. Data in OGC Geopackage format can be hosted on the MAGE server for download or opened directly by the MAGE mobile app and will be stored locally on the device for access without a network connection.
- Similar to what others are asking, what are the benefits of using MAGE vs Esri's Survey123 or Collector? Are there benefits of having it on our own system and not on a vendor's cloud system since they often are on arcgis.com links. Does MAGE allow more customization as opposed to using something like Survey123/Collector? Why is it important to support open source?
- To expand on the answer to question 2, MAGE provides a unique experience that combines elements from multiple field GIS data collection apps into a single app. Users have the ability to collect geospatial feature data in point, line, and polygon with attributes using custom forms that have the same dynamic options of any web survey form including text, checkbox, date, email, location, number, radio buttons, select, text area, and user select. Attributes selected in the form field can also control the map symbol or style information of the line or polygon geometry.
- All of the components required to run a MAGE system including a server with connection to mobile app users is completely free and open source and does not require any software licensing costs or subscriptions. Eliminating licensing costs enables a very significant cost savings when scaling an enterprise GIS capability to a large organization since many commercial GIS server products are licensed based on how many transactions, users, or servers are used. Instructions for deploying your own MAGE server can be found on our github page: https://github.com/ngageoint/mage
- NGA's open source software business model enables us to openly collaborate with government, industry, academic, and public partners and maximizes the return of our resource investments into capabilities like MAGE, MapCache, and Geopackage capabilities. Developing and using open source capabilities also allow us (and our mission partners) to have direct access to all source code to validate and ensure our applications meet information security and accuracy standards required for mission critical applications and services.
Can you export the data to a Geonode w/o losing attributes?
Hi Ben/Justin, I am interested in learning more about how you had planned for the user experience of the app. For example, what types of research did you do? Also, did you perform usability testing with your users during the development cycle? If yes, what types of techniques did you use? Thanks.
- Billy: started from ground-up, sitting with users. Some users came from apps like Survey123 - got to hear from users about what they liked and didn't like about those apps. Got user feedback while building the app.
- Google and Apple have resources for creating mobile apps and web apps (where your thumb reaches on a screen, where buttons should go). Apps are built in native environments - Android and iOS. Usability is different on these platforms as users expect different things coming from an Android or iOS background.
Can MAGE be a good use case for crowdsourcing information during disasters or any event from trusted users in your organization and others from the "public"? Can it be a good way to target certain crowds that you trust and already verify but still accessible outside of one's organization?
- MAGE supports hosting multiple events within a single server. Each MAGE event can be customized with specific basemaps, data collection forms, and user groups. Features created in MAGE also contain user attribution similar to how OpenStreetMap works. MAGE also allows creating teams of users that could represent trusted internal users and public users. When a new event is created on the MAGE server, you can simply assign teams to that event and any user within that team will have the ability to join the event and create observations an feature data.
Is MAGE connected to NOME (NGA Open Mapping Enclave)?
- The MAGE and NOME Programs at NGA work together to ensure data collected in each platform is interoperable. MAGE can use data exported from NOME using the OGC GeoPackage format and provides users with either a basemap or overlay of NOME data stored locally on the MAGE mobile app to enable offline use. NOME can also import geographic features including point, line, and polygon geometries with feature attributes from MAGE.
Im not clear on the difference between the two apps
- NGA’s MapCache mobile app is a stand-alone app that does not require a server and provides users with the ability to download maps directly from Web Map Service or Tile Map Services and create point, line, and polygon features which are stored locally on the mobile device using OGC Geopackage format. When MAGE users in the field need to quickly download a basemap for offline use, they can simply open MapCache, zoom to the area of interest, enter the URL of the map service, enter the min and max zoom levels and download the map. Once the map is downloaded to a mobile device, MAGE will automatically recognize the dataset and can directly open from your phone for use as a basemap.
- MapCache can also be used by itself as a map reference and navigational aid and displays your current location on the map using the GPS on your mobile device. The advantage MapCache has over Google and Apple Maps is that you can add your own map services like the USGS Topographic or Shaded Relief Maps.
Obvious application to USGS observations of discrete (PCFF) and continuous (SVMAQ) environmental variables (NWIS modernization)
MapCache allows you to create offline map tiles from any tile server - which you can then send to MAGE so you have a custom map background outside of Apple or Google
"MAGE is not hosted or managed by NGA, in order to use it you will need to get access to your agency or managing entities server." -> can someone from the public host a server or is this just a way for us to collaborate with other entities after we push a server instance up?
- If you are a DoD or a U.S. Federal Government user, you can request access to an NGA hosted MAGE server through our Protected Internet Exchange (PiX) at the following site: https://www.pixtoday.net
- You can also access everything needed to deploy a new MAGE server at our gitub site: https://github.com/ngageoint/MAGE
- If you need assistance or have technical questions about deploying a MAGE server, you can contact our team at firstname.lastname@example.org
Can we download the MAGE app already? (I missed that part)
- Ben: yes... both Google Play and Apple App Store... just search MAGE by NGA and MapCache by NGA
It's July 2020: Do you know what your data are doing? SAS Science Data Management: Contributing to USGS progress in management of scientific data
Field Data Collection using NGA's free and open source Mobile Awareness GEOINT Environment (MAGE) and MapCache Mobile Apps
Viv Hutchison, USGS
Justin Connelly and Ben Foster, National Geospatial-Intelligence Agency