Confluence Retirement

In an effort to consolidate USGS hosted Wikis, myUSGS’ Confluence service is scheduled for retirement on January 27th, 2023. The official USGS Wiki and collaboration space is now SharePoint. Please migrate existing spaces and content to the SharePoint platform and remove it from Confluence at your earliest convenience. If you need any additional information or have any concerns about this change, please contact myusgs@usgs.gov. Thank you for your prompt attention to this matter.
Skip to end of metadata
Go to start of metadata

January 13, 2021: Usability Collaboration Area Spotlight, Advanced Scientific Computing in the USGS, including Rescale, and CDI Pop-Up Lab: Collecting and serving imagery data online, including the NWIS Imagery Data Storage and Persistence Project

The Community for Data Integration (CDI) meetings are held the 2nd Wednesday of each month from 11:00 a.m. to 12:30 p.m. Eastern Time.


Meeting Recording and Slides

Recordings and slides are available to CDI Members approximately 24 hours after the completion of the meeting.

Log in to view the meeting resources. If you would like to become a member of CDI, join at https://listserv.usgs.gov/mailman/listinfo/cdi-all.

Agenda (in Eastern time)

11:00 am Welcome and Opening Announcements

11:15 am Group Announcements

11:25 am  Usability Collaboration Area Spotlight - Sophie Hou, contractor to USGS

11:45 am Advanced Scientific Computing in the USGS, including Rescale - Courtney Neu and Janice Gordon, USGS

12:05 pm CDI Pop-Up Lab: Collecting and serving imagery data online, including the NWIS Imagery Data Storage and Persistence Project - Frank Engel, USGS

12:30 pm  Adjourn

Abstracts

Usability Collaboration Area Spotlight

One of the newer CDI Collaboration Areas, the Usability group focuses on helping USGS tools and services be useful and their users be successful. In this spotlight, we will dig into the questions: What does usability mean? What can I apply usability concepts to? What were some highlights from the group last year? What are some things that can often be easily changed for greater usability? What is the role of usability experts in the USGS? 

Sophie Hou is a Data & Usability Analyst, and since she is currently a contractor to the USGS, she is enjoying the opportunity of supporting both the Science Data Management Branch and the Western Geographic Science Center at the same time while working remotely from Southern California. Sophie has worked on both physical and digital systems, and through her work, she has seen how interactions shape experiences and influence knowledge. She is interested in helping diverse communities to be more successful with their goals. This is why Sophie is passionate about usability.

Advanced Scientific Computing in the USGS

Advanced scientific computing is an umbrella term used to describe various architectures and computing methods designed to support data and computationally intensive scientific modeling workflows. This presentation will review platforms and resources available to researchers to support advanced scientific computing in the USGS. There will be a deep dive into the newest platform, Rescale, as well as a demonstration. Rescale is designed for computationally intensive scientific modeling and simulation, analytics, and machine learning workflows and makes high performance computing (HPC) in the Cloud easier.

Courtney Neu is a Service Management Specialist for USGS Cloud Hosting Solutions (CHS).

Janice Gordon is the USGS Advanced Research Computing Team Lead.

Highlights

  1. There will be four online data Carpentries trainings in 2021. Helpers are needed; email cdi@usgs.gov to assist. You can also apply to be an instructor here.
  2. The deadline for nominations for the FY2021 Leadership and Innovation Award is February 15th, 2021. More information here.
  3. The CDI Workshop will be held online from May 25-28. The theme will be "Designing Data-Intensive Science with CDI". Submit a session proposal here.
  4. The Geomorphology group is being revived. Sign up for the mailing list here.
  5. The proposal deadline for the annual RFP for the Risk community of practice is January 15th.
  6. The Usability collaboration area is changing up its meeting format to have resource reviews on odd months, and 30 minute Office Hours twice a month on even months. Learn more here.
  7. Rescale is a new tool for advanced scientific computing. Learn more here and apply to use Rescale (currently only for USGS employees) here.
  8. The NWIS Mod: Imagery and Remote Sensing Subprogram is working toward national consistency for imagery data. If you're interested, contact Frank (fengel@usgs.gov) or Cian (cbdawson@usgs.gov)

Notes

Welcome and Opening Announcements

  1.  Four online data Carpentries trainings throughout 2021
    1. Helpers are needed! Email cdi@usgs.gov to help out.
    2. No cost to attend
    3. Limited availability, to be continued in 2022, with the the help of newly training USGS instructors.
    4. USGS Carpentries Instructor Application: https://forms.office.com/Pages/ResponsePage.aspx?id=urWTBhhLe02TQfMvQApUlAxdiRifVmlAg0g-PN54QUVUNksxRzkwOVoyVEdQM0hDWlFWMzZVVFczOCQlQCN0PWcu 
  2. Kevin Gallagher recognized the contribution of CDI members to the community. 
  3. Deadline for nominations for the FY2021 Leadership and Innovation Award is February 15th, 2021.
    1. CDI Leadership and Innovation Award information: https://my.usgs.gov/confluence/x/DRzbJQ

  4. The CDI Workshop will be held online from May 25-28. The theme will be "Designing Data-Intensive Science with CDI"
    1. Provide training for data science skills
    2. Inform community about USGS resources for data-intensive coupling
    3. Highlight examples of collaborative, data-intensive projects
    4. Discuss data-intensive and responsive paths toward USGS EarthMap goals
    5. Submit a session proposal for the 2021 CDI Workshop: https://my.usgs.gov/confluence/x/VIBsKQ

Group Announcements

See slides above for full group announcements.

For more information on any of the collaboration areas, see https://my.usgs.gov/confluence/x/yhv1I 

  1. Geomorphology
    1. Jess LeRoy and Peter McCarthy are reviving the Geomorphology Focus Group. Sign up for the mailing list here.
  2. Risk
    1. Annual RFP Risk proposal is due January 15th, 2021.
    2. Next meeting: Jan 21
    3. ETWG Risk Community of Practice
  3. Usability
    1. Resource review: Jan 20, how to design an effective data dashboard for your users
    2. Next meeting: Feb 3, office hour kick-off, get in touch if you're interested in participating
  4. Software Development
    1. Next meeting: Jan 28
  5. Data Management Working Group
    1. Next meeting: Feb 8, data citation at the USGS
    2. Last meeting's talk on OpenRefine, a tool for data cleaning, is available: DMWG Meeting 2021_01_11
  6. eDNA
    1. USFS Conservation Genomics Internship: two research opportunities
    2. Standards for eDNA sampling data reporting in Canada, available for public review
    3. ETWG eDNA Community of Practice See today's announcements to the right of the page
  7. DevOps Sync
    1. Next meeting: Feb 2
  8. Tech Stack
    1. Next meeting: Feb 18

Usability Collaboration Area Spotlight - Sophie Hou, contractor to USGS

  1. What is the collaboration area aiming to do?
    1. to share usability best practices and lessons learned
    2. to facilitate the understanding of USGS' usability needs
    3. to allow everyone to contribute to USGS' usability solutions
    4. but especially: get people excited about usability!
  2. What are some of the highlight topics from last year?
    1. eight different presentations, and eight resource reviews
      1. the Paperwork Reduction Act presentation generated a lot of discussion
      2. the length of time/work involved for putting together the application packet for the normal method puts people off; there is an expedited process. See the usability collaboration areas' past meetings for more.
  3. Why is there a new format for this year?
    1. Resource reviews will continue to be posted on the 3rd Wednesday of the odd months, with the first one this year being posted on Jan 20.
    2. Instead of one Town Hall meeting, "office hours" will be held on the 1st and 3rd Wednesdays of the even months, with the first being Feb 3 and the second on Feb 17
      1. goals are to provide usability evaluations for services/tools/applications, discuss usability issues encountered by project teams, and demo/practice usability techniques
      2. basically, want to be sure that usability is useful!
  4. What does it mean to 'improve usability'?
    1. providing the functions your users need
    2. implementing the functions in a user-friendly manner
    3. enhancing the usability quality (i.e. learnability, efficiency, memorability, error prevention, user satisfaction)
    4. these bullets from Jacob Nielsen's work
    5. improve our users' lives!
  5. What are some 'quick fixes' that are often seen for improving usability?
    1. ask who are we designing for and why
    2. learn (more) about your target audiences
    3. allocate time for iterations
    4. solicit feedback for your designs
    5. document and share lessons learned with your team members
    6. if you want a great tool/product, you've got to test (adapted from a Steve Krug quote)
  6. What would be your vision of the ideal role of usability experts in the USGS?
    1. value is understood and supported
      1. usability is often not the first thing that people think of; hard to communicate the introduction of the usability role.
    2. roles and responsibilities are defined and recognized
    3. collaboration is the norm
      1. usability cannot be conducted alone; needs to be part of the project team; interactive to understand the community's leads
    4. professional growth is available
    5. usability experts are integral to tool/product strategies and implementations (adapted from quote by Jared Spool)
  7. How to participate
    1. join the listerv (https://listserv.usgs.gov/mailman/listinfo/cdi-usability)
    2. read the resource reviews, give feedback and thoughts on new topics
    3. attend the 'office hours'
    4. add
      1. a forum post to the collaboration area's wiki (Usability Collaboration Area)
      2. a message to the USGS CDI 'usability topics'
    5. reach out to Sophie Hou (chungyihou@contractor.usgs.gov)

Advanced Scientific Computing in the USGS, including Rescale - Courtney Neu and Janice Gordon, USGS

  1. What is advanced scientific computing?
    1. formed out of an EarthMap meeting
    2. an umbrella term used to describe various architectures and computing methods designed to support data and computationally intensive scientific modeling workflows
    3. encompasses High Performance Computing, High Through-put Computing, and AI/ML
  2. Why do we need advanced scientific computing?
    1. at some point in research, we must expand the current study area, integrate new data, and increase model resolution. This no longer works on a desktop
  3. Overview
    1. working across organizational boundaries to provide advanced scientific computing capabilities to our researchers
    2. goal is to provide advanced scientific computing capabilities and expertise to researchers for the acerbation of research
  4. ACC provides machine access, research, partnerships, and training and consulting
    1. machine access: HPC Platforms
    2. Yeti, Denali, Rescale
    3. AI/ML Learning platforms: Tallgrass and SageMaker
  5. Rescale overview
    1. FedRAMP certified software
    2. provides access to latest computer architecture on-demand, no waiting in a queue to run a job
    3. extracts the intricacies of Cloud away from user
    4. uses Amazon Web Services
    5. managed by Cloud Hosting Solutions and Advanced Research Computing
  6. Rescale Features
    1. built-in remote desktop and visualization capabilities
    2. custom budgeting tools - hard limits on spending
    3. 680+ HPC software applications, ported and tuned for cloud infrastructure
    4. hybrid cloud
      1. provides a single interface for using HPC resources regardless of their location
  7. See recording for demo of Rescale
  8. To get access, submit a request using this Microsoft Form:
    1. https://forms.office.com/Pages/ResponsePage.aspx?id=urWTBhhLe02TQfMvQApUlAzBlKGt_-RBskFhRQ-t3GRUMzlTRVowOUtKWVZGUTZKU0RURkpHTk9KTyQlQCN0PWcu
  9. Other Resources
    1. Rescale Resource Center: https://resources.rescale.com/

    2. ARC Rescale User Documentation: https://hpcportal.cr.usgs.gov/hpc-user-docs/Rescale/Overview.html

    3. CHS Rescale Overview: https://support.chs.usgs.gov/x/FwnVAg

CDI Pop-Up Lab: Collecting and serving imagery data online, including the NWIS Imagery Data Storage and Persistence Project - Frank Engel, USGS

  1. NWIS Mod: Imagery and Remote Sensing Subprogram
    1. toward national consistency for imagery data > enterprise cyberinfrastructure
    2. this project stems from a 2018 CDI project
    3. motivation:
      1. increasing growth & use cases
      2. need for data standards (imagery as observations)
      3. unhelpful variation in infrastructure
    4. why imagery?
      1. over 800 cameras currently installed at river streamgages
      2. cameras provide observations
        1. control conditions
        2. hydrologic variables
        3. near-range remote sensing
      3. non-contact means fewer equipment failures, and also potentially safer
  2. Process
    1. first step is trying to figure out what's on the landscape now, so we can see where users are being underserved, and where the problems are, in a gap analysis of sorts
    2. approach feasibility: se what is in scope, and start targeting users
    3. build out requirements in a software development roadmap
  3. How you can help
    1. continue interacting with the project team through CDI forums
    2. product owner? participate in our inventory survey
    3. stakeholder engagement
      1. data standards and practice for imagery
      2. testing software
    4. If you are a product owner and the project team hasn't talked to you, please email Frank (fengel@usgs.gov) or (cbdawson@usgs.gov)

Questions

  1. Who is the new point of contact for the PRA (I believe James retired)? Is there a site in the USGS which documents the PRA approval workflow?
    1. Chris Bartlett
  2. Can we load our own environments through Conda or Docker files on Rescale?
    1. Yes.
  3. I wouldn't have a clue how to budget for this activity. Are there any budget tools?
    1. We are in testing phase; recommend that you test it out yourself, then reach out to Rescale team for an estimation of the cost
  4. Does this replace Pangeo? Or does Rescale run on a particular machine, unlike Pangeo?
    1. Yes, these systems are complimentary; Rescale is not a replacement for Pangeo
  5. Are the cloud services available to other bureaus within DOI (National Park Service)?
    1. We cannot provide services to outside of USGS currently. There is some possibility for this in the future.
  6. Can rescale interact with CHS buckets?
    1. yes should be able to according to Rescale engineers
  7. Do you envision this encompassing data from UAS at some point, or are you just focusing on bridge/bank-mounted imaging?
    1. Focused on fixed cameras installed at streamgages. However, we are thinking of broader uses for imagery
  8. How much metadata are we thinking about?
    1. the way we're looking at it now, this is an application of an IOT device, which uses formatted JSON as metadata. We will likely develop our own metadata standard.
  9. Will the images you compile and serve be geo-referenced/orthorectified, as would be necessary for PIV, or would that be up to end user? If so, will the database include any provision for ground control points, scale objects, etc.?
    1. Yes, image processing steps are included in the backlog projects.