Research organizations may deal in different topical areas and use varied tools and approaches in their day-to-day operations but one thing is true for everyone in the science and research domain: data are getting bigger. Whether it is the size of dataset inputs or the resulting output that a particular analysis produces, authors and data managers are often hastily playing catch up in response to today’s data storage and access demands. How is this affecting the USGS? What are the current trends and the latest developments in data storage available to our researchers? Does the ‘Click to Download’ model still work for the data we are producing, given the size of our products and the workflows of other researchers? This session will provide an opportunity for an update on the current capabilities provided by the SAS mission area to support USGS scientists, as well as an open discussion for anyone dealing with large data challenges.
The working plan for this session would be for interested folks to present in a 10-15 minute time slot sharing experiences and lessons learned related to this topic.
Planned speakers are listed below. Please feel free to comment or ask questions below and stop by the talk in June!
Talk 1: Black Pearl - storing large data for use in High Performance Computing (HPC) systems - AND - GLOBUS: large data transfer, sharing and publishing. (Jeff Falgout / Matt Davis)
Talk 2: ScienceBase integration with Black Pearl, Amazon S3, and brokered relationship with EROS EE for large data handling. (Drew Ignizio)
Talk 3: Using the Unidata THREDDS Data Server to Provide Access to Large Datasets - NCAR's Research Data Archive Perspective (Doug Schuster, NCAR)
Talk 4: Cloud-friendly data formats. (Rich Signell)
Science Support Framework Category: - Data Management
Author(s): Drew Ignizio (email@example.com) - USGS Science Analytics and Synthesis
Notes Document: https://tinyurl.com/CDI0605-Ignizio