Note - the recording will be posted for those that weren't able to make the call.
Courtney Owens, CHS
See Forum post on Existing High Performance Computing Bioinformatics Resources
Alces Flight is a Linux-based software, you set it up in its own CHS environment
"Alces Flight Compute provides a fully-featured, scalable High Performance Computing (HPC) environment for research and scientific computing. Compatible with both on-demand and spot instances, Flight rapidly delivers a whole HPC cluster, ready to go and complete with job scheduler and applications."
Command line interface
1 cluster for each scientist
Spot Price Market - fraction of market rate
1240 software applications
671 Bioinformatics applications
2 different repositories with different levels of "readiness"
Main - thoroughly tested with latest release of Alces Flight
Volatile - have not been thoroughly tested - may have more issues
You can request for an application to be added by posting in the forum or the private support system.
You can install all bioinformatics apps with one command
There are three editions
Solo Community - USGS CHS started off with this for beta testing but for better support, switched to...
Solo Professional - started using at the beginning of February, get own USGS private support channel/forum - there are questions there from beta testers. There is a choice of HPC Scheduler. And there are Preinstalled software packs.
Initial Beta Testing Results
Issues installing packages from the volatile repository, specifically Qiime
We would like more bioinformatics scientists to test - contact Courtney at email@example.com
Testing is free
Q: If we have used other commercial package software, can they be added to Alces - do you need special licenses?
A: Courtney will look into that and let them know.
Scott Cornman: this is entirely commandline as it is being tested. Will there be a GUI option?
A: They can’t have a GUI because of security restrictions. But Courtney will look into that. Q: the commercial packages from the previous question use GUIs, FYI.
Denise Akob: How many beta testers so far?
A: Have had 5-6 total, 2 active now - Adam Mumford and Scott Cornman. Q: What software have they been using? Qiime and mother (?). Courtney is working with Janice Gordon and Jeff Falgout, they will be testing, maybe python.
Sophia: Is testing in person or remotely?
A: we are able to do it remotely - go into the environment, install packages and run models on them.
Denise: what is the time frame to open this up to people who have need for this computing environment?
Courtney: Enterprise launch at CDI workshop in May is a plan.
Denise: how will billing work? Will scientists get billed individually?
A: we are figuring that out. Testing is currently funded by OEI Tim Quinn, it is going to depend on how cost structure will break down. Not individual scientist billing at first. Hopefully will have funding through the end of the fiscal year.
Scott: Will the cost structure be roughly on the scale of Amazon rates?
A: Should be 20-30% lower than normal Amazon. Running models with Beta testing will help them figure out the breakdown.
Scott: is there a minimum set of users? Because it will take awhile to grow the base.
A: That would be supervisor’s decision. We hope it will grow.
Courtney: hope to have more releases with added features in the future. Based on the needs that they hear from customers. Big application registry with a lot of demand. CHS team is about 15 people. Moving to managed service route to help more customers.
Denise: how often are softwares updated? Regularly? Does it need to be requested?
A: Every 3-4 months. Some software is more updated than others.
Demo by Courtney
Comment: new users will probably need help setting up their environment
Alces can check the dependencies of volatile packages if you request it.
On the documentation, there is the structure in place to install packages outside of the Gridware system - will that be enabled?
A: we think so. Just need to make sure permissions are okay for everyone. Sometimes if you install numpy outside of qiime it is okay.
Do you need to be well versed in Linux to use this?
A: We’re hoping it would be easy to go in, set permissions, install packages, but you need to be used to using commandline.
Can this be done interactively, or only scheduling and batch mode?
A: Can be done interactively, don’t need to write a script to submit the job.
Will you be able to run interactively when it is scaled up.
If you have autoscaling turned on, then you should be able to. I’ll check with Jeff.
From Courtney Owens, 3/1/17:
Thank you for a great discussion on Alces Flight.
Here are answers to the questions I was not able to fully answer on the call:
Will there be a GUI option with Alces Flight?
Jeff Carson and I have started this discussion with the rest of our team. We need to figure out if there are any work-a-rounds available for having a GUI that would still abide by necessary security constraints. We are not sure on the timeline for this and will keep you updated.
Can commercial package software be added to Alces Flight?
We have not tested this yet, but believe it is possible to install commercial software packages on Alces Flight. We can verify this with Alces Flight if a user can provide us with a particular package that they are interested in. Furthermore, we would like to point out that the Alces Flight Platform is designed to be ephemeral. Because of this, we do not recommend installing commercial package software that you are not using in conjunction with software applications available in Alces Flight. We do not think that Alces Flight is the right tool for this particular use case and are hoping to have an offering that will accommodate this use case in the future.
When you run interactive jobs, do compute nodes still autoscale?
An interactive job will only use whatever compute nodes are available to them. This means you will have to choose the type and number of nodes you want to run interactively. Our scaling policies will terminate compute nodes provisioned this way when they are not being used, but will not scale them up.
Please let me know if you have any additional questions or are interested in helping us beta test.
Thank you for your time,
Hacking is not just with a negative connotation!
Leverage crowdsourcing, citizen science, and civic hacking
Promote free and open innovation skills
Hack for Change and Hack Red Tape. hacking in a positive way.
Not just shiny object at the end, but a socio-cultural, technological, organizational, political-policy advancement
There can be Art and design aspects
Sophia is organizing Date, Time, and Location - TBD
Have one at DOI, one at Reston USGS, non-federal location
Try to make it accessible via live streaming.
Project Themes relevant to this call
Critical Minerals and Resources
Threats to Biodiversity
Invasive Species project
How to leverage remote sensing data sources and products to detect invasive species?
eDNA - Sophia is working with Andrea Ostroff, JC Nelson, Denise Akob
Ideahack - explore ways to create sustainable viz/collab tools
Do you have comments, thoughts, questions, concerns?
Who might be primary users of eDNA and etc.
Denise: an option for the Bioinformatics CoP, we could talk through algorithms for bioinformatics at some sort of Hackathon.
Scott: can you clarify what the structure of a GeoHackJam is? What does it look like? It is a single day? A: Keynote speakers and short presentations on the projects and tools, goals, roundrobin introductions, geohacking, summary to share outputs, progress, next steps, Mappy Hour. Each topic has a day devoted. Then final sprint and presentations of general hacks.
Sophia: welcomes discussions on disciplinary aspect so she can learn more about the discipline. What datasets? Q: do you need location and nature of the sample, or do you need the detailed DNA results? Do you need more info? Ask Andrea Ostroff. If not published yet, that is understood. The more information available, then great.
What interest in eDNA is there on the phone? A: A significant minority are either interested or practicing.
There is further information on the Bioinformatics Forum post on GeoHackJam / GeoDataJam.
28 attendees, awaiting the WebEx report