CDI Software Development Cluster

Meeting Notes

July 25th, 2019 @ 3:30PM ET / 1:30PM MT

Topic: Deeper dive into containers and docker, WMA and NGTOC
 

Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/696628840

Or iPhone one-tap :

US: +1-669-900-6833,,696628840# or +14087403766,,696628840#

Or Telephone:

Dial(for higher quality, dial a number based on your current location):

US: +1 669 900 6833 or +1 408 740 3766 or +1 646 876 9923

Meeting ID: 696 628 840

Note that we have switched from GSTalk to Zoom for the time being, due to a number of usability/compatibility issues with the GSTalk platform.

 

Meeting Notes in Google Drive: Shared Google Drive Folder:

https://docs.google.com/document/d/1tC4Pmmhax_CTL2-wsjKlmqBA8DziRfelvJgbLKWPD5I/edit?usp=sharing

 

Attendees

Name

Email (if you are new)

Michelle Guy

 

Jeremy Newson

 

Mason Hauck

 

Hans Vraga

 

Nick Estes

 

Leslie Hsu

 

Dennis Walworth

 

Mary Bucknell

 

Shailendra Rahtapuri

 

Robert Djurasaj

 

Rob Miller

 

Ivan Suftin

 

Carl Schroedl

 

Eric Martinez

 

Tom Burley

 

Shayne Urbanowski

 

We had 25 guests total

 

 

 

 

Recommended Viewings prior to discussion:

    Docker Container Tutorial - How to build a Docker Container & Image

    Building Microservices with the 12 Factor App Pattern on AWS - AWS Online Tech Talks


Agenda

     Welcome and announcements

     Please fill in name and email in the attendees table

     Looking for contributors for our presentation: Software's role in data integration and delivery in the upcoming CDI Monthly Meeting on August 14 at 11a-12:30p Eastern Time. 30 minutes with Q&A. (email co-leads: Michelle Guy, Jeremy Newson, and Cassandra Ladino)

     Quick Sli.do Poll (code #M229)

     We are always looking for topics, and your input and participation!

     Submit Ideas for a presentation here:

     https://docs.google.com/forms/d/e/1FAIpQLSccsoCmFH4aT1OQNKaMDG7-ngIAlyGgmqSRQwJc_uYFf_tVUQ/viewform

    CDI bison connect google calendar of all the collaboration area meetings and events - name is “GS CDI” owner is gs_cdi@usgs.gov (Also accessible Calendar wiki page or the Google Calendar link . )

    Become a future co-lead! Apply below:

   https://forms.gle/djjoxmN22H76qSr69

     WMA Presentation, Carl Schroedl and Ivan Suftin (30min)

      Registries

     GitLab

     Artifactory

      CI/CD

     GitLab

     Jenkins

      Orchestration

     docker-compose

     ECS

      NGTOC Presentation, Robert Djurasaj (30min)

      Gitlab, AWS CDK and ECS

      Infrastructure as Code Gitlab runner

      AWS-CDK ECS Demo

      https://docs.google.com/presentation/d/1QILKUN0S2SmajNFwMw_iHXmVzuzGgfZZjRWBmeimW1w/edit?usp=sharing

     Next Month: Definitely allow more time for presentations and questions! We ran short or time this month

 

 

Discussion/Notes

      Notes on WMA presentation:

      Quick primer on Docker and DockerHub Image Registry

      Image Registries

      Water mission area has artifactory server hosted at EROS which is publicly accessible but no SLA and low bandwidth

      CHS has a registry too, free, next business day support (code.chs.usgs.gov), is internal to DOI access only. check out code.chs.usgs.gov/devops/images for some base images

      DockerHub is free when fully publicly accessible, high availability

      If you want internally hosted and publicly accessible registry available email gs-help-git@usgs.gov (one vote) that *could* be on code.usgs.gov

      Amazon offers a registry for when running in AWS, but Water does not have experience with it, likely not free

      WMA Docker Images

      Spring Boot base docker image available to share common practices on things like environment variable (including certificates) - See https://github.com/USGS-CIDA/docker-wma-spring-boot-base

      Waits “waitfor” script -  e.g. containers may need to wait for a DB to be accessible before a container should run as it would need to connect to the DB

      Things that did not work: thinking of war, jar files as deployable unit of code in addition to docker image, but then two different repositories but these two things were tightly coupled, so integrating docker file into same repo as application, so simplify by having deployment from one repo

      Wanted to talk about orchestration, HOPEFULLY NEXT TIME!!!

     Notes on NGTOC presentation

     Using cloud last 3 years, focus on AWS CDK and ECS

      AWS ECS (Fargate) introduced recently because more tools available there than for EKS which just came out by CHS

     Fargate - you only worry about containers and AWS takes care of everything else

      AWS CDK - cloud development kit ( https://aws.amazon.com.cdk )

     Don’t have to import as many things as used to (like roles and policy) these were also contributed back to CHS/AWS

     Cannot create VPC’s or security groups

     Had to override CDK to use specified permission boundary

     Version one just last week but have been using since version 0.2, so still pretty new, but recommended

      Gitlab runners - available in CHS service catalog; worked with CHS as needed more access/permissions

     5 commits will create 5 runners in parallel, this is problematic, gitlab is working on this

     Cannot create job specific variable set, so do not set in IAC, provide them

     gitlab-ci.yml file controls steps taken by runner

      Want runners to work for all groups, so works at NGTOC cloud level

      Demo - creates a load balancer, a cluster and and spins up container on ECS

     Import security group and VPC

     Create load balancer and cluster

     Use container registry

     CDK will show list of stacks being created

     In demo changed from nginx to http

     All steps done by CDK and gitlab runner, no need for console

      Lambda size limited (200MB), using AWS batch for larger jobs as containers on EC2 instances

      Manually triggered action to destroy all of cloud environment with one click when done. This lessoned learned as when using console tendency to leave things up and running

      More demo and talk at coming CHS user group meeting (on CDI calendar)

      Feel free to contact Robert he can “geek out” on this stuff all day long :-)