2019.7.25 Meeting.docx

CDI Software Development Cluster

Meeting Notes

July 25th, 2019 @ 3:30PM ET / 1:30PM MT

Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/696628840

Or iPhone one-tap :

Meeting Notes in Google Drive: Shared Google Drive Folder:

https://docs.google.com/document/d/1tC4Pmmhax_CTL2-wsjKlmqBA8DziRfelvJgbLKWPD5I/edit?usp=sharing

●     Docker Container Tutorial - How to build a Docker Container & Image

●     Building Microservices with the 12 Factor App Pattern on AWS - AWS Online Tech Talks

Agenda

●      Welcome and announcements

○      Please fill in name and email in the attendees table

○      Looking for contributors for our presentation: Software's role in data integration and delivery in the upcoming CDI Monthly Meeting on August 14 at 11a-12:30p Eastern Time. 30 minutes with Q&A. (email co-leads: Michelle Guy, Jeremy Newson, and Cassandra Ladino)

○      Quick Sli.do Poll (code #M229)

○      We are always looking for topics, and your input and participation!

○      Submit Ideas for a presentation here:

■      https://docs.google.com/forms/d/e/1FAIpQLSccsoCmFH4aT1OQNKaMDG7-ngIAlyGgmqSRQwJc_uYFf_tVUQ/viewform

○     CDI bison connect google calendar of all the collaboration area meetings and events - name is “GS CDI” owner is gs_cdi@usgs.gov (Also accessible Calendar wiki page or the Google Calendar link . )

○     Become a future co-lead! Apply below:

■    https://forms.gle/djjoxmN22H76qSr69

●      WMA Presentation, Carl Schroedl and Ivan Suftin (30min)

○       Registries

■      GitLab

■      Artifactory

○       CI/CD

■      GitLab

■      Jenkins

○       Orchestration

■      docker-compose

■      ECS

●       NGTOC Presentation, Robert Djurasaj (30min)

○       Gitlab, AWS CDK and ECS

■       Infrastructure as Code Gitlab runner

■       AWS-CDK ECS Demo

○       https://docs.google.com/presentation/d/1QILKUN0S2SmajNFwMw_iHXmVzuzGgfZZjRWBmeimW1w/edit?usp=sharing

●      Next Month: Definitely allow more time for presentations and questions! We ran short or time this month

Discussion/Notes

●       Notes on WMA presentation:

○       Quick primer on Docker and DockerHub Image Registry

○       Image Registries

■       Water mission area has artifactory server hosted at EROS which is publicly accessible but no SLA and low bandwidth

■       CHS has a registry too, free, next business day support (code.chs.usgs.gov), is internal to DOI access only. check out code.chs.usgs.gov/devops/images for some base images

■       DockerHub is free when fully publicly accessible, high availability

■       If you want internally hosted and publicly accessible registry available email gs-help-git@usgs.gov (one vote) that *could* be on code.usgs.gov

■       Amazon offers a registry for when running in AWS, but Water does not have experience with it, likely not free

○       WMA Docker Images

■       Spring Boot base docker image available to share common practices on things like environment variable (including certificates) - See https://github.com/USGS-CIDA/docker-wma-spring-boot-base

■       Waits “waitfor” script -  e.g. containers may need to wait for a DB to be accessible before a container should run as it would need to connect to the DB

■       Things that did not work: thinking of war, jar files as deployable unit of code in addition to docker image, but then two different repositories but these two things were tightly coupled, so integrating docker file into same repo as application, so simplify by having deployment from one repo

○       Wanted to talk about orchestration, HOPEFULLY NEXT TIME!!!

●      Notes on NGTOC presentation

○      Using cloud last 3 years, focus on AWS CDK and ECS

○       AWS ECS (Fargate) introduced recently because more tools available there than for EKS which just came out by CHS

■      Fargate - you only worry about containers and AWS takes care of everything else

○       AWS CDK - cloud development kit ( https://aws.amazon.com.cdk )

■      Don’t have to import as many things as used to (like roles and policy) these were also contributed back to CHS/AWS

■      Cannot create VPC’s or security groups

■      Had to override CDK to use specified permission boundary

■      Version one just last week but have been using since version 0.2, so still pretty new, but recommended

○       Gitlab runners - available in CHS service catalog; worked with CHS as needed more access/permissions

■      5 commits will create 5 runners in parallel, this is problematic, gitlab is working on this

■      Cannot create job specific variable set, so do not set in IAC, provide them

■      gitlab-ci.yml file controls steps taken by runner

○       Want runners to work for all groups, so works at NGTOC cloud level

○       Demo - creates a load balancer, a cluster and and spins up container on ECS

■      Import security group and VPC

■      Create load balancer and cluster

■      Use container registry

■      CDK will show list of stacks being created

■      In demo changed from nginx to http

■      All steps done by CDK and gitlab runner, no need for console

○       Lambda size limited (200MB), using AWS batch for larger jobs as containers on EC2 instances

○       Manually triggered action to destroy all of cloud environment with one click when done. This lessoned learned as when using console tendency to leave things up and running

○       More demo and talk at coming CHS user group meeting (on CDI calendar)

○       Feel free to contact Robert he can “geek out” on this stuff all day long :-)