The DevOps Handbook – Architecting for Low-Risk Releases

Our journey into the world of DevOps continues with The DevOps Handbook as Michael doesn't take enough tangents, Joe regrets automating the build, err, wait never regrets (sorry), and ducks really like Allen.

If you're reading these show notes via your podcast player, you can find this episode's full show notes at https://www.codingblocks.net/episode137, where you can be a part of the conversation.

Sponsors

• Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after your first dashboard.
• Secure Code Warrior – Start gamifying your organization's security posture today, score 5,000 points, and get a free Secure Code Warrior t-shirt.

Survey Says

News

• Thank you to everyone that left us a new review!
  • iTunes: justsomedudewritingareview, stupub, andrew.diamond, scipiomarcellus
  • Stitcher: Bicycle Repairman, BrunoLC
• Using Bazel to build and test software of any size, quickly and reliably. (bazel.build)
• Reflections on Trusting Trust (cs.cmu.edu)
• Build your own Linux distro! (linuxfromscratch.org)
• That System76 Oryx Pro keyboard though? (System76)

Fast, Reliable. Pick Two

Continuously Build, Test, and Integrate our Code and Environments

• Build and test processes run constantly, independent of coding.
• This ensures that we understand and codify all dependencies.
• This ensures repeatable deployments and configuration management.
• Once changes make it into source control, the packages and binaries are created only ONCE. Those same packages are used throughout the rest of the pipeline to ensure all environments are receiving the same bits.
• What does this mean for our team culture?
  • You need to maintain reliable automated tests that truly validate deploy-ability.
  • You need to stop the "production pipeline" when validation fails, i.e. pull the andon cord.
  • You need to work in small, short lived batches that are based on trunk. No long-lived feature branches.
    • Short, fast feedback loops are necessary; builds on every commit.

Integrate Performance Testing into the Test Suite

• Should likely build the performance testing environment at the beginning of the project so that it can be used throughout.
• Logging results on each run is also important. If a set of changes shows a drastic difference from the previous run, then it should be investigated.

Enable and Practice Continuous Integration

• Small batch and andon cord style development practices optimize for team productivity.
• Long lived feature branches optimize for individual productivity. But:
  • They require painful integration periods, such as complex merges, which is "invisible work".
  • They can complicate pipelines.
  • The integration complexity scales exponentially with the number of feature branches in play.
  • They can make adding new features, teams, and individuals to a team really difficult.
• Trunk based development has some major benefits such as:
  • Merging more often means finding problems sooner.
  • It moves us closer to "single piece flow", such as single envelope at a time story, like one big assembly line.

Automate and Enable Low-Risk Releases

• Small batch changes are inherently less risky.
• The time to fix is strongly correlated with the time to remediate, i.e. the mean time to find (MTF) and the mean time to remediate (MTR).
• Automation needs to include operational changes, such as restarting services, that need to happen as well.
• Enable "self-service" deployments. Teams and individuals need to be able to dynamically spin up reliable environments.

Decouple Deployments from Releases

• Releases are marketing driven and refer to when features are made available to customers.
• Feature flags can be used to toggle the release of functionality independent of their deployments.
• Feature flags enable roll back, graceful degradation, graceful release, and resilience.

Architect for Low-Risk Releases

• Don't start over! You make a lot of the same mistakes, and new ones, and ultimately end up at the same place. Instead, fix forward!
• Use the strangler pattern instead to push the good stuff in and push the bad stuff out, like how strangler vines grow to cover and subsume a fig tree.
• Decouple your code and architecture.
• Use good, strong versioned APIs, and dependency management to help get there.

Resources We Like

• The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations (Amazon)
• The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win (Amazon)
• The Unicorn Project: A Novel about Developers, Digital Disruption, and Thriving in the Age of Data (Amazon)
• Andon (manufacturing) (Wikipedia)
• LEGO Nintendo Entertainment System (lego.com)
• Comparing Git Workflows (episode 90)
• StranglerFigApplication (MartinFowler.com)

Tip of the Week

• Be sure to check out the gaming channel in Slack to find and connect with some great people for your next game.
  • Also, look up in the Slack channel. There may be cool information in the channel's description.
• Within JetBrains IDEs, such as Datagrip and IntelliJ, press CTRL+SHIFT+V to see and choose from the list of items recently copied to the clipboard.
• Tired of working from home? Work from a park!

• Save your sanity by customizing your Macbook Pro Touchbar in Chrome by going to View -> Customize Touchbar.