Transactions in Distributed Systems

Joe goes full shock jock, but only for a moment. Allen loses the "Most Tips In A Single Episode: 2021" award, and Michael didn't get the the invite notification in this week's episode.

The full show notes for this episode are available at https://www.codingblocks.net/episode168.

Sponsors

• Datadog – Sign up today for a free 14 day trial and get a free Datadog t-shirt after creating your first dashboard.
• Shortcut - Project management has never been easier. Check out how Shortcut (formerly known as Clubhouse) is project management without all the management.

Survey Says

Well...no survey this week, but this is where it would be!

News

This book has a whole chapter on transactions in distributed systems

• Thank you to everyone that left a review!
  • "Podchaser: alexi*********, Nicholas G Larsen, Kubernutties,
  • iTunes: Kidboyadde, Metalgeeksteve, cametumbling, jstef16, Fr1ek
  • Audible: Anonymous (we are like your mother - go clean your room and learn Docker)

• Atlanta Code Camp is right around the corner on October 9th. Stop by the CB booth and say hi! (AtlantaCodeCamp.com)

Maintaining data consistency

• Each service should have its own data store
• What about transactions? microservices.io suggests the saga pattern (website)
• A sequence of local transactions must occur
  • Order service saves to its data store, then sends a message that it is done
  • Customer service attempts to save to its data store…if it succeeds, the transaction is done. If it fails, it sends a message stating so, and then the Order service would need to run another update to undo the previous action
• Sound complicated? It is…a bit, you can't rely on a standard 2 Phase Commit at the database level to ensure an atomic transaction
• Ways to achieve this - choreography or orchestration

Choreography Saga

- The Order Service receives the POST /orders request and creates an Order in a PENDING state
- It then emits an Order Created event
- The Customer Service's event handler attempts to reserve credit
- It then emits an event indicating the outcome
- The OrderService's event handler either approves or rejects the Order

• Each service's local transaction sends a domain event that triggers another service's local transaction
• To sum things up, each service knows where to listen for work it should do, and it knows where to publishes the results of it's work. It's up to the designers of the system to set things up such that the right things happened
• What's good about this approach?

"The code I wrote sux. The code I'm writing is cool. The code I'm going to write rocks!"

Thanks for the paraphrase Mike!

Orchestration Saga

- The Order Service receives the POST /orders request and creates the Create Order saga orchestrator
- The saga orchestrator creates an Order in the PENDING state
- It then sends a Reserve Credit command to the Customer Service
- The Customer Service attempts to reserve credit
- It then sends back a reply message indicating the outcome
- The saga orchestrator either approves or rejects the Order

• There is an orchestrator object that tells each service what transaction to run
• The difference between Orchestration and Choreography is that the orchestration approach has a "brain" - an object that centralizes the logic and can make more advanced changes
• These patterns allow you to maintain data consistency across multiple services
• The programming is quite a bit more complicated - you have to write rollback / undo transactions - can't rely on ACID types of transactions we've come to rely on in databases
• Other issues to understand
  • The service must update the local transaction AND publish the message / event
  • The client that initiates the saga (asynchronously) needs to be able to determine the outcome
    • The service sends back a response when the saga completes
    • The service sends back a response when the order id is created and then polls for the status of the overall saga
    • The service sends back a response when the order id is created and then submits an event via a webhook or similar when the saga completes
• When would you use Orchestration vs Choreography for transactions across Microservices?
• Friend of the show @swyx works for Temporal, a company that does microservice orchestration as a service, https://temporal.io/

Tips for writing Great Microservices

Fantastic article on how to keep microservices loosely coupled
https://www.capitalone.com/tech/software-engineering/how-to-avoid-loose-coupled-microservices/

• Mentions using separate data storage / dbs per service
  • Can't hide implementation from other services if they can see what's happening behind the scenes - leads to tight coupling
• Share as little code as possible
  • Tempting to share things like customer objects, but doing so tightly couples the various microservices
    • Better to nearly duplicate those objects in a NON-shared way - that way the services can change independently
• Avoid synchronous communication where possible
  • This means relying on message brokers, polling, callbacks, etc
• Don't use shared test environments / appliances
  • May not sound right, but sharing a service may lead to problems - like multiple services using the same test service could introduce performance problems
• Share as little domain data as possible - ie. important pieces of information shouldn't be passed around various services in domain objects. Only the bits of information necessary should be shared with each service - ie an order number or a customer number. Just enough to let the next microservice be able to do its job

Resources

• https://microservices.io/
• Sam Newman books (Thanks Jim!)
  • Monolith to Microservices
  • Building Microservices
• https://segment.com/blog/goodbye-microservices/
• https://stackoverflow.blog/2020/11/23/the-macro-problem-with-microservices/
• Designing Data Intensive Applications
• https://www.dashcon.io/

Tip of the Week

• Podman is an open-source containerization tool from Red Hat that provides a drop in replacement for Docker (they even recommend aliasing it!). The major difference is in how it works underneath, spawning process directly rather than relying on resident daemons. Additionally, podman was designed in a post Kubernetes world, and it has some additional tooling that makes it easier to transition to Kubernetes- like being able to spawn pods and generate Kubernetes yaml files. Website
  • Check out this episode from Google's Kubernetes podcast all about it: Podcast
• Unity is the most popular game engine and they have a ton of resources in their Learning Center. Including one that is focused on writing code. It walks you through writing 5 microgames with hands on exercises where you fix projects and ultimately design and write your own simple game. Also it's free! https://learn.unity.com/course/create-with-code
  • Bonus: Make sure you subscribe to Jason Weimann's YouTube channel if you are interested in making games. Brilliant coder and communicator has a wide variety of videos: YouTube
• Educative.io has been a sponsor of the show before and we really like their approach to hands on teaching so Joe took a look to see if they had any resources on C++ since he was interested in possibly pursuing competitive programming. Not only do they have C++ courses, but they actually have a course specifically for Competitive Programming in C++. Great for devs who already know a programming language and are wanting to transition without having to start at step 1. Educative Course
• The most recent Coding Blocks Mailing List contest asked for "Summer Song" recommendations, we compiled them into a Spotify Summer Playlist. These are songs that remind you of summer, and don't worry we deduped the list so there is only one song from Rick Astley on there. Spotify
• Finally, one special recommendation for Coding Music. It's niche, for sure, but if you like coding to instrumental rock/hard-rock then you have to check out a 2018 album from a band called Night Verses. It's like Russian Circles had a baby with the Mercury Program. If you are familiar with either of those bands, or just want something different then make sure to check it out. Spotify