Delivering software

Software Delivery Phases

For the purposes of this chapter, we refer to four software delivery phases, namely build, release, deployment, and rollout.

Software must first be built into packages. Packages should be immutable and versioned.
Packages must then be released. Release notes and change logs are updated, and packages are published to a centralized repository.
Published release artifacts must be deployed to preproduction and production environments. Deployed software is not yet accessible to users—it’s just been installed.
Once deployed, software is rolled out by shifting users to the new software. Once the rollout is complete, software is delivered.

Branching Strategies

Release packages are built from code in version control systems.

Trunk— sometimes main or mainline—contains the main version of a codebase with a history of changes.
Branches are “cut” from trunk to alter the code;

The two main families of branching strategies:

In trunk-based development, all developers work off of trunk. Branches are used for a single small feature, bug fix, or update.
- Trunkbased development works best when branches are merged back to trunk quickly, in a matter of days if not hours, and not shared between developers.
- Frequently merging is known as continuous integration (CI). CI reduces risk because changes quickly propagate to all developers, making them less likely to diverge from each other significantly. Keeping developer codebases in sync prevents potential lastminute integration hurdles and surfaces bugs and incompatibilities early.
- As a trade off, bugs in trunk will slow down all developers. To prevent breakages, fast automated tests are run to validate that tests pass before a branch is merged into trunk.
- the general expectation is that trunk should always be okay to release, and releases tend to be fairly frequent.
In feature branch–based development, many developers simultaneously work on longlived feature branches, each associated with a feature in the product. Because feature branches are long lived, developers need to rebase—to pull in changes from trunk—so the feature branch doesn’t diverge too far.
- When a release is being prepared, feature branches are pulled into the release branch. Release branches are tested, while feature branches may continue to evolve. Packages are built off stable release branches.
- The most popular feature branch approach, described by Vincent Driesen in 2010, is called Gitflow.

Feature branch–based development is more common in shrinkwrapped software where different customers run different versions; service-oriented systems usually use trunkbased development strategies.

Stick with a trunkbased branching strategy unless you truly need longlived feature branches. Managing feature branches gets complicated.

Build Phase

Software packages must be built before they’re delivered. Building software takes many steps: resolving and linking dependencies, running linters, compiling, testing, and, finally, packaging the software.

Builds outputs packages. Packages can contain binary or source code, dependencies, configurations, release notes, documentation, media, licenses, checksums, and even virtual machine images.

Version Packages: Unique identifiers help operators and developers tie a running application to specific source code, feature sets, and documentation.

Package Different Resources Separately:

Software is not just code. Configuration, schemas, images, and language packs (translations) are all part of software. Different resources have different release cadences, different build times, and different testing and verification needs.
Different resources should be packaged separately so they can be modified without having to rebuild the entire software package.
If you are shipping a complete application to a customer, the final package is then a meta-package: a package of packages.
If you are shipping a web service or a selfupgrading application, you can ship packages separately, allowing configuration and translation to upgrade separately from code.

Python packaging as an example of Build phase.

Packaging for Python tools and libraries

.py – standalone modules
sdist – Pure-Python packages: is a group of .py files—modules—that are compressed into .tar.gz archives
wheel – Python packages: include compiled native libraries written in C, C++, Fortran, R, or any other language that the Python package might depend on.

Packaging for Python applications

PEX – libraries included: include Python code and all of its library dependencies
anaconda – Python ecosystem: provides an ecosystem to manage all installed libraries, not just those your application depends on.
freezers – Python included: bundle not only libraries but also the Python runtime
images – system libraries included
containers – sandboxed images
virtual machines – kernel included
hardware – plug and play

Release Phase

Release publication makes software available to users and enables deployment, the next phase of delivery.

Complex software with multiple teams committing to it will often have a release manager role. Release managers coordinate the process—tests, feature validation, security procedures, documentation, and so on.

Use these tips to make release phase a success.

Don’t Throw Releases Over the Fence: Take responsibility for your software’s release. Even if your organization has a release engineering or operations team, you should know how and when your software ends up in front of users. Ultimately, it’s your responsibility to ensure the software is appropriately deployed and well functioning.

Publish Packages to a Release Repository: Release packages are usually published to a package repository or simply tagged and housed in a VCS like Git.

Package repositories make release artifacts (another word for a deployable package) available for deployment.
Repositories also act as archives—previous release artifacts are accessible for debugging, rollback, and phased deployments.
Package contents and metadata are indexed and browsable.
Release repositories are also built to meet deployment demands, handling thousands of users simultaneously downloading a new release.
Version control systems work as release repositories, but they aren’t built for this purpose. VCSs don’t have as many useful search and deployment features. They are not built for large deployments and can get overwhelmed
Docker Hub, GitHub Release Pages, PyPI, and Maven Central are all public repositories.

Keep Releases Immutable: Once published, never change or overwrite a release package. Immutable releases guarantee that all application instances running a specific version will be identical, byte for byte. Identical release packages let developers reason about what code is in an application and how it should behave.

Release Frequently: In practice, rapid release cycles produce more stable software that is easier to repair when bugs are found. Fewer changes go out per cycle, so each release carries less risk.

Be Transparent About Release Schedules: Release schedules define how frequently software is released. Regardless of release style, be clear about release schedules. Publish schedules and notify users when new releases are published.

Publish Changelogs and Release Notes: Changelogs and release notes help your users and your support team understand what is included in a release.

Changelogs list every ticket that was fixed or commit that was made in a release. It’s usually automated
Release notes are a summary of the new features and bug fixes contained in a release.
Changelogs are primarily read by the support and development team, while release notes are for users.

Deployment Phase

Deploying software is the act of getting software packages where they need to be to run.

Use these deployment tips.

Automate Deployments: Automated deployments are more predictable because script behavior is reproducible and version controlled.

Scripts are less likely to make mistakes than humans, and they remove the temptation to make manual system tweaks, log in to machines, or manually copy packages during deployment.
Highly evolved automation leads to continuous delivery.
With continuous delivery, humans are completely removed from deployment. Pack aging, testing, release, deployment, and even rollout are all automated.
We recommend automating your deployments with off-the-shelf tools. Custom deployment scripts are easy to start with but grow unwieldy fast. Off-the-shelf solutions like Puppet, Salt, Ansible, and Terraform integrate with existing tooling and are purposebuilt for deployment automation.

Make Deployments Atomic:

A partially deployed application can cause future deployments to fail if scripts assume installation locations are empty. To avoid failed partial deployments, make deployment all or nothing (atomic).
The easiest way to make deployments atomic is by installing software in a different location than the old version; don’t overwrite anything. Once packages are installed, a single shortcut or symbolic link can be flipped atomically.

Deploy Applications Independently: Avoid deployment ordering requests. Ordering requirements slow down deployment since applications must wait for each other. Ordering also leads to conflicts where two applications depend on each other to be upgraded.

Rollout Phase

Once the new code is deployed, you can turn it on (roll it out). Switching everything to the new code at once is risky. No amount of testing will eliminate the potential for bugs, and rolling out code to all users at once can break things for everyone simultaneously. Instead, it’s best to roll changes out gradually and monitor health metrics.

There are many rollout strategies: feature flags, circuit breakers, dark launches, canary deployments, and bluegreen deployments.

Feature flags allow you to control what percentage of users receive one code path versus another.
Circuit breakers automatically switch code paths when there’s trouble.
Dark launches, canary deployments, and blue-green deployments let you run multiple deployed versions of your software simultaneously.

These patterns mitigate the risk of dangerous changes when used appropriately.

Don’t go crazy using sophisticated rollout strategies, though—they add operational complexity.

Rollout tips.

Monitor Rollouts: Monitor health metrics such as error rates, response times, and resource consumption as new code is activated. Advanced release pipelines automatically roll a change out to more users or roll the change back based on observed statistics. Remember, your job is not done when code is committed, and it’s still not done when the code is rolled out. Hold the champagne until metrics and logs are showing your changes running successfully.

Ramp Up with Feature Flags: allow developers to control when new code is released to users. Code is wrapped in an if statement that checks a flag (set by static configuration or from a dynamic service) to determine which branch of code should be run. Feature flags can be on-off Booleans, allow lists, percentage-based ramps, or even small functions. Feature-flagged code that mutates state needs special attention.

Protect Code with Circuit Breakers: Most feature flags are controlled by humans. Circuit breakers are a special kind of feature flag that are controlled by operational events, like a spike in latency or exceptions. Circuit breakers have several unique characteristics: they’re binary (on/off), they’re permanent, and they’re automated.

Circuit breakers also protect against permanent damage. Applications that take irreversible action, such as sending an email or transfer ring money out of a bank account, use circuit breakers when it’s unclear whether to proceed.
Databases can also protect themselves by flipping to a read-only mode. Many databases and filesystems will do this automat ically if they detect disk corruption.

Ramp Service Versions in Parallel: A percentage of incom ing service calls are shifted from the old version to a new version using a switch similar to feature flags, but the switch is at the application entry point—usually a load balancer or proxy.

Canary deployments and bluegreen deployments are the two most common parallel deployment strategies.
- Canary: A new application version is first deployed to a limited set of machines. A small user subset is routed to the canary version.
- Blue-green deployments run two different versions of the application: one active and one passive. Figure 8-7 shows a passive cluster (called blue) running version 1.0 and an active cluster (called green) running 1.1. The new version is deployed to the passive environment; when it is ready, traffic is flipped to the new version, and it becomes active, while the previous version becomes passive. Like canaries, if the new version has problems, the traffic can be flipped back.

Launch in Dark Mode: Dark launches, sometimes called traffic shadowing, expose new code to real-life traffic without making it visible to end users at all.

In a dark launch, an application proxy sits between the live traffic and the application. The proxy duplicates requests to the dark system. Responses to these identical requests from both systems are compared, and differences are recorded. Only the production system’s responses are sent to the users.
The system is said to be in “dark reads” mode when only read traffic is sent to it and no data is modified. A system might be using the same datastore as the production system when operating in dark reads.
It is said to be in a “dark writes” mode when writes are also sent to the system and it is using a completely independent datastore.
Since operations are happening twice for the same request, once on the production system and once in the dark, you should take care to avoid duplication-related errors. Traffic to the dark system should be excluded from user analytics, and side effects like double billing have to be avoided.
Istio, Gloo, Diffy etc. are some of the tools which provide with this feature.

Note

Sources: