Trunk Based Development

Today, after the presentation in my company, I want to share with you a little bit about my approach to daily work as a developer. With my team we worked a few months in the system I would like to propose in this article — and with every day we discover new benefits of the new workflow.

Feature Branch Development

The classical approach to the development of a product is creating a new branch for every feature we take care of and maintain it as long as we can merge it with the mainline. In the meantime, we must check out to a hot-fix branch, resolve merge conflicts, remember about our branches, etc. This workflow is called Feature Based Development and it is shown in the picture below.

source: https://fpy.cz/pub/slides/git-workshop/#/step-1

This is an exampled flow of one developer work. Could you image how many branches we would have if our company grew to 100 developers? And what would happen if teams number countered to 100? There would probably be continuous merging development. And any merges often would end with conflicts. No one likes a merge conflict. When it occurs one needs to focus on his/her part of a code as well as on “theirs”.

Trunk Based Development

There is a rescue for merge hell but changes can take some time. Trunk Based Development rejects any feature branches, hotfix branches, parallel releases branches. There is the only one available for the developer branch — Trunk.

This new approach depends on four simple rules:

• there is one branch called ‘trunk’ where developers directly commit,
• a release manager can create release branches and no one can commit on those,
• developers commit small changes as often as they can,
• commits should be reviewed and tested and must not destroy mainline.

Please take a look at the picture below. Developers perform commits, as many as their can, on the ‘trunk` branch. Every commit is a small part of the code, e.g. one function or method with unit tests (green squares). At some point, when trunk branch contains every feature that we want, one person creates a new release branch from trunk. Such a person is called the release manager or the release engineer.

source: https://paulhammant.com/2013/04/05/what-is-trunk-based-development/

Typical cases in developers work

In our job, there are some typical cases such as:

• planning a new feature,
• creating a new feature,
• fixing a bug in production,
• code review,
• testing,
• merging/resolving conflicts,
• and drinking coffee

Let me take a closer look at all of these in context of trunk based development.

Planning a new feature

In my opinion, this case is the most important part of our working as developers. Before we will start coding we should consider how this feature would affect the other modules in our system. We should estimate how complex this task is and what we need to complete it (information, knowledge, testing cases or other resources). At my planning meetings I always ask:

• Can we split this story/feature into smaller tasks?
• Can we develop this story/feature parallel to the team?
• What do we have to write to complete every smaller task in this feature?

Answering these questions helps to visualize in my mind how the code for a feature would look like (classes, objects, functions etc). All my teammates have an idea of what is going to happen in my task as well. The information sharing is one of the keys to future success and fast code review (keep in mind that you will commit as often as you can). You review only code, not usage, context or architecture of the solution.

Wrapping up, planning session is extremely important in trunk based development. It helps to coordinate work, allows sharing information and it is great introduction to fast code review.

Creating a new feature

After planning our board should have many tasks, pretty small but well-describing new features. Now, we must remember that every code we write should be well-tested (it is a great opportunity to introduce TDD in your work). There is one core rule in this process — every day we should deploy a new commit to trunk. It is a progress symbol and also prevents any possible merge conflicts. So let us try to imagine a typical developer’s day when using trunk based development approach. In the morning the first thing will be fetching from the origin/trunk and creating locally a branch (creating it is not obligatory, it depends on your Continous Integration and review process). Now we should consider how to create a commit, as small as possible, which contains a working part of a feature. Also, keep in mind that no commit can break the main-line (the trunk branch). From every point of history, a release manager can create a release branch, so it requires that we cannot deliver any code of an unfinished feature and allow a user to enter it. I propose two clever techniques which can help with these problems: branch by abstraction and feature flag.

Branch by abstraction

This technique is extremely useful when a developer must replace components. Let us imagine that in our system we have a Memcached instance as a cache manager. Everything goes well, more and more clients (interfaces) have used the cache and today we have a situation like in the picture below.

We have only 3 components which depend on Memcached in our system, but in a real world, it is possible that the number of components will go huge. The first step is to create an abstraction layer between one client (e.g. LastUploadedChunk) and Memcached (please remember that in every day committing we must not break trunk). The current situation is illustrated in the picture below.

If our trunk is still ok, we can go a step further and move all clients to the abstraction layer. As we do so we have the opportunity to improve a unit test coverage of our code by implementing tests between the clients and the abstraction layer.

Now we can select one of the clients and start working on the new cache feature. We will still use the same abstraction layer but in two ways. First, for the chosen client we develop the new feature and, at the same time, the rest of the clients is supported by the old component (Memcached). In this way, we will still have the trunk working for our client with no difference, ready for a release, but we will be able to develop the new feature. The main question is how to create a route which is enabled only for development. We can use simple ‘if’ statement or something more elegant as feature flag (more information about this technique in next section). In the picture below we have chosen ‘LastGeneratedReports’ component for development.

After this most complicated step (I mean creating the new feature and connecting it with one client), we have the well-written component able to work with Redis. So the next obvious step will be removing the temporary route (made with ‘if’ statement or a ‘feature flag’) and connecting Redis with the rest of the clients. Again, remember that the clue point is doing as small commits as possible — the perfect way is one commit per client.

And that is almost the end. Now we can delete the old and no longer needed component — ‘Memcached’ with all unit tests for it. Also, we can remove the abstraction layer but it is not obligatory and in my opinion, it can be useful in the future.

In conclusion, branch by abstraction can be used in a few simple steps:

1. Create an abstraction layer for one client using old component.
2. Redirect all clients to use the abstraction layer.
3. Create a new component for only one client and connect them via the abstraction layer.
4. Develop the new component and test it.
5. Redirect the abstraction layer to use only the new component.
6. Remove the old component.

With this technique we can easily replace old and no longer required components with new ones without breaking the trunk line.

Feature flag

The second technique which is often used in Trunk Based Development is called the feature flag. One of the most common arguments for Feature Branch Development is that some features are developed longer than one day (a few commits), even for one release, so how to bring it together with TBD? We run into this issue quite a lot and feature flags are a handy tool to deal with it. The basic idea of this approach is to create a config file (or a database table) with a bunch of flags with a unique name. In such a file, all pending features have their own flags. The simplest example of the config file is a simple map (dictionary) or an object with static members. In bigger projects, we can store this flags in a database or even use an external library which simplifies this process (links at the end of this article). The most popular cases of using feature flags are dealing with a user interface. We just toggle the visibility of a feature. Sometimes we can add a simple ‘if’ statement in a code or, with more advanced cases and using more elegant way, we can use decorators or generators (in Python). The point where we check the flag is called a toggle point. Everything that is ‘hidden’ behind the ‘if’ is called a toggle test. There is only one rule about the toggle point and the feature flag in general. We should maintain, as far as possible, the minimum number of toggle points to ensure the new feature is properly hidden. Let us imagine the situation that we develop a web page generating reports. There may be many places where we display weekly reports on the page but, if there exists only one link redirecting to the page with reports, our toggle test should hide exactly this link. In other words, we should put the minimal amount of toggle tests associated with one flag. The situation is described in the picture below.

Another popular question about feature flags concerns a release testing. Shall we run tests on all possible combinations of flags or just a few of them? For integration tests it is enough to check only two scenarios:

• the toggles of all features expected to be in the next release are on,
• all toggles on, even in unfinished features.

One important mark — it is a good practice to remove flags of features which are completed and used in production. If we skip this step we will have a number of weird flags in the future and no one will remember who introduced them, when and what they showed or hid. Summarizing, we can introduce the feature flag technique in a few steps:

1. Create a new flag in a config file/a database.
2. Hide a new feature behind the flag.
3. Develop the new feature as long as it is required (it is still covered by the flag).
4. Prepare a list of flags for the current release.
5. Remove flags of features which are in production from the config file/the database.

Fixing bugs in production

It happens. It is sad but possible to catch a bug in a release branch. From the developer point of view we cannot commit directly to the release branch. No one can do it. What we can do however is locating the bug on the trunk branch and trying to fix it on the mainline with an additional commit. Be aware that the mainline and the release branch contain the same code (or similar) so it should not be a problem to reproduce any failure from the release branch on the trunk. After committing we will send the commit id to a release manager and he or she will cherry-pick it to the release branch (more in the next section).

Code review

“Code is like humor. When you have to explain it — it is bad”, said Cory House and the best judge of clarity of your code will be a code review (peer review). Do you remember that we do small commits with only one function or method and we plan all features together with the team? Great! So the conclusion can be only one: any review will be a piece of cake. The feature has already been well described by your team and you have planned how it should be done. It means that every day your team produces a few commits to be reviewed — just a few small commits which need your attention. It may sound not so well at the beginning but in practice it is done very fast — one function and tests. What is more, you are familiar with contexts of all features after planning, so basically you will check only technical aspects of the code. As a reviewer you will “lost” maximum 15 minutes a day. That is all.

Testing

This aspect of TBD is pretty simple. We must cover our feature with tests. There are two possibilities of testing. In the first one, we deliver unit tests for a newly created method or function. Remember, unit tests should cover only the tested function, other functions should be a stub or mock. Another group of tests covers an integration process. At some point, we might write a code which uses functions from a separate module. This connection should also be tested.

For a moment, let us try to look at testing from the perspective of a tester. We have a bunch of flags so we can easily turn on and off a feature. It is a great opportunity to introduce to our company A/B testing.

Merging and resolving conflicts

There is only one branch, there are not any other branches, there is no merging. No merge equals no conflict. This is one of the great benefits of Trunk Based Development.

Typical cases of release manager

The specific role of Trunk Based Development plays a release manager. This is the only person(s) who can create release branches and fix bugs in production. The release manager has just two responsibilities. The first one is creating a new release branch and the second is cherry-picking a possible hot-fix. Let me take a deeper look at those actions.

Creating a new release

… branch — I have intentionally skipped the word ‘branch’ in this section title. Why? There are two options how to properly create releases in Trunk Based Development. A choice should be based on how frequently the releases are made. If considering a company where new features are delivered to clients rarely (once a month or longer), a release manager should create a release branch for every minor version. We should keep one release branch alive for a long time because, in case of a bug, clients cannot wait a month or longer for a fix. This way we can easily increase a version of the release on the release branch with a cherry-picked hot-fix. Trunk Based Development supports semantic versioning (please take a look at bibliography section for a great article on this topic). This situation is visualized in the picture below.

Source: https://trunkbaseddevelopment.com/branch-for-release/

Teams with a very high release cadence do not need release branches. That is why I did not include ‘branch’ in the title above. They can use the trunk to perform a release. Teams often use commit ids or timestamps as versions of releases. It may also be a good idea to use a version control mechanism such as tags. Bugs can be treated as a normal feature and fixed on trunk. In this way the release becomes very fast. Here is an example flow.

Source: https://trunkbaseddevelopment.com/release-from-trunk/

Cherry-pick hot-fix

One of the most important rules of Trunk Based Development is: commit directly to mainline, so in case of a bug on release we cannot push changes into the release branch. So the correct way to fix the bug is reproducing it on the trunk, then perform a fix also on the main branch and now the release manager can pick this commit into release (I mean cherry-picked). Why do not commit directly into release branch and then merge it to the trunk? First, as remember, one of the main advantages of Trunk Based Development is no merge hell. If we introduce merging from release branches into mainline, we would have merge conflicts. Second, there is a chance you might forget to merge it down, and then there is going to be a regression at the next release moment.

Summary

Trunk Based Development is not a new idea, but nowadays it claims more popularity. If still, you have doubts about this workflow, may my last argument convince you — companies like Google and Facebook used this approach in their projects successfully and it is good moment for you to introduce Trunk Based Development in your company.

Bibliography

• Trunk Based Development
• What is Trunk-Based Development?
• Enabling Trunk Based Development with Deployment Pipelines
• Branch By Abstraction
• Branch By Abstraction by Martin Fowler
• Feature Flags
• Feature Toggles by Martin Fowler
• Feature Flags by Pete Hodgson
• 11 proven practices for peer review
• Semantic Versioning