Git Design Pattern for Sandbox and Testing CI/CD Pipelines

Posted on Dec 26, 2020

The Problem

It is common to need a full sandbox, or testing environment that users can manually review to perform quality assurance, or business acceptance testing before release. This is especially relevant in fields like Business Systems Integration, where many highly configurable applications are being coordinated.

Continuous integration has made it easy to deploy changes by merging a new branch into main. It would be ideal to have a similar setup with a sandbox, but this becomes complicated.

Dedicated Sandbox Branch

The obvious solution is be to have a dedicated branch, which is set up to automatically build to your sandbox environment. The problem then becomes getting changes from your sandbox into production.

This can work if you are dealing with very small applications and few developers, but imagine if you have two features being worked on simultaneously. Both feature branches are merged into your sandbox branch. Testing passes for one of them, but the business is not ready to release the other.

What do you do? If you have not deleted the feature branch you could merge it into main, however, this is dangerous. The feature was tested in conjunction with some other new code you are not pushing. It is possible that there is a dependency between the two piece of code, which is why it works in sandbox, but if you only push one feature to production it will not work.

Specify the Branch to Deploy to Sandbox

If merging code into a sandbox branch causes problems the next obvious course is to take the sandbox branch out of the equation and instead update the continuous integration to deploy the branch of your choosing to your testing environment.

The problem here is that this is not so straightforward to do. CI/CD software is set up to deploy the main branch to production, but it is not designed around dynamic branches, not is it easy to update.

Even with something like GitLab CI, where the CI configuration is specified in a file committed to the repository updating the branch that builds to sandbox will lead to confusion about what code is currently running in sandbox. In order to deduce this you will have to look at the configuration of the CI that ran last. This will require digging through the CI logs and at any time someone else could update the CI in another branch, or just commit new code that triggers the CI and overwrite your code.

The Solution: Git Force Push

The solution I came across recently, implemented by Shopify’s kernel team, is to take advantage of git force pushes.

git push --force origin {your-branch}:{sandbox-branch}
git push --force origin fix/formatting:sandbox

This command overwrites the code in one branch with the code from another.

This enables the combination of the two previous strategies. On the one hand you can set up the CI to work with a single branch and can always easily see the code that is running in your testing environment. On the other hand you can ensure that only one update is being tested at a time and you do not run into any dependencies between updates.

Sandbox Claiming Process

In order for this to work there needs to be a process for claiming the sandbox for testing. If there is not it is easy for one developer to overwrite the code in sandbox during testing. I have seen a slack bot work well for this.

This does not, however, easily fit into git change management continuous integration flows. These are set up to create branches and then merge them into main with pull requests.