Getting Started creating CI/CD Pipelines

1. Stages - what do you want it to do, when
2. Commands - command line instructions to execute those stages
3. Requirements - what software and credentials do you need to execute those commands
4. Configuration - configure your pipeline with what you figured out in steps 1-3

It is hard to know where to get started when you want to set up a build pipeline. If you are lucky there will be examples of what you are trying to do, but reading yml configuration files is confusing without a lot of context and every programming language, framework, project setup, pipeline software and have different requirements and capabilities. Dealing with many different flexible technologies that often have overlapping functionality is confusing.

Since you are trying to create a build pipeline it can be tempting to start by configuring your pipeline software, but this should actually be your last step. Configuration files, like yml, which most build pipelines are defined with, do not provide many prompts to help you get started. This can lead to reading circles in the documentation. Pipelines do usually define stages, though, and these are a good place to start.

Stages

What do we want our pipeline to do and when? At its core a build pipeline is automatically taking actions for you. We need to start by figuring out what those actions are. These actions can vary depending on the technologies you are using. At a minimum it is a good idea to have two stages: Test - run your automated tests before merging code to ensure the changes you are pushing will not break your application Deploy - Once your code is merged deploy it to the server

If you are using a compiled language you may also want a build phase, to compile your new code that needs to run before testing, or deploying it. If you have a sandbox you may want to add a stage to build your code to your sandbox.

Commands

Now that we know what you need your build pipeline to do we need to figure out how to make a computer do it. Build pipelines generally use bash to execute, so we need to figure out the appropriate bash commands.

For example Django applications execute tests with the command: python manage.py test. Java applications managed with maven use the command: mvn test. Figure out how to test your application on your local computer and write down what those commands are.

You will also need to figure out how to deploy your application using the command line. This will vary based on where you are hosting it. Look through your host’s documentation to understand how to deploy application updates to their platform.

Figure out how to use your terminal to execute all of the stages you defined in step 1.

Requirements

What did you need to install, or configure in order to run the commands in step 2? This could include credentials for your host, software for running your tests, software for deploying to your hosting provider. You are going to need to tell your pipeline about all of these in order for it to function properly.

Configuration

Now we have all the pieces. It is time to put them all together into our pipeline!

Lets start at the beginning again with stages. Most pipeline software enables you to define different stages with different triggers and dependencies. Look back at the stages you defined in step 1 and add them to your configuration.

Next add your commands to each of the stages. If multiple stages use the same command, or there are a lot of commands consider using a .sh script, so that you only have to have one command in your pipeline configuration to run the .sh script, sh {path-to-script}/{script-name}.sh.

Finally ensure that the build pipeline has all of the requirements to execute your commands.

Software - frequently you are able to specify a docker image in the pipeline configuration that can either get you started, or have all of the requirements you need. For example if you are deploying using maven there is a maven docker image you can use. You can also add to your stage commands. For example with python you will usually have a requirements.txt file containing all of the packages necessary to run your code that you can install with the command pip install -r requirements.txt.

Credentials - your build pipeline will probably need credentials to deploy your software. You will need to store these securely and give the pipeline access. Frequently build pipeline software gives you places to define secrets like this and reference them through environment variables in your pipeline. This setup may again require modifying your commands to use the secure variables in your pipeline.