Management of environment files using CircleCI contexts

Everyone knows that pushing sensitive data to git repositories early or late may lead to security issues. Usually to avoid such situations all the files which may contain any sensitive information, developers add to .gitignore files which tells git that it should skip those files from adding to the repository and all the more pushing them to the remote one. There are many types of files which usually don’t add to repositories. Depending on used technologies they can be .env files which contain environment variables, various configuration secret files, certificates, private keys etc.

Actually if you are managing just a single monolithic app most likely you will not encounter any problem during maintenance of it. The management of environment files becomes quite complex when you have a microservice architecture with lots of services and have deployed them to multiple environments such as testing, staging, production. In this article I have described a mechanism which allows to facilitate management of .env files by using CircleCI contexts and a little bit scripting.

The official website of CircleCI gives the following definition of contexts:

Contexts provide a mechanism for securing and sharing environment variables across projects. The environment variables are defined as name/value pairs and are injected at runtime.

So to be brief, contexts allow us to define environment variables through the CircleCI dashboard and use them during job runs. It’s possible to create multiple contexts with different names and include various environment variables into it.

To get access to added environment variables during job runs it’s required to have context names defined in .circleci/config.yml file. It’s possible to add multiple context names to the same job which means all the variables from those contexts will be available during the job run.

Considering the following snippet of their documentation:

You can combine several contexts for a single job by just adding them to the context list. Contexts are applied in order, so in the case of overlaps, later contexts override earlier ones.

it must be said that CircleCI doesn’t group variables by contexts during job runs instead overrides earlier contexts with the same variable names. And this is one of the problems which will be resolved during implementation of the current mechanism.

To be simple let’s assume there is an api service which requires a .env file to be in the root directory of it. Also let’s assume there are testing and staging environments where the mentioned application is running.

So there are two environments which means it’s required to have two .env files defined. By navigating to the contexts section of the organization settings in CircleCI dashboard, you need to create two different contexts for each .env file with appropriate names.

Let’s name them this way:

Later by adding new variables into each of these contexts, they will automatically appear into the appropriate environment file.

Let’s create a folder named scripts in the .circleci directory of the project. In the new created scripts folder create a mappings.txt file with the following content:

Left side of each line in the file above is an environment file path and on the right side is defined a prefix which later in CircleCI contexts will be added to variable names.

Adding variables to contexts

As we already have created the mappings file it’s time to add environment variables to the contexts. Names of variables should comply with the following structure:

{service_prefix}_{variable_name}

where service_prefix is a prefix defined in the mappings.txt for the current environment and variable_name is an actual environment variable name which will be included in .env file.

After adding variables to the contexts, they should look like this:

All the variables defined in CircleCI contexts are stored as environment variables in the machines where jobs are running.

Purpose of creating shell scripts is to generate .env files in those machines and move them to external machines(servers) if needed.

Existence of the environment file for testing environment (.env.test) can be necessary in the current executer machine, for running automation tests. The environment file intended for staging is necessary for copying it from the executor machine to staging server.

Let’s create a script called generate_env_files.sh in .circleci/scripts directory. Here is the code example which should contain the script:

The script above does the following:

  • Reads mappings.txt and adds each line of it’s content as an element of the mappings() array
  • Loops over mappings() array and extracts prefixes and environment file paths
  • Using printenv and grep shell commands collects all the environment variables with current prefixes
  • Stores collected variables into the file which destination is stored in env_path variable

So after running the script above .env and .env.test files with environment variables will be created in the root directory of the project in the executer machine. Now tests can be run as .env.test file already exists in the root directory of the app.

The next step is moving environment files to the external servers. Let’s create a file called copy_env_files.sh in .circleci/scripts directory and place there the following code snippet:

The script above does the following:

  • Reads mappings.txt and adds each line of it’s content as an element of the mappings() array
  • Loops over mappings() array and by extracting environment file paths copies them from current machine to the staging server
Note: xx.xx.xx.xx should be replaced with the real ip address of the machine

Here is a sample CircleCI configuration which runs tests and deploy changes by automatically generating environment files.

There are two jobs defined in the yaml file above, test and deploy. Let’s go through the steps of each job and give a brief explanation of each one.

test

  • Line 27: Checking out current version of the project
  • Line 30: Generating environment files by executing generate-env-files.sh script
  • Line 33: Running app specific tests

deploy

  • Line 5: Checking out current version of the project
  • Line 8: Adding fingerprint of ssh key of staging server
  • Line 11: Adding IP address of the staging server to the known hosts
  • Line 14: Generating environment files by executing generate-env-files.sh script
  • Line 17: Copying just created environment files to the staging server by executing copy-env-files.sh script
  • Line 21: Accessing to the staging server and running appropriate commands for deployment

Also please take a look at 40 and 43 lines where contexts for each environment are defined.

Everything described in this article is a working example of the real application. We applied this method in one of our projects with microservices architecture.

Thank you for reading !