Given a Java application that runs in an EC2 instance, we want to enable it to call AWS services. This is useful for example to call services like Secrets Manager to retrieve secrets.
Authentication with AWS services when running locally
When running locally, the SDK has several ways of obtaining the credentials. We will describe two of these alternatives.
Specifying credentials via the credentials file
This is the recommended approach. This option involves setting the credentials in the AWS credentials profile file on your local system, located at:
~/.aws/credentials on Linux, macOS, or Unix
C:\Users\USERNAME\.aws\credentials on Windows
This file should contain lines in the following format:
Authentication when running inside an EC2 instance
When the application runs in an EC2 instance, the AWS SDK authenticates via the instances’ role. For each service you want to use, you’ll need to attach a policy (i.e. a rule) to the role. For example, to use AWS Secrets Manager, you need to add the policy SecretsManagerReadWrite to the instance’s role.
Adding the Java SDK to the project
You can either add the specific modules you are going to use (recommended approach) or you can add the entire AWS Java SDK as a dependency.
To add a specific module, check the exact name and latest version of the module you are looking for in https://mvnrepository.com/artifact/com.amazonaws. For example, for AWS Secrets Manager, the dependency to add, at the time of writing is:
Our objective is to build an tool that can build a given application on the cloud, producing an artifact, and deploy it to an EC2 instance on AWS. For this tutorial, the application is a Java program stored in a Github repo and built as a Maven project that produces a jar file. The AWS instance we will deploy to is a preexisting EC2 instance. Our automation will be triggered by our commits, so that every time we commit to the master branch of our repo, it will automatically build the application and deploy it to our instance, following the principles of continuous integration and continuous deployment (CICD).
What you’ll need
A Github repo containing a Java Spring web application with standard maven structure. The repo doesn’t need to be public.
A working AWS account.
An EC2 instance in that AWS account, running Linux.
An S3 bucket in that AWS account, to use as artifact repository and to store logs.
All resources used in this tutorial are available in the AWS Free Tier, so if the account is less than a year old and you only have one EC2 instance running at any given time, there’s no cost involved.
Architecture overview
The following diagram illustrates the components we will use to solve this problem and how they interact with one another.
As we can see, our components are organized into four groups:
Code repository: a version control system that stores our code. In this tutorial, this will be a Github repo.
Continuous Integration automation: a tool that runs on the cloud and builds the source code, generating an artifact. In this tutorial, our continuous integration automation will be an AWS Code Pipeline. We will refer to this as Continuous Integration Pipeline or CI Pipeline for short.
Artifact repository: a server that stores the generated artifact and allows to retrieve it later. In this tutorial, our artifact repository will be an AWS S3 bucket.
Continuous Deployment automation: a tool that runs on the cloud and deploys the artifact generated earlier to the target host and puts it to run. In this tutorial, our continuous deployment automation will be another AWS Code Pipeline. We will refer to this as Continuous Deployment Pipeline or CD Pipeline for short.
A key tool here is AWS CodePipeline. A code pipeline is a predefined process made of stages, which are in turn composed of actions. Each stage executes a step of the process (e.g. downloading code from a repo) and can produce as output any number of files, called artifacts. Artifacts are produced as output in a stage and can the be used as input in the stages that follow. When we create a code pipeline in AWS, AWS automatically creates an S3 bucket that the pipeline will use to store its internal artifacts so they can be shared between its stages. Note this is not the same as our artifact bucket. Whereas the code pipeline’s internal bucket is created automatically and used to share artifacts between stages of the same code pipeline, our artifact bucket is separate from the code pipelines, will be created manually and used to share an artifact between the two pipelines.
Our CI pipeline will be composed of the following stages:
Source stage: downloads the code from our Github repo. This stage produces a zip file with the contents of our repo as its output artifact.
Build stage: builds the code, generating an artifact. This stage uses as input the output artifact of the source stage. The output of this stage is our application artifact, which is a zip containing the jar file of our application and some metadata.
Upload stage: uploads the application artifact generated in the build stage to an artifact repository. In this case, the artifact repository is an S3 bucket.
Our CD pipeline will be composed of the following stages:
Source stage: downloads the application artifact from the artifact repository.
Deploy stage: copies the application artifact to a target EC2 instance and starts the application on it (previously stopping the process if it was already running).
Repository directory structure
For this tutorial, we will use the following Github repo as an example: aws-cicd-poc. The directory structure of this repo can be seen in the following picture:
As we can see, the repo contains:
A folder called basicsprintbootwebapplication, which contains the skeleton code of a Spring Boot application generated using Spring Initializr.
A folder called cicd, which contains the following files:
buildspec.yml: we will use this file when creating the CI pipeline. More details below.
appspec.yml: we will use this file when creating the CD pipeline. More details below.
start_server.sh: a script we will use to start the application inside the EC2 instance.
stop_server.sh: a script we will use to stop the application inside the EC2 instance.
Note: In your repo, you can place these files wherever you want. When you create the pipelines, you’ll tell AWS the paths to find them in your repo.
Buildspec file
The buidlspec is a yaml configuration file whose purpose is to tell the CI pipeline how to map from the files in your repo to the files you want to include in the application artifact. In this tutorial, we are using a buildspec file that contains the following:
The commands section tells CodePipeline what are the commands required to build the project once the source code is downloaded. Our repo is a standard maven project, so we just move to the project directory and call mvn package, which generates a jar file in the target directory.
The files section tells CodePipeline which files from the repo we want to include in the output artifact. In our case, we include the jar of our application, the start and stop scripts, and the appspec.yml file. This appspec.yml file will be required for the deployment stage later on.
The discard-paths: yes option tells AWS to leave all these files in the root folder of the output artifact. You can choose to create a directory structure in the artifact if you whish, and the buildspec allows you to indicate where to put each file in the directory structure (see this page for the details of how to do this). If you choose to do this, keep in mind that the appspec file must always be in the root of the artifact in order to be able to deploy it using CodeDeploy (more details about this below).
Our application artifact will be a zip file with the following content:
basicspringbootwebapplication-0.0.1-SNAPSHOT.jar
start_server.sh
stop_server.sh
appspec.yml
CI pipeline
To create the CI pipeline, follow these steps:
In the AWS console, go to Services / Developer Tools / CodePipeline / Create pipeline.
In Pipeline name, enter a suitable name for your pipeline. In this tutorial, we will use set the name basicspringbootwebapplication-ci-codepipeline.
In Service role, select New service role. This is the role that will be used when CodePipeline runs.
In Role name, enter a suitable name. Make sure the name is at most 64 characters long, otherwise AWS may let you move forward with the process but show an obscure error at the end when it tries to create the code pipeline.
Click Next.
In Source provider, select Github.
Click on Connect to Github.
A dialog will pop up where you’ll need to enable AWS to access your Github account. This is required in order for the CI pipeline to be able to download the code from your repo.
In Repository, select which of the repos in your account you want to use.
In Branch, enter the name of the branch you want to build from. This is usually master.
Leave the rest to defaults and click Next.
In Build Provider, select AWS CodeBuild.
Click on Create project. This will open a new window with a dialog where we will create a CodeBuild project to tell AWS how to find our buildspec file in the repo. Once we are done there we will return to the CodePipeline wizard. The following steps describe how to set up the CodeBuild project.
In Project Name, enter a suitable name for the CodeBuild project. In this tutorial, we will set the name basicspringbootwebapplication-CodeBuildProject.
In Environment / Operating System, select Ubuntu.
In Runtime, select Standard.
In Image, select the latest version number available.
In Service role, select New service role. This is the role with which AWS will run when building.
In Role name, enter a suitable name. Make sure the name is at most 64 characters long, otherwise AWS may let you move forward with the process but show an obscure error at the end when it goes to create the CodeBuild project.
In Buildspec, select Use a buildspec file.
In buildspec name, enter the path to the buidlspec.yml file relative to the root if your repo. In our example repo, this is basicspringbootwebapplication/cicd/aws/ec2/buildspec.yml.
In Logs, you choose where to store the build logs. You can use CouldWatch, S3, or both. For this tutorial, we will uncheck CloudWatch and check S3. In Bucket, select the S3 bucket you want to use to store the logs. In Path prefix, enter a suitable filename prefix for your build logs, such as basicspringbootwebapplication.
Click on Continue to CodePipeline to return to the CodePipeline wizard.
Leave the rest to its defaults and click Next.
In Add deploy stage, click Skip deploy stage.
Click on Create pipeline.
Now our code pipeline has been created. Open your pipeline in the pipeline list. It has two stages, Source and Build. We are still missing a stage, the Upload stage, where we take the generated artifact and store it in our artifact repository.
Click on Edit, and then click Add stage at the bottom. Note it’s important to click the last Add stage button, so that we add the stage at the bottom.
In Stage name enter Upload, and click Add stage.
In the Upload stage, click on Add action group to add an action.
In Action name, enter Upload.
In Action provider, enter Amazon S3.
In input artifact, select Build artifact. This tells AWs that the artifact we want to upload is the one generated previously in the build stage.
In Bucket, select the S3 bucket you want to use as artifact repository.
In S3 object key, enter the path where you want to store the artifact in the bucket. Take note of the path you enter here, you will need it later. In this tutorial we will use artifacts/basicspringbootwebapplication/snapshot/basicspringbootwebapplication-SNAPSHOT.zip.
Leave the rest to its defaults and click Done.
Click Save to save your stage addition to the pipeline.
You are all set! Click on Release change to run the CI pipeline manually. If it runs without errors, you should have the build artifact stored in the S3 bucket in the path you chose.
Note: if you open the CodeBuild project in the AWS console, you’ll note that it has the source field set to CodePipeline. This means that the input artifact of the build stage will the output artifact generated in the previous source stage (a zip file containing the content of your Github repo) which is what we want. The CodeBuild project is set up this way because we created it from inside the CodePipeline wizard, which automatically sets the source field to CodePipeline. This is the only way to set the source field with that value. If you create the CodeBuild project from the CodeBuild wizard, the source field won’t have CodePipeline in its available options.
The CI pipeline created this way will run every time you push a commit to the master branch (or whatever branch you specified in the CodeBuild project). You can also run it manually any time you want by clicking on Release change. Every pipeline execution will generate a new application artifact and store it in the S3 bucket, overwriting the previous artifact.
Troubleshooting tips
If a stage of the pipeline fails when executing, CodePipeline will stop the pipeline at that point and will not run the stages that come after it. To find out what happened, click on Details for the stage that failed. This will take you to a page that is specific for the AWS service the stage is based on. Some examples:
If the stage that failed is CodeBuild, the details page has a Logs tab that lets you download a log file of the build, where you can check why it failed.
If the stage that failed is CodeDeploy, the details page has a View events link where you can see the steps f the deployment and check where it failed.
CD pipeline
When it comes to deploying the application artifact to the infrastructure where it will run, there are several approaches, depending on how the architecture of the system. Some possible approaches are:
Copying the artifact to an existing EC2 instance and run it there. This can be done using a tool like AWS CodeDeploy.
Creating a new EC2 instance for the application to run on. This can be done using a tool like AWS CloudFormation.
Creating a container for the application to run on. This can be done using a tool like Docker, and the creation of the container within a cluster can be orchestrated using tools like Kubernetes and AWS EKS.
In this tutorial, we will use approach 1. The purpose of our CD pipeline is to copy the artifact to the instance, stop the running process (in case it was already running) and start a new process from the new binary.
When we add CodeDeploy as a stage in a code pipeline, the stage will carry out the following steps, in order:
Find the set of instances to deploy to, based on a criteria that we will specify (in this tutorial, there is only one instance, but this is not required by CodeDeploy).
Copy the artifact to the instance(s).
Extract the artifact inside the instance(s) and copy the files to their target locations in the file system, based on a mapping that we will specify.
Stop the application if it is running, by calling a script that we will provide.
Start the application from the new binary, by calling a script that we will provide.
The following sections go over the steps we will need in order to set up the CD pipeline.
In order to deploy an application with CodeDeploy, we must specify three things:
The application artifact to deploy (also known in AWS as a revision). This artifact must contain a metadata file called appspec.yml which AWS will read to know how to work with the artifact (more details about this below). We will specify the revision by providing the S3 bucket where our revision is stored.
The deployment configuration we will use. This means indicating to AWS to how many instances your application revisions should be simultaneously deployed and to describe the success and failure conditions for the deployment.
The set of instances to deploy to. This is specified in what AWS calls a deployment group, which is a set of rules that defines the set of instances to deploy to.
We will specify these three elements by creating a CodeDeploy application. In CodeDeploy, an application is a name that functions as a container to ensure that the correct combination of revision, deployment configuration, and deployment group are referenced during a deployment.
Appspec file
In order for an application to be deployed using CodeDeploy, the application artifact must contain a file named appspec.yml in the root of the artifact. Note you can lay out the rest of the files in the artifact however you want, but the appspec must always be in the root. The function of the appspec file is to tell CodeDeploy how to map from the files in your artifact to the files that must be deployed in the instance. At at minimum, this involves indicating the path to our executable and how to start and stop the application.
The files element contains a list of files from the artifact that we want to copy to the instance. In our case, it’s just one, the jar file. The source field indicates the path to the file in the artifact relative to its root. The destination field indicates the parent folder you want the file to be copied under (note it’s not the path, it’s the parent folder).
The ApplicationStart hook tells CodeDeploy where to find the script to start the application relative to the root of the artifact.
The ApplicationStop hook tells CodeDeploy where to find the script to stop the application relative to the root of the artifact.
In this case, we are not copying the scripts to any other location within the instance, we will just run them from the directory where the artifact was extracted, which is enough for our purposes. If we wanted to copy the scripts and run them from another location, we would need to include it in the files element.
The application jar fle does need to be copied, so that we have a clear location to reference it from the start script (more details about this below).
CodeDeploy agent
Remember we are going to use AWS CodeDeploy to deploy our application to an EC2 instance. In order to be able to do this, CodeDeploy requires that the instance is running the CodeDeploy agent. The agent is a service process that runs in the instance and acts as the interface between the instance and CodeDeploy. Installing the CodeDeploy agent only needs to be done once per instance. You can find instructions on how to install the agent on this page.
Tagging the instance
CodeDeploy determines the set of instances it will deploy to based on a condition specified in a deployment group. The set of all instances that meet this condition is the set CodeDeploy will deploy to. The condition can be expressed in a number of ways. In this tutorial we will select based on the existence of a tag in the instance, so the deployment set is the set of instances that have that tag (which in this tutorial is composed of only one instance). In order for this to work, we need to add the tag to the instance we want to deploy to. This only needs to be done once per instance.
In the AWS console, go to Services / EC2.
Select Instances in the side panel.
Look for the instance you want to deploy to and click on it.
Go to the Tags tab and click on Manage tags.
Add the following tag:
Key: basicspringbootwebapplication
Value: (leave empty)
You can use whatever string you like as key as long as it’s the same string you input later on when you create the deployment group (more details about this below).
Start script
This is what our example start script looks like:
#! /bin/bash
SERVICE_HOST=localhost
SERVICE_PORT=8080
STARTUP_WAIT_TIME=10
APPLICATION_JAR_PATH=/basicspringbootwebapplication/basicspringbootwebapplication-0.0.1-SNAPSHOT.jar
LOG_FILE_PATH=/basicspringbootwebapplication/general.log
APPLICATION_MAIN_CLASS_NAME=com.sgonzalez.basicspringbootwebapplication.App
echo "Starting application..."
nohup java -jar $APPLICATION_JAR_PATH $APPLICATION_MAIN_CLASS_NAME > $LOG_FILE_PATH 2>&1 &
sleep $STARTUP_WAIT_TIME
nc -z $SERVICE_HOST $SERVICE_PORT
if [ ! $? -eq 0 ]; then
echo "Could not start application"
exit 1
fi
echo "Application started OK"
Key takeaways:
An & at the end makes the application run in the background.
Starting the process with nohup lets the java process run in the background even after the user is logged out.
Ten seconds after starting the process we check if it’running by attempting a connection to local port 8080 with nc -z and if this fails we fail the script. This is important so that if the application fails to start for whatever reason, CodeDeploy will fail the stage and CodePipeline will let us know that there was an error.
Stop script and how to do graceful shutdown with the Spring Boot Actuator module
Every time we deploy a new version of the application to our instance, CodeDeploy will call our stop script which is expected to do a graceful shutdown of the process. In this tutorial, we will use the Spring Boot Actuator module to do the graceful shutdown. Actuator is a module that runs inside the application and exposes a URL (/actuator/shutdown) that shuts down the process when we send an HTTP POST to it. There are two approaches regarding how to call this URL:
HTTP (unsecured request). Any HTTP POST request to the URL is accepted. This is a security concern since an attacker that can get a request through to this URL can kill your application. This is simple and useful for development purposes, but it’s not suitable for a production system.
HTTPS (secured request). The request is only acted upon if it carries correct authentication information. This is the best approach for a production system. It requires some extra set up steps:
The application must have credentials defined for the URL, which should be stored in some credential manager service like AWS Secrets Manager or Hashicorp Vault.
The stops script must retrieve these credentials from said credential manager service and send them in the request to the shutdown URL.
For the sake of simplicity, in this tutorial we will use approach 1. In order to do this, we include the following in the application.properties file:
# In order to access the actuator endpoints using HTTP, we need to both enable and expose them. Here, we expose all endpoints.
management.endpoints.web.exposure.include=*
# Explicitly enable the shutdown endpoint
management.endpoint.shutdown.enabled=true
endpoints.shutdown.enabled=true
endpoints.shutdown.sensitive=false
In the stop script, we call the Spring /actuator/shutdown endpoint over HTTP. The following is an example of a viable stop script:
#! /bin/bash
SERVICE_HOST=localhost
SERVICE_PORT=8080
pid=`ps aux | grep -i <my_app_name> | grep -v grep | awk '{ print $2 }'`
if [ ! $pid ]; then
echo "Process not runing, nothing to do"
exit 0
fi
echo "Sending shutdown request to service..."
curl -X POST http://$SERVICE_HOST:$SERVICE_PORT/actuator/shutdown
curl_result=$?
echo -ne "\n"
if [ ! $curl_result -eq 0 ]; then
echo "Could not do the graceful shutdown of the service. This may mean that we couldn't send the shutdown request. We will kill the process anyway"
echo "Curl exit code: $curl_result"
fi
kill $pid
kill_result=$?
if [ ! $kill_result -eq 0 ]; then
echo "Could not kill process. You should kill it manually"
echo "Kill exit code: $kill_result"
exit 1
fi
echo "Service stopped"
Key takeaways:
The configuration above enables and exposes the shutdown URL.
Note there’s nothing in the configuration that sets the shutdown URL as unsecured. It is unsecured because we intentionally left out the Spring Security module in the pom file, so all URLs are unsecured by default. If we add Spring security later, the shutdown URL will be secured by default and the above command will fail with 401. In that case we’ll need to either explicitly configure the shutdown endpoint as unsecured, or have the stop script pass the credentials.
Enabling versioning on the artifact bucket
In order to use an S3 bucket as source for a deployment, the bucket must be versioned. To enable this, navigate to the bucket we are using as artifact repository in the AWS console, open the properties tab, go to Versioning and select “Enable versioning”.
Installing appropriate version of the Java JRE on the instance
If your application’s pom.xml is configured to build with the 1.8 version of the JDK or above, you’ll need to install the 1.8 version of the JRE or above on the instance in order to be able to run it.
Follow these steps to install the Java 1.8 JRE on an Amazon Linux 1 EC2 instance:
$ sudo yum install java-1.8.0-openjdk
$ sudo alternatives --config java # Follow the instructions prompted and set java 1.8 as the alternative to use
Note: the purpose of the alternatives command is to configure the system so that the command java will run the java 1.8 binary. The command above is setting /usr/bin/java as a symbolic link to /etc/alternatives/java, which in turn will point to /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java which is the actual binary we want to call. We are using Linux alternatives system here. Without using the alternatives system, you can get the same effect by just adding the directory where the JDK 1.8 is installed (/usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java) to the beginning of the PATH, so that it comes before /usr/bin. In these instructions we have used the alternatives system instead. Note though the alternatives system requires configuring each command individually, whereas using PATH overrides all JDK commands to point to whatever directory is found first in the environment variable.
Creating the CD pipeline
Follow these steps to create the CD pipeline:
In the AWS console, go to Services / IAM.
On the side panel, click on Roles.
Click on Create role.
In Select type of trusted entity, select AWS service.
In Choose a use case, select CodeDeploy.
There are several variants of use cases involving CodeDeploy. In Select your use case, click on CodeDeploy.
Click on Next: Permissions. The AWSCodeDeployRole managed policy will already be attached to the role.
Cick on Next: Tags.
Click on Next: Review.
In Role name, enter a suitable name for the role. In this tutorial we will use codedeploy-role. Take note of this role name, we will use it later when we create the deployment group.
Click on Create role.
In the AWS console, go to Services / Developer Tools / CodeDeploy.
On the side panel, click on Applications, then on Create Application.
In Application name, enter a suitable name for the set of conditions we will use to specify the instances o deploy to. In this tutorial, we will use basicspringbootwebapplication.
In Compute platform, select EC2/On premises.
Click on Create application.
This will take you back to the application list. Click on the application you just created.
Click on Create deployment group.
In Deployment group name, enter a suitable name for the deployment group. In this tutorial, we will use basicspringbootwebapplication-deployment-group.
In Enter a service role, enter the name of the service role you created earlier.
In Deployment type, select In-place.
In Environment configuration, select Amazon EC2 instances.
Now we need to enter the actual condition based on which instances will be selected to deploy to. The condition is a set of tags. The set of instances that the application will be deployed to is the set of instances that have all the tags specified, with the same values (if given). In this tutorial, we will match based on only one tag, regardless of value. Add the following tag:
Key: basicspringbootwebapplication
Value: (leave empty)
In Matching instances, check that your instance shows up. This is the instance CodeDeploy will deploy to.
In Install AWS CodeDeploy Agent, select Never. We can skip this step because we already installed the CodeDeploy agent previously (if you haven’t done so, do it now using the instructions above).
In Deployment configuration, select CodeDeployDefault.OneAtATime.
In Load balancer, uncheck Enable load balancing.
Click on Create deployment group.
Now that we have an Application and a deployment group created, we will create the cd pipeline using them.
In the AWS console, go to Services / Developer tools / CodePipeline.
Click on Create pipeline.
In Pipeline name, enter a suitable name for your pipeline. In this tutorial, we will use set the name basicspringbootwebapplication-cd-codepipeline.
In Service role, select New service role. This is the role that will be used when CodePipeline runs.
In Role name, enter a suitable name. Make sure the name is at most 64 characters long, otherwise AWS may let you move forward with the process but show an obscure error at the end when it tries to create the code pipeline.
Click Next.
In Source provider, select Amazon S3.
In Bucket, enter the name of the S3 bucket you configured as artifact repository in the Upload stage of the CI pipeline.
In S3 object key, enter the path to the application artifact within the artifact repository as you configured it in the Upload stage of the CI pipeline.
In Change detection options, select Amazon CloudWatch events. This makes sure that whenever we update the artifact in the artifact repository, the CD pipeline will run automatically to deploy the new artifact.
Click Next.
Click Skip build stage. Our CD pipeline doesn’t need a build stage, because we have already set up the build separately as part of the CI pipeline.
In Deploy provider, select AWS CodeDeploy.
In Application name, enter the name of the CodeDeploy Application you created earlier.
In Deployment group, select the deployment group that you created earlier.
Click Next.
Click Create pipeline.
You are all set! Click on Release change to run the CD pipeline manually. If it runs without errors, the EC2 instance should be running your application.
If the pipeline fails, see the troubleshooting tips above for some pointers on how to investigate.
On this post we will present a basic introduction to Docker. By the end you should have an understanding of its main concepts and usage scenarios.
First, let’s establish some groundwork on virtualization.
Hypervisor virtualization
In this mode, one or more independent machines run virtually on physical hardware via an intermediation layer.
Each virtual machine has its own operating system. This provides the possibility of the guest operating system being completely different than the host’s operating system, at the cost of additional overhead since we are running one kernel on top of another.
Common tools that follow this approach are VirtualBox and VMWare.
Operating system-level virtualization (also called container virtualization)
In Operating system-level virtualization, several isolated user space instances (called containers) run on top of a host operating system’s kernel. Each user space instance consists of a set of processes which are isolated in such a way that from the point of view of the processes inside the container, they are the only processes running on the system.
Containers do not require an emulation layer or a hypervisor layer to run and instead use the operating system’s normal system call interface. In other words, the processes that run inside a container are just normal processes in the host kernel (you can see them in the host with ps). This reduces the overhead required to run containers and can allow a greater density of containers to run on a host, since there is only once kernel involved.
Since containers use the underlying kernel, they are limited to run applications which are compatible with said kernel.
Docker is an example of a tool based on this approach.
What is Docker?
Docker is an open-source virtualization engine based on operating system-level virtualization. It provides tools that allow to create containers by assembling applications (called packages in Docker terminology).
Relationship with Kubernetes
While both Docker and Kubernetes are tools for container automation, they operate at different levels.
Docker is a tool for creating a container by combining a set of packages.
Kubernetes is a tool for creating a muti-application system by combining containers. For this reason, it’s common for large software projects to use both tools in combination, since the latter is based on the former.
Installation
In order to play around and do tests with Docker on your local machine you will need to install it. For this we will refer you to the official documentation on the matter, which you can find here.
Also, if you are working on a Linux system, it is practical to set up Docker so that you can run docker commands without using sudo, by adding your user to the docker group. You can find the instructions on how to do that here. Keep in mind this makes it easier to play around in your development environment, but should be done with caution when in a production environment. Any user that belongs to the docker group can run Docker without sudo.
In the sections that follow, all command examples will be provided without sudo.
Once you complete the installation, try the following command which shows general information about the docker daemon running on your host:
$ docker info
If you see an empty table in the output, you are all set! Docker is successfully installed on your system. The output is empty because we haven’t created any containers yet. We will get to that shortly. Let’s go over some concepts first.
Getting help
The first docker command you should know is the help command, which is used to access the official documentation:
$ docker help <command>
For example, to read the documentation of the docker run command, use:
$ docker help run
or alternatively:
$ man docker-run
Components of Docker
The Docker installation is composed of a root-privileged daemon (dockerd) that takes care of operations that can’t be performed by normal users, and a client program (the docker binary) which takes commands from the user and executes them by sending them to the daemon.
Usually when we run docker commands we are working against the locally running daemon. This is the default and it’s what the docker client does when we don’t specify a daemon address explicitly. We can connect to a daemon running remotely by specifying its address to the docker client, like so:
$ docker -H tcp://<daemon_ip_address>:2375
We can also specify the address of the daemon using the DOCKER_HOST environment variable:
Containers, Images, Tags, Repositories and Registries
Containers in Docker have an id and a name. Both of these values are unique and can be used interchangeably in commands that take a container as argument.
A Docker image is a read-only template that contains a set of instructions and file system data necessary for creating a container that can run on the Docker platform. Every container is created from a given image.
It’s possible to create an image by starting from an existing image and adding things on top of it. When we do this, the image we start from is called the base, and the image we create is said to have two layers. Note any image can be used as a base to build another image.
Images are identified by an image id. An image can have any number of tags, which are labels that help to differentiate image versions. The docker images command lists the images that are stored on the local cache. Note that the output of this command may show the same image on several lines, one line per each tag. What the command is listing is really tags, not images.
Images live on repositories. A repository is like a folder containing several images. In docker commands that accept an image as parameter, it’s possible to specify them as <repository_id>:<image_tag> or simply <repository> where the tag is understood to be latest. Repositories don’t need to be created explicitly, instead they are “created” implicitly when the first image referencing them is created. It’s common for repository names to begin with the username of their owner, e.g. jamtur01/apache2, but this is not mandatory.
Repositories live on registries. A registry is a server containing many repositories. The default registry is the public registry managed by Docker, Inc., Docker Hub.
Docker and the Union file system
A Union File System is a mechanism that allows to create a file system by stacking several file systems on top of each other. When two file systems have a file in the same path, the topmost version (i.e. most recently mounted) is the one that is used, and the others are “hidden” as if they didn’t exist.
Each docker container has an isolated file system able to hold its own state. This is accomplished by giving each container a Union File System. The base is a boot file system, which contains the files necessary to boot. On top of that Docker adds a root file system, which contains the files that make up a given Linux distribution (e.g. a Debian or Ubuntu file system). The root file system is mounted read-only, and Docker mounts a read-write file system on top of the root.
Docker images are also based on a Union File System. When a container is created, the layers of the image it is created from become the layers of the container’s file system. The initial read-write layer of the container is empty. Note the data in the image’s files is not duplicated. You can have several containers created from the same image and all the common files will be stored only once. When a file is modified, it is copied from the read-only layer into the read-write layer on top of it. The read-only version of the file still exists but it is hidden underneath the copy.
This pattern is usually called “copy on write”, and it is one of the reasons Docker containers are more lightweight that full-fledged virtual machines.
Basic workflow
In this section we will go over a simple example that illustrates the life cycle of a container.
As we said above, in order to create a container we need to specify an image to create it from. We use the docker images command to list the images currently installed in our local cache:
$ docker images
Since we haven’t downloaded any images yet, the output will show an empty table.
We can search for images in the Docker Hub registry with the docker search command:
$ docker search ubuntu
We use the docker pull command to download images to our host:
$ docker pull <repository>:<tag>
The tag can be omitted, in which case docker will default to the tag latest. Let’s download the ubuntu image in its latest version:
$ docker pull ubuntu
Once the command finishes the download, you will be able to see the Ubuntu image listed in your local cache:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu latest 8e428cff54c8 4 months ago 72.9MB
You can use the docker inspect command to get details about the image:
$ docker inspect ubuntu
This displays a json document showing information such as the creation date of the image, its parent (if any), its author, etc.
We use the docker run command to create and start a container:
$ docker run -i -t --name <container_name> <image> <command>
The first thing to understand about the run command, is that it does several things at once. The above command will create a container, start it, attach to it, and run the given command inside the container. Attaching to a container allows you to view its ongoing output or to control it interactively, as though the commands were running directly in your terminal. You can attach to the same contained process multiple times simultaneously, from different sessions on the Docker host.
The -i option instructs docker to keep the standard input of the container open in our terminal, even if the container is detached. The -t option tells docker to allocate a pseudo-tty, that connects our terminal (running on the host) to the process running inside the container. For interactive processes (like a shell), you must use -i -t together in order to allocate a tty for the container process. -i -t is often written -it.
The --name option specifies the name for the container. This option can be omitted, and docker will assign a random name.
The image can be specified with an image id, or with a <repository>:<tag> combination.
The <command> argument can be omitted, which will make docker start the container with the image’s default command (more on how this default command is specified later). Note that when we specify the command, the path we are giving refers to the container’s internal file system.
For example, let’s start a container from the ubuntu image, run bash inside it, and attach to the bash process from our terminal:
$ docker run -it ubuntu /bin/bash
This will attach your terminal to the bash process running inside the container. You can enter some commands and see their output in your terminal.
Now in another terminal, enter the following command:
$ docker ps
You will see an output similar to this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
707a2d53267a ubuntu "/bin/bash" 4 seconds ago Up 1 second trusting_hypatia
The docker ps command shows the containers that are running on the host. Note the output shows, among other things, the container id, which we will use as input to other commands that operate on containers.
You can also use docker container ls as an alternative to the docker ps command.
You can list the processes running inside the container with the docker top command:
$ docker top <container_id>
You will see an output similar to this:
UID PID PPID C STIME TTY TIME CMD
root 8064 8026 0 18:50 pts/0 00:00:00 /bin/bash
You can see the recent output of a container with the docker logs command:
$ docker logs <container_id>
You can add the -f option to continue monitoring the output in real time.
When a container is running, it can be paused with the docker pause command (get the container id from the output of docker ps):
$ docker pause <container_id>
If you go back to the terminal where you started the container, you will notice that it appears to be blocked, not reacting to text input. When a container is paused, its processes are frozen in memory and don’t receive the CPU. Let’s resume the container with the docker unpause command:
$ docker unpause <container_id>
This puts the process to run again, picking up at the point where it left off. Note that the bash process remained in memory even when it was paused, so it retains all its state.
Now, in a different terminal from the one where you started the container, enter the following command:
$ docker stop <container_id>
The docker stop command sends a SIGTERM signal to the main process running inside the container, and after a grace period, if the processes hasn’t finished, it sends a SIGKILL.
As an alternative, you can also stop a container with docker kill <container_name>. This is analogous to stop, but doesn’t send the SIGTERM and instead goes straight to SIGKILL. The kill command is also useful to send arbitrary signals to the container, e.g. we could send a SIGHUP like so:
$ docker kill -s=SIGHUP <container_id>
After stopping the container, if you try the docker ps again, you will see that the container no longer shows. The container still exists, but it’s not running anymore. When the bash command finished its execution, the container went into stopped state. We can still see it though, by adding the -a option in docker ps:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
707a2d53267a ubuntu "/bin/bash" 6 minutes ago Exited (0) 2 minutes ago trusting_hypatia
You can restart the host and the container will remain there, in stopped state. If you made any changes to the container’s internal file system while it was running, these changes will remain too, persisted within docker’s local cache.
You can get detailed information about a container with the docker inspect command (note, the container doesn’t need to be running):
$ docker inspect <container_id>
This provides a json document showing details such as the container’s state, the image it was started from, etc.
The stopped container can be started again at any time with the start command:
$ docker start <container_id>
When you run this you will notice it looks like nothing has happened. However, a docker ps will show you that the container is again in running state. We don’t see anything because we are not attached to it, so we will need to attach again. We didn’t need to attach explictly before because when we started the container for the first time, we did it with the docker run command, which attaches automatically.
To attach to the container, use the docker attach command:
$ docker attach <container_id>
To definitely destroy the container, we go to another terminal, stop the container again an then use the docker rm command:
This will remove all persistent data belonging to the container. If you do a docker ps -a, the container will not show anymore. Note we must stop the container before destroying it. If we attempt to destroy a running container, we will get an error message.
You can also start a container without attaching to it. For this, we use the -d option in docker run:
$ docker run -d ubuntu /bin/sh -c "while true; do echo This is running inside the container; sleep 1; done"
This will create the container and start it, but without attaching to it. The container will continue running its command in the background. Note we have used a long running command in this example. If we had used bash as command, bash would have ended as soon as starting due to not having a terminal to read as input, and the container would have stopped after the termination of its command.
You can attach to a running container from any other terminal. Use docker ps to get the container id, then try the following:
$ docker attach <container_id>
You terminal will start showing the output of the loop that is running inside the container.
You can attach to the same container from any number of terminals. All of them will be able to provide input to the container, and its output will show in all of them.
In one of the attached terminals, you can use Ctrl-C to stop the loop, which will send the container to Stopped state. After this, you can destroy the container with docker rm as we did above.
Container states
The workflow above gives you an informal overview of the states a container can go through during its life cycle. Let’s go into a little more detail. Once a container is created, it can transition between the following states:
Created
The initial state. A container starts out in this state when it is created with docker create, and can transition out of this state by being started with docker start. Once a container has been started, it never goes back to created, instead it will alternate between Running, Stopped and Paused. It’s common to create containers using docker run, which is analogous to doing a create followed by a start.
Running
The state a container goes to when it is started with docker start or docker run. These operations launch a process inside the container.
A paused container also transitions into Running with docker unpause.
A running container occupies spaces in memory due to its running processes, and also on the hard drive due to its file system state.
Stopped:
The state a running container goes to when it is stopped with docker stop. When a container is stopped, its main process is terminated with SIGTERM and then after a grace period, SIGKILL.
When a container is stopped, its processes are terminated. The container’s file system is persisted across restarts of the host and it can be started again at any time or destroyed with docker rm.
A stopped container does not occupy memory since it has no processes running, but it does occupy space on the hard drive due to its file system state.
Paused
The state a running container goes to when it is paused with docker pause. When a container is paused, its main process is suspended with SIGSTOP.
When a container is paused, its processes remain in memory, but the kernel will not give them CPU time until the container is unpaused. The container’s file system is persisted across restarts of the host and it can be started again at any time.
A paused container occupies spaces in memory due to its processes, and also on the hard drive due to its file system state. In this way the Paused state is similar to Stopped, with the difference being that when the container is Paused, its processes are still loaded in memory ready to resume execution, whereas in Stopped state the container has no processes running.
A Paused container can be unpaused with docker unpause. This resumes execution of the processes at the point where they were at when the container was paused, and transitions the container into Running state.
Building docker images
The ability to automate the creation of container images is one of the main features of Docker. There are two approaches to creating docker images:
Using the commit command: this starts from a running container and creates an image by taking a snapshot of its filesystem state.
Using a Dockerfile: this starts from a metadata file that describes what the image is made of (its layers) and creates an image by processing it. This is the method that lends itself best to automation and reuse.
Next we will study both methods with more detail.
Building images using the commit command
Create an image by taking a snapshot of a container and save it in our local image cache:
$ docker commit <container_id> <repository_id>
The commit command creates an image by taking a snapshot of a container and saving it in our local image cache. This command will assign the image id automatically and will assign the new image the tag latest. It’s also possible to assign the image id and tags explicitely via arguments to the commit command.
By default, the commit command will pause the container right before taking the snapshot, to ensure the files are saved in a consistent state.
It’s worth noting that what this command is actually saving is not the entire filesystem of the container, but the differences between the current state of the container and its base image
Building images using a Dockerfile (recommended approach)
The other way to build an image in docker is by creating a Dockerfile. This is the recommended approach. A Dockerfile is a file written in a Docker-specific Domain Specific Language that contains instructions to build a container proceduraly, step by step, from a “context”. The context is a directory passed as an argument at build time that must contain the Dockerfile and other files that can be used in the build. The following is an example Dockerfile from The Docker Book:
# Version: 0.0.1
FROM ubuntu:14.04
MAINTAINER John Doe "jdoe@example.com"
RUN apt-get update
RUN apt-get install -y nginx
RUN echo 'Hi, I am in your container' \
>/usr/share/nginx/html/index.html
EXPOSE 80
First off, any line that begins with # is a comment.
As you can see, the Dockerfile starts from an existing base image (in this case ubuntu:14.04) creates a container from it, runs some commands inside it, and produces an image by commiting the final state of the container. A key element to understand here is that each line in the Dockerfile is executed in an entirely new container, and once the command is executed, the container is commited, thus creating an intermediate image for each line. Each one of these intermediate images becomes a layer of the final image. This means that state like environment variables or in memory information is not preserved between lines, only files are. If a command fails, the process stops, and the last image created can be used to troubleshoot the problem.
The EXPOSE instruction tells Docker to enable port 80 for listening (we will review this instruction as well as others in more detail later).
Once we have our Dockerfile defined, we can tell docker to execute it with the following command:
$ docker build <path_to_context>
The above command will:
Look for a file named Dockerfile in the given path.
Send the contents of said path to the docker daemon.
The docker daemon will then execute the steps of the Dockerfile one by one, creating a new container for each, and commiting a new image in each step, as described above.
You can add the option -t="<output_repository>/<output_tag>" to the docker build command, to indicate the desired repository and tag of the output image.
The docker build command can also be run with a git repo as path. This causes docker to download the repo and send a copy of its contents to the daemon, as build context.
The output of the build command will look something like this:
$ docker build -t="jdoe/static_web" .
Sending build context to Docker daemon
2.56 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:14.04
---> ba5877dc9bec
Step 1 : MAINTAINER John Doe "jdoe@example.com"
---> Running in b8ffa06f9274
---> 4c66c9dcee35
Removing intermediate container b8ffa06f9274
Step 2 : RUN apt-get update
---> Running in f331636c84f7
Step 2 : RUN apt-get update
---> Running in f331636c84f7
---> 9d938b9e0090
Removing intermediate container f331636c84f7
Step 3 : RUN apt-get install -y nginx
---> Running in 4b989d4730dd
---> 93fb180f3bc9
Removing intermediate container 4b989d4730dd
Step 4 : RUN echo 'Hi, I am in your container'
>/usr/share/ ↩
nginx/html/index.html
---> Running in b51bacc46eb9
---> b584f4ac1def
Removing intermediate container b51bacc46eb9
Step 5 : EXPOSE 80
---> Running in 7ff423bd1f4d
---> 22d47c8cb6e5
Successfully built 22d47c8cb6e5
The lines like Running in b8ffa06f9274 indicate the id of the temporary container that was created to execute that step. The lines like ---> 4c66c9dcee35 show the id of the image that was created after successfully executing the step. If the build fails, you can use these image ids to create a container for debugging and manually run the step that failed inside it.
Once the docker build command finishes, you can use the docker history command to trace back the list of intermediate images that were created during the execution of the Dockerfile:
$ docker history <image_id>
Image building and the image cache
When Docker builds a dockerfile, each instruction results in an image that gets stored in the image cache. If you run the build multiple times, Docker will reuse the intermediate images stored in the cache and not build them again, unless the instruction that originated the image has changed in the Dockerfile since the time when the image was built. In that case, Docker will rebuild that instruction and all instructions that follow it.
You can take advantage of this by putting an innocuous instruction at the beginning of the Dockerfile that you update every time you want to trigger a full rebuild. Take this example from “The Docker Book”:
FROM ubuntu:14.04
MAINTAINER John Doe "jdoe@example.com"
ENV REFRESHED_AT 2021-08-18
RUN apt-get -qq update
Every time you update the dockerfile, simply update the date in the ENV instruction and docker will trigger a full rebuild of all instructions that come after it.
Next we will go into detail about some of the most used Dockerfile instructions.
Declaring ports
To declare that the application listens on a given port number, use the EXPOSE instruction:
EXPOSE <port_number>
The EXPOSE instruction does not actually publish the port. It functions as a type of documentation between the person who builds the image and the person who runs the container, about which ports are intended to be published.
To actually publish the port when running the container, you must use the -p flag on docker run . When you run a container, Docker has two methods of assigning ports on the Docker host:
Docker can randomly assign a high port from the range 49000 to 49900 on the Docker host that maps to port 80 on the container. Example:
docker run -p 80 <image_id>
You can see what host port was assigned with the docker ps command, or also with docker port <container_id> <container_port>
You can specify a specific port on the Docker host that maps to port 80 on the container. Example:
docker run -p 2000:80 <image_id>
This maps port 2000 of the host to port 80 of the container.
You can also use the shortcut -P in docker run which publishes all ports that are present in EXPOSE instructions in the dockerfile, mapping them to random port numbers on the host.
Networking in docker
Within a host, there can be several docker networks, which are created with the docker network create command. Each container can be part of several networks. A container is connected to a network with the docker network connect command.
The networks a container is connected to determine what other containers it can talk to. Containers within the same network can talk to each other via their container-level ports. The host can talk to the containers via their host-level ports. Containers in separate networks are isolated and cannot talk to each other.
Setting persisting and single-command environment variables
To set an environment variable in an image in such a way that all subsequent containers created from it in the dockerfile will have the variable defined, use the ENV instruction:
ENV <key> <value>
To set an environment variable for a single command, add it inline in the command. Example:
RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y ...
Specifying the command the container should run
Every docker container has a command that it runs when it starts. There are two approaches to specify this command:
Using the CMD instruction in the Dockerfile.
Using the ENTRYPOINT instruction in the Dockerfile.
Note, the Dockerfile must contain either one of those instructions. It’s also possible to use both as we’ll see later.
To use the CMD instruction, include a line in the Dockerfile like the following:
When the image provides a CMD instruction, anything you provide in the docker run command after the image will be interpreted as <command> <arguments>. If you don’t provide any arguments after the image, the command and arguments will be taken from the CMD instruction.
In the example above, if you run the container with docker run <image_id> /usr/sbin/nginx -g "daemon off;" the container will the run the command /usr/sbin/nginx -g "daemon off;". If you run the container with docker run <image_id>, it will run /bin/bash", "-l. As you can see, the CMD instruction provides a default command with its arguments, and allows to override both the command and the arguments when calling docker run.
To use the ENTRYPOINT instruction, include a line in the Dockerfile like the following:
ENTRYPOINT ["/usr/sbin/nginx"]
When the image provides an ENTRYPOINT instruction, anything you provide in the docker run command after the image will be interpreted as arguments to the executable in the ENTRYPOINT. The executable is fixed to what was specified in the ENTRYPOINT instruction and cannot be overiden. If you don’t provide any arguments after the image, the executable in the ENTRYPOINT will be run without arguments.
Consider this example:
ENTRYPOINT ["/usr/sbin/nginx"]
Now if you run the container like so:
docker run -t -i <repo_name>:<image_tag> -g "daemon off;"
This will have the effect that the container will run /usr/sbin/nginx -g "daemon off;"
You can also use ENTRYPOINT with a CMD of the form CMD ["param1","param2"] to specify default arguments. When both are present and we run the container with arguments, the command defined in ENTRYPOINT is executed with those arguments, but if no arguments were given, the arguments in the CMD are used as arguments to the executable in the ENTRYPOINT. This can be used to define a default behavior of the container when no arguments were given.
Note: when specifying the command to run in the container, the command must be such that it runs in the foreground, because Docker is designed around this approach. This is usually not a problem, but it’s a concern in some cases like nginx, which be default, forks itself to start a daemon and the the original process stops. For cases like these, you’ll need to configure the process to run in the foreground via its configuration files.
Copying files from the build environment to the container
To copy files and directories from the build environment to the container’s file system, use the ADD instruction. Examples:
# Copy a file into a given destination directory (destination ends in a slash)
ADD nginx/global.conf /etc/nginx/conf.d/
# Copy a file to a given destination path (destination ends without a slash)
ADD nginx/nginx.conf /etc/nginx/nginx.conf
Note the different interpretation of the destination path depending on whether it ends in a slash or not.
The paths of source files in the ADD instruction are interpreted as relative to the context directory given at build time.
It’s also possible for the source to be a URL. See Dockerfile reference for more details.
Volumes
Volumes are specially designated directories within one or more containers that bypass the layered Union File System to provide persistent or shared data for Docker. Think of it like a “shared folder” from a virtualization engine like VirtualBox o VMWare. Volumes are not included when we commit or build an image. Volumes can also be shared between containers and can persist even when containers are stopped. Using volumes, we can create containers that persist their state in storage media that is external to the containers themselves. This is useful, for example, to containerize database engines.
A volume can be attached to a container when the container is created by using the option -v like so:
$ docker run -v <path_on_host_filesystem>:<mount_point_in_container>:ro <repository>:<image_tag> <command>
If the mount point doesn’t exist inside the container, it will be created. In this example we have specified the volume to be Read Only (ro). We could also specify it as Read Write (rw).
Logging into a container without SSH
In the container world, it’s common for a container to not run any ssh daemon inside it, thus we can’t ssh into it. However, the Linux kernel provides a tool we can use to “log in” a container without using SSH. The nsenter tool works with the kernel namespaces concept, and can start a shell inside a given container that our terminal attaches to. Example:
The above command requires having the nsenter tool installed. This tool can be installed via your Linux distribution’s package manager.
It’s worth noting that nsenter can be used to run any kind of process inside the container, by appending it a the end of the command above. The following example shows how to run ls inside the container:
As a final note, we will talk about a best practice about creating containers:
Each container should have only one concern, one purpose (this usually translates to one process).
There are several advantages to this philosophy:
It makes it possible to deploy a new version of the application by destroying the container and creating a new one, without having to shut down other services.
It makes it easier to reuse containers.
It makes it easier to scale applications horizontally.
Docker’s logging mechanism is based on each container having one internal log. This lends itself better to a one concern per container approach.
Note that in practice, one concern will usually translate to one process, but this is not always the case. For example, some programs might spawn additional processes of their own accord. For instance, Celery can spawn multiple worker processes, and Apache can create one process per request.
Karate is a framework for integration testing. It’s relevant to understand that although Karate borrows some concepts from Cucumber and BDD (Behavior Driven Development), it is not a framework for BDD. See Peter Thomas’ answer about this for more details.
Architecture of a Karate program
A Karate program is a Java program that runs the Karate Java library, which contains a JavaScript interpreter that runs a JavaScript program which is what the user provides. This JavaScript program is structured using a gherkin-like domain specific language.
The Java part of the program can be extended by user-provided Java code. The stack can be described as follows:
Karate script (JavaScript code inside a .feature file)
The Karate Java library can be invoked from a JUnit test class for easy integration with build systems like Maven o Gradle, or can be invoked directly from a standalone runner.
Running a Karate test
Running with Maven
With this approach, you create a JUnit test class that kickstarts Karate, and run it with mvn -Dtest=MyRunner test, where MyRunner is the name if your test class. This way of running tests is the most natural if your script depends on custom Java code, because it ensures that your utility code gets built and then included in the classpath for running every time you run the test. This is a very common use case since things like checking a database between API calls requires Java JDBC code.
Note also that calling the test class as …Runner (without ending in Test or Tests) is useful so that Maven will not attempt to run it as part of the unit tests (but it will still get built and included in the classpath).
Given a REST service in a repo structured according to the standard Maven project structure, and a URL called myresource in your API that you want to test, follow these steps to create a Karate test and run it.
Create the myresource.feature test script, with the following contents:
Feature: tests the myresource API.
more lines of description if needed.
Background:
# this section is optional !
# steps here are executed before each Scenario in this file
# variables defined here will be 'global' to all scenarios
# and will be re-initialized before every scenario
Scenario: Send a GET and validate the HTTP status code
Given url 'https://localhost:8080/api/v1/myresource'
When method GET
Then status 200
Create the karate-config.js config file, with the following contents:
function fn() {
// Comments in this file cannot be in the first line
karate.configure('ssl', { trustAll: true }); // This is necessary of your service uses a self-signed certificate. Otherwise, you can omit this line.
return {};
}
Careful with comments in this file! The first line MUST start with the keyword function, otherwise Karate will not parse it correctly.
Karate looks for the karate-config.js file in the root of the classpath (that is src/test/java) be default. We prefer to have all Karate related files under src/test/java/integration, so we have put our karate-config.js there.
If you want to configure the log level used by the Karate logger, add the following in your logback.xml:
<logger name="com.intuit.karate" level="debug"/>
Use the following command to build and run the test:
mvn -Dtest=MyResourceRunner -Dkarate.config.dir=target/test-classes/integration/ test
Karate looks for the karate-config.js file in the root of the classpath (that is src/test/java/) be default, and this is the setup found on most examples in the documentation. We prefer to have all Karate related files under src/test/java/integration, so we have put our karate-config.js there, and that’s why we pass the karate.config.dir argument when running. If you want, you can leave the karate-config.js in src/test/java/, and then you can omit the argument -Dkarate.config.dir when running.
Running with the standalone executable
All of Karate (core API testing, parallel-runner / HTML reports, the debugger-UI, mocks and web / UI automation) is available as a single, executable JAR file, which includes even the karate-apache dependency. Under this mode of execution, you would run a Karate script with the following command:
java -jar karate.jar my-test.feature
If your script depends on custom Java code, you will need to add that code to the classpath when running the Karate jar. You can use the standalone JAR and still depend on external Java code – but you have to set the classpath for this to work. The entry-point for the Karate command-line app is com.intuit.karate.Main. Here is an example of using the Karate Robot library as a JAR file assuming it is in the current working directory.
If on Windows, note that the path-separator is ; instead of : as seen above for Mac / Linux. Refer this post for more details. The karate-config.js will be looked for in the classpath itself.
Splitting a test in several .feature files
This is already implicit in the above, but it’s still worth noting that you can split your test across several feature files by simply defining several test methods in the runner class, like so:
# assigning a string value:
Given def myVar = 'world'
# using a variable
Then print myVar
# assigning a number (you can use '*' instead of Given / When / Then)
* def myNum = 5
* print 'the value of myNum is:', myNum
Note that def will over-write any variable that was using the same name earlier. Keep in mind that the start-up configuration routine could have already initialized some variables before the script even started. For details of scope and visibility of variables, see Script Structure. Note that url and request are not allowed as variable names. This is just to reduce confusion for users new to Karate who tend to do * def request = {} and expect the request body or similarly, the url to be set.
JavaScript integration
Remember, a karate script is a thinly-veiled JavaScript program, so JavaScript integration is straightforward.
Defining a one-line function
* def greeter = function(title, name) { return 'hello ' + title + ' ' + name }
* assert greeter('Mr.', 'Bob') == 'hello Mr. Bob'
Standard JavaScript syntax rules apply, but the right-hand-side (or contents of the *.js file if applicable) should begin with the function keyword. This means that JavaScript comments are not supported if they appear before the function body. Also note that ES6 arrow functions are not supported. Finally, especially when using stand-alone *.js files, you can use fn as the function name, so that your IDE does not complain about JavaScript syntax errors, e.g. function fn(x){ return x + 1 }.
Regardless of how a JavaScript function is defined, if it’s a one-argument function, you can call it with the call construct. This makes for a somewhat more readable code because it makes the parameter names explicit at the point of invocation:
The caveat mentioned above about comments in the karate-config.js file applies to all JavaScript files: the first line MUST start with the function keyword.
Java integration
Calling Java static methods
Calling static methods can be done as if they were Javascrpt native functions:
To access primitive and reference Java types from JavaScript, call the Java.type() function, which returns a type object that corresponds to the full name of the class passed in as a string.
* def dateStringToLong =
"""
function(s) {
var SimpleDateFormat = Java.type('java.text.SimpleDateFormat');
var sdf = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ");
return sdf.parse(s).time; // '.getTime()' would also have worked instead of '.time'
}
"""
* assert dateStringToLong("2016-12-24T03:39:21.081+0000") == 1482550761081
Given this custom, user-defined Java class:
package com.mycompany;
import java.util.HashMap;
import java.util.Map;
public class JavaDemo {
public Map<String, Object> doWork(String fromJs) {
Map<String, Object> map = new HashMap<>();
map.put("someKey", "hello " + fromJs);
return map;
}
public static String doWorkStatic(String fromJs) {
return "hello " + fromJs;
}
}
This is how it can be called from a test-script via JavaScript, and yes, even static methods can be invoked:
* def doWork =
"""
function(arg) {
var JavaDemo = Java.type('com.mycompany.JavaDemo');
var jd = new JavaDemo();
return jd.doWork(arg);
}
"""
# in this case the solitary 'call' argument is of type string
* def result = call doWork 'world'
* match result == { someKey: 'hello world' }
# using a static method - observe how java interop is truly seamless !
* def JavaDemo = Java.type('com.mycompany.JavaDemo')
* def result = JavaDemo.doWorkStatic('world')
* assert result == 'hello world'
Note that JSON gets auto-converted to Map (or List) when making the cross-over to Java. Refer to the cats-java.feature demo for an example.
Modularizing tests
A Karate program is written in a feature file, which is stored with extension .feature. Each feature file can call other feature files. For every feature file, Karate creates a context object that the JavaScript program can mutate. Each feature file has its own isolated context by default, but using shared context is also possible. When you use the def construct to assign a variable, what you are doing is adding a new field in the context object.
To call another feature file, use the call function. To pass paramaters to a called script, pass a json object with the data you want. To return data from the called script to the caller, have the caller create fields in its context with def. The caller can then access the callee’s context as the return value of the call function. Example:
Caller script:
Feature: which makes a 'call' to another re-usable feature
Background:
* configure headers = read('classpath:my-headers.js')
* def signIn = call read('classpath:my-signin.feature') { username: 'john', password: 'secret' }
* def authToken = signIn.authToken
Scenario: some scenario
# main test steps
Called script:
Feature: here are the contents of 'my-signin.feature'
Scenario:
Given url loginUrlBase
And request { userId: '#(username)', userPass: '#(password)' }
When method post
Then status 200
And def authToken = response
# second HTTP call, to get a list of 'projects'
Given path 'users', authToken.userId, 'projects'
When method get
Then status 200
# logic to 'choose' first project
And set authToken.projectId = response.projects[0].projectId;
Key takeaways:
The call read syntax an abbreviation for two separate calls:
def someFunction = read(‘classpath:my-signin.feature’) : reads the file and evaluates it, returning a JavaScript function that contains the executable code.
call someFunction : executes the function.
The called script uses set authToken.projectId to set a property in an object.
The called script uses def authToken = response to store an object as a field in its context. Note the context is not referenced explicitely, rather you use def to access it.
The caller accesses the context of the called script by assigning the result of the call function: def signIn = call, thus it can then acccess its fields as signIn.authToken.
When the caller script is run, only the scenarios from the caller script are reported in the output as “Tests run”. If the called script has several scenarios, these will be executed, but they won’t be really counted.
Also note that if you invoke the called script from the background, it will be run before every scenario of the parent script.
The two previous points show that the call construct is meant to be used to call small feature files that contain routine code, not to separate a complex test with several scenarios per file.
Test selection with tags
You can tag a set of scenarios to be able to run only those. You can also tag an entire feature to select that feature. To add tags, use this syntax:
@myTagForTheFeature
Feature: tags demo - first
@myTagForAScenario
Scenario: Bla
...
Then you can run only the scenarios tagged with myTagForAScenario using the command: mvn -Dtest=MyRunner -Dkarate.options=”–tags @myTagForAScenario” test
SQL integration
We can work with a relational database within Karate scripts, by leveraging the Java integration and JDBC. The JDBC part can be further simplified by using Spring’s JdbcTemplate utility class. The dogs.featue example and its accompanying JDBC utility code in Karate’s repo show how to do this.
In this article we will go over the details of some of the most used data types in relational database systems and the factors that go into deciding which type to use for each situation.
Let’s start with the types as defined in the SQL standard.
Data types in standard SQL
CHAR(N) data type
In standard SQL, the CHAR(N) datatype represents a string of fixed length. All stored values occupy N characters on the database and are filled with spaces if a smaller value is inserted. CHAR(N) may be more cache-friendly than VARCHAR(N) because the DBMS can store it in-record, but this depends on the DBMS (Oracle for intance stores CHAR(N) outside of the record exactly the same as a VARCHAR(N), so there’s no performance gain).
VARCHAR(N) data type
In standard SQL, the VARCHAR(N) data type represents a string of variable length with a maximum length of N characters. If a value is smaller than N characters, only the used characters are stored. Values larger than N characters cannot be stored. DBMSs usually implement this datatype by having the record hold a pointer to the value which is stored separately.
Exact numeric data types
Exact numeric types hold numeric values without digits after the decimal or with a firm number of digits after the decimal. All exact numeric types are signed.
NUMERIC(<p>,<s>) and DECIMAL(<p>,<s>) denotes two types which are nearly the same. <p> (precision) defines a fixed number of all digits within the type and <s> (scale) defines how many of those digits follow the decimal place. Numeric values with more than (p – s) digits before the decimal place cannot be stored and numeric values with more than s digits after the decimal place are truncated to s digits after the decimal place. p and s are optional. It must always be: p ≥ s ≥ 0 and p > 0.
SMALLINT, INTEGER and BIGINT denote data types without a decimal place. The SQL standard did not define their size, but the size of SMALLINT shall be smaller than the size of INTEGER and the size of INTEGER shall be smaller than the size of BIGINT.
Approximate numeric
Approximate numeric types hold numeric values in floating-point format. All approximate numeric types are signed. Their primary use cases are scientific computations.
There are three types: FLOAT (<p>), REAL and DOUBLE PRECISION. In the FLOAT datatype, p denotes the number of bits of the mantissa (note it’s not the number of digits). The precision of REAL and DOUBLE PRECISION is implementation defined.
Data types in Oracle
CHAR(N)
The CHAR datatype stores fixed-length, single-byte character strings. When you create a table with a CHAR column, you must specify a string length (in bytes or characters) between 1 and 2000 bytes for the CHAR column width. The default is 1 byte. Oracle then guarantees that:
When you insert or update a row in the table, the value for the CHAR column has the fixed length.
If you give a shorter value, then the value is blank-padded to the fixed length.
If a value is too large, Oracle Database returns an error.
VARCHAR2(N)
The VARCHAR2 datatype stores variable-length, single-byte character strings. When you create a table with a VARCHAR2 column, you specify a maximum string length (in bytes or characters) between 1 and 4000 bytes for the VARCHAR2 column. For each row, Oracle Database stores each value in the column as a variable-length field unless a value exceeds the column’s maximum length, in which case Oracle Database returns an error. Using VARCHAR2 and VARCHAR saves on space used by the table.
The VARCHAR datatype is synonymous with the VARCHAR2 datatype.
Internal representation of CHAR and VARCHAR2 types in Oracle
Oracle stores the CHAR datatype exactly the same as a VARCHAR2, just blank-padding the value. There is no in-record storage of CHAR values, so in Oracle the CHAR datatype offers no performance improvement at all over VARCHAR2. If we also take into account the fact that CHAR uses more space to store the same value, there is little reason to use CHAR in any situation, so VARCHAR2 is recommended.
NCHAR and NVARCHAR2 data types
NCHAR and NVARCHAR2 are Unicode datatypes that store Unicode character data. The character set of NCHAR and NVARCHAR2 datatypes can only be either AL16UTF16 or UTF8 and is specified at database creation time as the national character set. AL16UTF16 and UTF8 are both Unicode encoding.
The NCHAR datatype stores fixed-length character strings that correspond to the national character set.
The NVARCHAR2 datatype stores variable length character strings.
When you create a table with an NCHAR or NVARCHAR2 column, the maximum size specified is always in character length semantics. Character length semantics is the default and only length semantics for NCHAR or NVARCHAR2.
NUMBER data type
The NUMBER datatype defines a sigmed fixed-point number. The following illustrates the syntax of the NUMBER data type:
NUMBER[(precision [, scale])]
The Oracle NUMBER data type has precision and scale.
The precision is the total number of digits in a number (including digits after the decimal point). It ranges from 1 to 38.
The scale is the number of digits to the right of the decimal point in a number. It ranges from -84 to 127.
If precision and scale are not specified explicitely, their values depend on the syntax used to define the type.
NUMBER(p) is equivalent to NUMBER(p, 0) and defines an integer (i.e. fixed point number with precision 0).
NUMBER (both parameters ommited) defines a number that can store numeric values with the maximum range and precision.
Oracle allows the scale to be negative, for example the following number will round the numeric value to hundreds: NUMBER(5,-2).
Oracle contains a number of aliases that you can use to define numeric columns as shown in the following table:
ANSI data type
Oracle NUMBER data type
INT
NUMBER(38)
SMALLINT
NUMBER(38)
NUMBER(p,s)
NUMBER(p,s)
DECIMAL(p,s)
NUMBER(p,s)
Data types in MySQL
Numeric data types
MySQL supports all standard SQL numeric data types. These types include the exact numeric data types (INTEGER, SMALLINT, DECIMAL, and NUMERIC), as well as the approximate numeric data types (FLOAT, REAL, and DOUBLE PRECISION). The keyword INT is a synonym for INTEGER, and the keywords DEC and FIXED are synonyms for DECIMAL. MySQL treats DOUBLE as a synonym for DOUBLE PRECISION (a nonstandard extension). MySQL also treats REAL as a synonym for DOUBLE PRECISION (a nonstandard variation), unless the REAL_AS_FLOAT SQL mode is enabled.
The type can be declared unsigned with the following syntax:
CREATE TABLE classes ( 2 total_member INT UNSIGNED 3);
The DATE type is used for values with a date part but no time part. MySQL retrieves and displays DATE values in ‘YYYY-MM-DD’ format. The supported range is ‘1000-01-01’ to ‘9999-12-31’.
The DATETIME type is used for values that contain both date and time parts. MySQL retrieves and displays DATETIME values in ‘YYYY-MM-DD hh:mm:ss’ format. The supported range is ‘1000-01-01 00:00:00’ to ‘9999-12-31 23:59:59’.
The TIMESTAMP data type is used for values that contain both date and time parts. TIMESTAMP has a range of ‘1970-01-01 00:00:01’ UTC to ‘2038-01-19 03:14:07’ UTC.
MySQL converts TIMESTAMP values from the current time zone to UTC for storage, and back from UTC to the current time zone for retrieval. (This does not occur for other types such as DATETIME.) By default, the current time zone for each connection is the server’s time. The time zone can be set on a per-connection basis. As long as the time zone setting remains constant, you get back the same value you store. If you store a TIMESTAMP value, and then change the time zone and retrieve the value, the retrieved value is different from the value you stored. This occurs because the same time zone was not used for conversion in both directions. The current time zone is available as the value of the time_zone system variable.
Morning Flow 