How to secure the graceful shutdown of a Spring Boot application running on AWS EC2 instance

Problem

We want to have a way to automatically instruct a Spring Boot application to do a graceful shutdown. This is useful for example when we use CodeDeploy to deploy the application to an EC2 instance, because it allows us to let the application shutdown gracefully when we are deploying a new version.

Assumptions

For the remainder of this text we will work on the following assumptions:

1. The application is a Spring Boot application.
2. The application is deployed on an EC2 instance using AWS CodeDeploy (which involves custom start and stop scripts) but the stop script calls the shutdown endpoint in an insecure way over HTTP (not HTTPS). See previous post Setting up a simple CICD pipeline with AWS CodePipeline for instructions on how to set this up.
3. The credentials for graceful shutdown will be stored in AWS Secrets Manager.

Steps

Follow these instructions to automate the graceful shutdown.

• Configure HTTP basic authentication for all the endpoints under /actuator, following the instructions at How to secure an API endpoint with HTTP basic authentication in Spring Boot .
• I you haven’t configured the AWS CLI locally and on the EC2 instance, do it following the instructions at How to configure the AWS CLI.
• This is necessary so that the shutdown script can retrieve the password for the shutdown endpoint from AWS Secrets Manager.
• Edit the start script so that:
  • The delay between the command where the application is started and the command where you check if the application started ok is long enough taking into account the time the application needs in order to read the admin password from the secret management service.
• Edit the shutdown script so that:
  • It retrieves the admin password from the secret management service and passess it in the curl call to the shutdown endpoint.
  • It doesn’t use the -v option in the curl call, so that the credentials base64 string doesn’t appear in the output
  • It calls the actuator/shutdown endoint with https and not http.
  • If your application uses a self-signed certificate, make sure that the shutdown script calls curl with the option -k, so that curl accepts the self-signed certificate from our application.
• The following is an example of a viable stop script updated to read the admin password from AWS Secrets Manager and sending it in the request to the /actuator/shutdown endpoint:

#! /bin/bash

SERVICE_HOST=localhost
SERVICE_PORT=8080
ADMIN_PASSWORD_SECRET_NAME=<the_admin_password_secret_name>

pid=`ps aux | grep -i <my_app_name> | grep -v grep | awk '{ print $2 }'`
if [ ! $pid ]; then
  echo "Process not runing, nothing to do"
  exit 0
fi

ADMIN_PASSWORD_JSON=`aws secretsmanager get-secret-value --secret-id $ADMIN_PASSWORD_SECRET_NAME`
if [ ! $? -eq 0 ]; then
  echo "Could not retrieve admin password from AWS Secrets Manager. Check that aws configure has been ran in the host. The service will remain running"
  exit 1
fi

ADMIN_PASSWORD=`echo "$ADMIN_PASSWORD_JSON" | jq -r '.SecretString'`
if [ ! $? -eq 0 ]; then
  echo "Could not parse the admin password from the AWS Secrets Manager json response. Check that jq is installed in the host. The service will remain running"
  exit 1
fi

echo "Sending shutdown request to service..."
curl -k -X POST --user "admin:$ADMIN_PASSWORD" https://$SERVICE_HOST:$SERVICE_PORT/actuator/shutdown
curl_result=$?
echo -ne "\n"

if [ ! $curl_result -eq 0 ]; then
  echo "Could not do the graceful shutdown of the service. This may mean that we couldn't send the shutdown request, or that the admin password we retrieved from AWS Secrets Manager doesn't match what the service is expecting. We will kill the process anyway"
  echo "Curl exit code: $curl_result"
fi

kill $pid
kill_result=$?
if [ ! $kill_result -eq 0 ]; then
  echo "Could not kill process. You should kill it manually"
  echo "Kill exit code: $kill_result"
  exit 1
fi

echo "Service stopped"

• Edit the application.properties so that it has:

management.endpoints.web.exposure.include=*
management.endpoint.shutdown.enabled=true

• This is all that is needed (and the first line is optional to the effects of enabling shutdown).
• (One time only per EC2 instance) Install the jq tool in the EC2 instance, with the following command:

$ sudo yum install jq

• This is necessary because the stop script above needs to use jq to parse the AWS Secrets Manager json response.