make-deploy

A simple Makefile based deployment system.

make-deploy is licensed under the terms of the MIT-License. See LICENSE for info.

Installation

1. Clone make-deploy as a submodule into your repository:

   $ git submodule add https://git.dnix.de/dnix/make-deploy

2. Create a symlink to the Makefile:

   $ ln -s make-deploy/Makefile .

3. Create config.mk (for common config options) and secrets.mk (for sensitive information, should be git-crypted) in your project.

Example config.mk:

DEPLOY_NAME = my-project
DEPLOY_PATH = /srv
DEPLOY_TYPE = compose
DEPLOY_HOSTS = server01.example.com server02.example.com
DEPLOY_SUBPATH = my-sub-path

DOCKER_IMAGE = dr.example.com/my-project:latest
DOCKER_LOGIN = 1
DOCKER_REGISTRY = dr.example.com

SSH_USER = ci

Example secrets.mk:

DOCKER_USER = root
DOCKER_PASS = secret1234

DEPLOY_SUBPATH is used to define a location for DEPLOY_TYPE = compose that states where the docker-compose.yml is placed at. If that variable is undefined, it is expected to be in projects root.

By setting DEPLOY_CONFIG_OVERRIDE, an alternative config can be loaded to override existing configuration settings. This is useful in script calling make deploy or in a .gitlab-ci.yml in order to control settings for different targets.

Usage

Philosophy

make-deploy lets you deploy software on the commandline with make.

Doing as much config as you can in config.mk, via DEPLOY_OVERRIDE_CONFIG and in secrets.mk will keep this ability intact without depending on complex build systems (e.g. Gitlab deployments controlled by .gitlab-ci.yml and other stuff like that). OTOH it does not stop you from doing this: simply call make deploy in your build pipeline in order to get the best of both worlds.

Relying only on GNU Make and simple CLI tools makes deployments robust and still possible, when big parts of your infrastructure are broken and have to be redeployed.

Working principle

make-deploy calls several stages in the deployment process, which are mandatory cleanup prepare build test upload pre-deploy pre-local pull start notify post-local post-deploy reload.

mandatory: Checks if all needed variables are set in config.mk.

cleanup: Runs cleanup.sh on each remote target, meant to be used to wipe contents of prior installations (only if DEPLOY_CLEANUP is set).

prepare: Creates needed directory for the deployment on the target system, sets secure file permissions for secrets.mk.

build: Calls build.sh locally. This is for building purposts, e.g. docker build and push.

test: Calls test.sh locally.

upload: rsyncs the contents of the repo to the target systems.

pre-deploy: Runs pre-deploy.sh remotely on the target hosts. Used for service specific purposes.

pre-local: Runs pre-local.sh locally. Used for service specific purposes.

pull: Pulls docker images (if appropriate).

start: Starts the service.

notify: If DEPLOY_NOTIFY_HOSTS, DEPLOY_NOTIFY_MAIL and DEPLOY_NOTIFY_MSG are all set, sends a mail to DEPLOY_NOTIFY_MAIL telling that a manual deployment is needed on the listed hosts. This is useful when some hosts cannot be reached automatically (e.g. they are in maintenance) but should still be notified for a manual run.

post-local: Runs post-local.sh locally. Used for service specific purposes.

post-deploy: Runs post-deploy.sh remotely on the target hosts.

reload: Runs reload.sh remotely on the target hosts. Used for service specific reloading/restarting.

Start the deployment process

make deploy

You can also run the individual stages on their own, e.g. make pull, make start, make stop, make logs, make restart, make reload, make notify, make prerequisites, make mandatory or make cleanup.

Some stages accept optional parameters. For example make logs service=my-service passes my-service to the compose/swarm log command, and make deploy DEPLOY_CONFIG_OVERRIDE=myconfig.mk loads an alternative config.

Update make-deploy submodule in your project

make self-update

Available options for DEPLOY_TYPE

simple

Just copy the repo to the remote location(s). Put additional logic into build.sh, start.sh, pre-deploy.sh and post-deploy.sh.

If start.sh is present it is executed remotely on each host during start.

DEPLOY_START_DELAY: Seconds to wait after starting the service(s) (default 0).

compose

Docker Compose deployment.

COMPOSE_CMD: The compose command to use, e.g. docker-compose (default) or docker compose (default 'docker-compose').

DEPLOY_SUBPATH: Subdirectory (relative to DEPLOY_PATH/DEPLOY_NAME) where the docker-compose.yml is placed. If unset, the compose file is expected in the project root.

compose-file

Like compose, but you can name the compose file and the project explicitly:

DEPLOY_COMPOSE_FILE: Name of the docker-compose file (mandatory).

DEPLOY_COMPOSE_PROJECT: The compose project name (mandatory).

swarm

Docker Swarm deployment. Uses docker stack deploy with stack.yml.

k8s

K8S deployment.

K8S_CONTEXT: K8S context to be used. Make sure to properly set up your kube credentials (kube config) and provide a local copy of kubectl.

K8S_MANIFEST_FILENAME: The k8s manifest to apply (default kubernetes.yaml).

K8S_ADDITIONAL_PARAMS: Optional extra parameters passed to kubectl apply.

If a secrets.yaml file is present in the repo, it is appended to the manifest and both are run through envsubst before being applied.

helm

K8S deployment via helm.

K8S_CONTEXT: K8S context to be used. Make sure to properly set up your kube credentials (kube config) and provide a local copy of kubectl.

K8S_NAMESPACE: K8S namespace the chart is installed into (mandatory).

HELM_VALUES_FILE: Contains the specific variables for this deployment (mandatory).

HELM_CHART_NAME: The helm chart name (mandatory).

HELM_CHART_PATH: Path to the helm chart (mandatory).

HELM_CHART_VERSION: Specific chart version to install (use latest if empty).

HELM_REPO_URL: If set, the helm repo is added (helm repo add) and updated before deployment.

TEST_MODE: If set, only renders the chart with helm template --debug without deploying.

DEBUG_MODE: If set, the deployment is run with --debug.

The helm deployment processes the contents of HELM_VALUES_FILE with envsubst and pipes the result to helm upgrade to deploy the application.

NAMESPACE is the old name for K8S_NAMESPACE and still works as an alias, but K8S_NAMESPACE should be used instead.

kustomize

K8S deployment via kustomize.

K8S_CONTEXT: K8S context to be used. Make sure to properly set up your kube credentials (kube config) and provide a local copy of kubectl.

KUSTOMIZE_FOLDER: Path to the folder containing the kustomize base/overlay (mandatory).

NAMESPACE: K8S namespace. Used for kubectl apply and for kustomize edit set namespace (if KUSTOMIZE_EDIT_NAMESPACE is set) (mandatory).

KUSTOMIZE_EDIT_NAMESPACE: If set, kustomize edit set namespace is run before building.

The kustomize deployment builds the customization with kustomize build and pipes the result into kubectl apply (run through envsubst).

cron

Deploy cronjob.

CRON_USER: The user whose crontab is updated (mandatory).

The crontab file in the repo is installed into the crontab of CRON_USER on each host, tagged with #make-deploy-cron:$(DEPLOY_NAME) so it can be removed on stop.

systemd

systemd service deployment.

The systemd.service file in the repo is installed as /etc/systemd/system/$(DEPLOY_NAME).service and enabled/started on each host.

apt

APT deployment on debian-like distributions.

APT_PACKAGES: The packages to install (mandatory).

After installing the packages, start.sh is executed if present.

tf

Deploy via Terraform. Config is mostly done in terraform.

TF_FLAGS: Flags to pass to terraform apply (mandatory).

TF_TARGET: Limits deployment to the given terraform state file (-t).

TF_CMD: The terraform binary/command to use (default terraform).

Optional configuration variables

The following variables are available across the deployment types:

DEPLOY_NAME: Name of the project (mandatory).

DEPLOY_HOSTS: Space-separated list of target hosts (mandatory).

DEPLOY_PATH: Base path on the remote hosts where the project is deployed (mandatory).

SSH_USER: User used for ssh/rsync (mandatory).

DEPLOY_TYPE: One of the deployment types above (mandatory).

DEPLOY_PROJECT_DIR: Optional extra directory appended to the project dir.

DEPLOY_START_DELAY: Seconds to wait after start (default 0).

DEPLOY_CLEANUP: If set, enables the cleanup stage.

DEPLOY_CONFIG_OVERRIDE: Alternative config file to load for overriding settings (see above).

DEPLOY_NOTIFY_HOSTS: Hosts that need a manual deploy (for notify).

DEPLOY_NOTIFY_MAIL: Recipient of the notify mail.

DEPLOY_NOTIFY_MSG: Message body of the notify mail.

Helper scripts

The following optional scripts are called by the deployment where applicable:

• prerequisites.sh: run before prepare (if present).
• cleanup.sh: run on remote hosts during cleanup.
• build.sh: run locally during build.
• test.sh: run locally during test.
• pre-deploy.sh / pre-deploy_<host>.sh: run remotely before deployment.
• pre-local.sh / post-local.sh: run locally before/after deployment.
• post-deploy.sh / post-deploy_<host>.sh: run remotely after deployment.
• start.sh: run remotely during start (simple/apt).
• reload.sh: run remotely during reload.