Automated Deployment of Vault Server with NGINX Using Ansible

Lately, I encountered a familiar challenge: deploying a service that operates on an unprivileged port, then securely exposing it through a reverse proxy with TLS termination. While this problem isn’t new, my specific scenario involves Hashicorp Vault.

The Problem

By default, Vault runs on TCP port 8200, this is an unprivileged port, meaning that any user can bind it to a service. Vault enforces this philosophy so if you’ve properly hardened your installation it won’t just bind to a privileged port, and since we’re trying to use Vault for very sensitive data we really want this to be a secure platform and we’ll probably want to be ultimately offering the service over HTTPS using TCP port 443.

When Push Comes To Shove

From a technical standpoint, TLS termination can be executed directly on the Vault instance, exposing it through TCP port 8200. However, adhering to strict firewall regulations may discourage opening additional ports. Streamlining port usage is preferable, hence let’s explore utilizing NGINX as a reverse proxy to properly expose Vault over TCP port 443.

Vault NGINX setup

Implementing NGINX

Install NGINX
```
sudo apt-get install nginx
```

With the Certificate and Private Key located in /etc/ssl/certs and /etc/ssl/private respectively we can just use the NGINX default configuration file which is located in /etc/nginx/sites-enabled/default:

server {
    listen 443;
    server_name <FQDN_cloud>;

    ssl on;
    ssl_certificate /etc/ssl/certs/mc-vault.cer;
    ssl_certificate_key /etc/ssl/private/mc-vault.key;

    ssl_prefer_server_ciphers on;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;

    location / {
            proxy_pass http://127.0.0.1:8200;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
    }
}

Restart NGINX in order to load the configuration:
```
sudo /etc/init.d/nginx restart
```

Implementing VAULT

Check my detailed blog here

Using ANSIBLE

Create an Ansible Role for Vault and NGINX setup so we can reuse it.

.
├── README.md
├── inventory
│   └── hosts
├── roles
│   ├── nginx
│   ├── vault
│   ├── vault-configure
│   ├── vault-init
│   └── vault-unseal
└── vault-nginx-deployment.yaml

Every role is set up with the following structure

.
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── tasks
│   └── main.yaml
└── templates
    ├── init.service.j2
    └── vault-config.hcl.j2

Variables are stored under defaults/main.yaml so that it can be reference across roles. Store your configuration files (vault.hcl, NGINX service) in jinja templates

Download Vault Archive

- name: Install prerequisites
  package:
    name: "{{ item }}"
    update_cache: yes
  with_items: "{{ vault_install_prerequisites }}"
  become: true

- name: Create directories for Vault
  file:
    path: "{{ item }}"
    state: directory
  loop:
    - ~/vault
    - ~/vault/data
    - ~/logs

- name: Download binary
  get_url:
    url: https://releases.hashicorp.com/vault/{{vault_version}}/vault_{{vault_version}}_linux_amd64.zip
    dest: /tmp/vault_{{vault_version}}_linux_amd64.zip
    # Allow owner:+rwx group:+rw others:+rw
    mode: 0755
    # Verify integrity
    checksum: "{{vault_checksum}}"
  register: vault_download

- name: "Unzip vault archive"
  unarchive:
    src: "{{ vault_download.dest }}"
    dest: /usr/local/bin
    copy: no
    mode: 0755

Set binary capabilities on Linux, to give the Vault executable the ability to use the mlock syscall without running the process as root. This prevents secrets to be transferred to the host memory.

- name: "Set vault binary capabilities"
  capabilities:
    path: /usr/local/bin/vault
    # This capability allows the process to lock memory into RAM, which can be useful for preventing the system from swapping Vault's sensitive data to disk. +ep indicates both effective (e) and permitted (p) capabilities.
    capability: cap_ipc_lock+ep
    state: present

Create systemd init file to manage the persistent vault daemon

[Unit]
Description=a tool for managing secrets
Documentation=https://vaultproject.io/docs/
After=network.target
ConditionFileNotEmpty=~/vault/vault-config.hcl

[Service]
ExecStart=/usr/local/bin/vault server -config=~/vault/vault-config.hcl
ExecReload=/usr/local/bin/kill --signal HUP $MAINPID
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
SecureBits=keep-caps
NoNewPrivileges=yes
KillSignal=SIGINT

# This directive specifies the target unit(s) that should "want" (or require) this unit. When a target unit is activated, systemd also activates all units that it "wants". In this case, 
# multi-user.target is specified, which is a target unit representing a system state where multiple users are logged in and the system is fully operational.
[Install]
WantedBy=multi-user.target

Set above content into Systemd service file. Finally, start the vault server

- name: Copy systemd init file
  template:
    src: init.service.j2
    dest: /etc/systemd/system/vault.service
  # handler  
  notify: systemd_reload

- name: config file
  template:
    src: vault-config.hcl.j2
    dest: "{{ vault_config_path }}"

- name: vault service
  service:
    name: vault
    state: started
    enabled: yes

Initialize vault server

- name: Create unseal directories
  file:
    path: "{{ unseal_keys_dir_output }}"
    state: directory
  delegate_to: localhost

- name: Create root key directories
  file:
    path: "{{ root_token_dir_output }}"
    state: directory
  delegate_to: localhost

- name: Initialise Vault operator
  shell: vault operator init -key-shares=1 -key-threshold=1 -format json
  environment:
    VAULT_ADDR: "http://127.0.0.1:8200"
  register: vault_init_results

- name: Parse output of vault init
  set_fact:
    vault_init_parsed: "{{ vault_init_results.stdout | from_json }}"

- name: Write unseal keys to files
  copy:
    dest: "{{ unseal_keys_dir_output }}/unseal_key_{{ item.0 }}"
    content: "{{ item.1 }}"
  with_indexed_items: "{{ vault_init_parsed.unseal_keys_hex }}"
  delegate_to: localhost

- name: Write root token to file
  copy:
    content: "{{ vault_init_parsed.root_token }}"
    dest: "{{root_token_dir_output}}/rootkey"
  delegate_to: localhost

- name: Generate self-signed certificate
  shell: >
    cd ~/vault && openssl req -out aim-qm-vm.aimqm.app.crt -new -keyout aim-qm-vm.aimqm.app.key -newkey rsa:4096 -nodes -sha256
    -x509 -subj "/O=aimqm/CN=aimqm" -addext "subjectAltName = IP:10.94.95.4,DNS:aim-qm-vm.aimqm.app"
    -days 3650
  args:
    executable: /bin/bash

Unseal Vault

- name: Reading unseal key contents
  command: cat {{item}}
  register: unseal_keys
  with_fileglob: "{{ unseal_keys_dir_output }}/*"
  delegate_to: localhost
  become: false

- name: Unseal vault with unseal keys
  shell: |
    vault operator unseal {{ item.stdout }}
  environment:
    VAULT_ADDR: "http://127.0.0.1:8200"
  with_items: "{{unseal_keys.results}}"

Configure NGINX

---
- name: Install Nginx
  ansible.builtin.package:
    name: nginx
    state: present

- name: Copy Nginx configuration file
  template:
    src: nginx_config.j2
    dest: /etc/nginx/sites-available/default

- name: Restart Nginx
  ansible.builtin.service:
    name: nginx
    state: restarted

This is the final playbook used to start vault

---
- hosts: azure-linux-vm
  become: true
  become_user: root
  become_method: sudo
  vars:
    unseal_keys_dir_output: "{{ playbook_dir }}/unsealKey"
    root_token_dir_output: "{{ playbook_dir }}/rootKey"
  roles:
    - vault
    - vault-init
    - vault-unseal
    - vault-configure
    - nginx

Below is the inventory hosts file

[azure-linux-vm]
aim-qm-vm ansible_host=<IP_ADDRESSS> 
ansible_user=<REMOTE_USERNAME> 
ansible_ssh_private_key_file=~/.ssh/<PRIVATE_KEY>.pem

Run Ansible

ansible-playbook vault-nginx-deployment.yml

Access complete code here