Server TLS

Every service on February that accepts a connection needs a TLS certificate. The certificate is what lets a client verify it is talking to the right server, and what makes the connection encrypted. Without one, every connection is either plaintext or throwing up a browser warning that most people will click through without reading.

The complication is that February hosts two different kinds of service: some are publicly accessible (a webmail interface, a Nextcloud instance, maybe a Kavita ebook server), and some are private to the homelab network (ChirpStack, Grafana, internal monitoring, MariaDB). These two categories need different certificates from different sources, and mixing them up creates problems that are annoying to unpick later.

The two-track approach

Public services get Let’s Encrypt certificates.

Let’s Encrypt is a free, automated, publicly trusted certificate authority. Any device on the internet trusts Let’s Encrypt certificates without any additional configuration. For services that are reachable from outside the network, this is the right answer. The certificates are valid for 90 days and renew automatically via a systemd timer. There is no manual certificate management to do.

Private services get certificates from the internal CA.

The internal CA (covered in the CA section of this series) is a private certificate authority that only the homelab trusts. For services that never need to be reachable from the public internet, an internal CA certificate is cleaner and more appropriate than a Let’s Encrypt certificate. It does not expire in 90 days, it does not require port 80 to be reachable for validation, and it works for hostnames that are not in public DNS.

The practical split for February looks like this:

Services that sit behind nginx as a reverse proxy only need the certificate on nginx itself. nginx terminates TLS and proxies the request over plain HTTP to the backend service on localhost. The backend service does not need its own certificate in that case.

Let’s Encrypt with Certbot

Installation

sudo apt install certbot python3-certbot-nginx

The python3-certbot-nginx plugin handles nginx configuration automatically. It can modify nginx config files to add TLS settings and redirect HTTP to HTTPS.

Prerequisites

For Let’s Encrypt to issue a certificate, it needs to verify that you control the domain. The standard method (HTTP-01 challenge) works by placing a file at a known path on the server and having Let’s Encrypt’s servers fetch it over HTTP. This requires:

• A public DNS record pointing the domain to your server’s public IP address
• Port 80 open on your router and forwarded to February
• Port 443 open on your router and forwarded to February

If the service will only be reachable via VPN and does not have a public DNS record, use the internal CA instead. Let’s Encrypt cannot issue certificates for hostnames that are not in public DNS.

Obtaining a certificate

For a single domain:

sudo certbot certonly --nginx -d nextcloud.yourdomain.net

The certonly flag obtains the certificate without modifying nginx configuration. This keeps nginx configuration under your control rather than letting certbot rewrite it.

For multiple domains on the same certificate:

sudo certbot certonly --nginx \
    -d nextcloud.yourdomain.net \
    -d mail.yourdomain.net

Certbot stores certificates at /etc/letsencrypt/live/nextcloud.yourdomain.net/:

cert.pem       # The certificate
chain.pem      # The intermediate CA chain
fullchain.pem  # cert.pem + chain.pem (use this in nginx)
privkey.pem    # The private key

In nginx, reference the certificate like this:

ssl_certificate     /etc/letsencrypt/live/nextcloud.yourdomain.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/nextcloud.yourdomain.net/privkey.pem;

Automatic renewal

Certbot installs a systemd timer on Ubuntu that renews certificates automatically when they are within 30 days of expiry. Check it is active:

sudo systemctl status certbot.timer

Test that renewal would work without actually renewing:

sudo certbot renew --dry-run

If the dry run succeeds, automatic renewal is working. The timer runs twice daily, so in practice certificates renew well before they expire.

nginx needs to reload after renewal to pick up the new certificate. Add a deploy hook to handle this:

sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh << 'EOF'
#!/bin/bash
systemctl reload nginx
EOF

sudo chmod 0755 /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh

Internal CA certificates

For private services, certificates come from the internal CA. The CA section of this series covers building the CA itself. This section covers generating keys and certificate signing requests (CSRs) on February, and installing the signed certificate once the CA returns it.

Directory structure

Create a structured directory for server TLS material:

sudo mkdir -p /etc/ssl/{certreqs,certs,private}
sudo chmod 0700 /etc/ssl/private

OpenSSL configuration

Create a server certificate configuration file. This is reused for every internal service certificate by setting the CN environment variable before running OpenSSL.

sudo tee /etc/ssl/openssl-server.cnf << 'EOF'
#
# OpenSSL configuration for server certificate signing requests.
#
# Requires the following environment variable to be set:
#   export CN=service.yourdomain.net
#

CN = $ENV::CN

[ req ]
default_bits        = 4096
default_keyfile     = /etc/ssl/private/$ENV::CN.key.pem
encrypt_key         = no
default_md          = sha256
req_extensions      = server_req_ext
prompt              = no
distinguished_name  = req_distinguished_name
string_mask         = utf8only
utf8                = yes

[ server_req_ext ]
keyUsage                = critical, digitalSignature, keyEncipherment
extendedKeyUsage        = serverAuth, clientAuth
subjectKeyIdentifier    = hash
subjectAltName          = @subj_alt_names

[ req_distinguished_name ]
countryName             = GB
stateOrProvinceName     = England
localityName            = Manchester
organizationName        = yourdomain.net
commonName              = $CN

[ subj_alt_names ]
DNS.1 = $CN
EOF

Server keys are not passphrase-protected. This is intentional: a server key with a passphrase cannot be loaded at boot without manual intervention, which defeats the purpose of running services automatically. The /etc/ssl/private/ directory has permissions set to 0700 to compensate.

Generating a key and CSR

Set the hostname for the service, then generate:

export CN=grafana.yourdomain.net
export OPENSSL_CONF=/etc/ssl/openssl-server.cnf

sudo openssl req -new \
    -out /etc/ssl/certreqs/${CN}.req.pem

This creates:

• /etc/ssl/private/grafana.yourdomain.net.key.pem - the private key
• /etc/ssl/certreqs/grafana.yourdomain.net.req.pem - the CSR

Lock down the key:

sudo chmod 0400 /etc/ssl/private/${CN}.key.pem

Verify the CSR looks correct before sending it to the CA:

sudo openssl req -verify \
    -in /etc/ssl/certreqs/${CN}.req.pem \
    -noout -text \
    -reqopt no_version,no_pubkey,no_sigdump \
    -nameopt multiline

The output should show the correct CN and the SAN entry matching it. If the CA section is not set up yet, save the CSR and come back to this step once it is.

Installing the signed certificate

Once the CA returns the signed certificate, install it:

sudo cp grafana.yourdomain.net.cert.pem /etc/ssl/certs/
sudo chmod 0644 /etc/ssl/certs/grafana.yourdomain.net.cert.pem

For services that need the full chain (certificate plus intermediate CA), concatenate them:

sudo cat /etc/ssl/certs/grafana.yourdomain.net.cert.pem \
         /etc/ssl/certs/intermed-ca.cert.pem \
    > /etc/ssl/certs/grafana.yourdomain.net.chained.cert.pem

Use the chained file in service configuration wherever the service documentation asks for the certificate file.

Trusting the internal CA on the server itself

Some services on February connect to other internal services as clients (Postfix talking to another mail server, borgmatic connecting to the NAS, monitoring agents connecting to local services). For those connections to trust internally-signed certificates, the root CA certificate needs to be in the system trust store:

sudo mkdir -p /usr/local/share/ca-certificates/yourdomain.net
sudo cp root-ca.cert.pem \
    /usr/local/share/ca-certificates/yourdomain.net/root-ca.crt
sudo update-ca-certificates

After this, any service that uses the system trust store (/etc/ssl/certs/ca-certificates.crt) will trust certificates signed by the internal CA without additional configuration.

Certificate expiry monitoring

A certificate that expires and is not renewed takes a service offline for everyone. The ssl-cert-check utility monitors certificate expiry dates and sends email if a certificate is approaching expiry.

sudo apt install ssl-cert-check

Create a list of services to monitor. The format is hostname port:

sudo tee /etc/ssl/cert-check-list.txt << 'EOF'
nextcloud.yourdomain.net 443
mail.yourdomain.net 443
grafana.yourdomain.net 443
chirpstack.yourdomain.internal 8080
EOF

Test the check manually:

sudo ssl-cert-check -f /etc/ssl/cert-check-list.txt -a -e root@yourdomain.net -x 30

The -x 30 flag warns about certificates expiring within 30 days. The -a flag sends the notification email.

Add a cron job to run this weekly:

sudo tee /etc/cron.weekly/ssl-cert-check << 'EOF'
#!/bin/bash
ssl-cert-check \
    -f /etc/ssl/cert-check-list.txt \
    -a \
    -e root@yourdomain.net \
    -x 30
EOF

sudo chmod 0755 /etc/cron.weekly/ssl-cert-check

Let’s Encrypt certificates renew automatically, so the main value of this monitoring is catching internal CA certificates approaching expiry, since those do not renew themselves.

What is next

With TLS in place, individual services can be configured with confidence that connections to them are encrypted and verified. The nginx reverse proxy article covers the server block configuration that applies these certificates to web-facing services. The CA article covers building the internal CA that signs private service certificates.