System Configuration Backup

Posted on 3 2026

The system configuration backup covers everything that would be lost in a fresh install: /etc, the boot configuration, locally installed software, package lists, custom scripts, and certificates. It does not cover user data (that is the next section) and it is not an image backup. The goal is to capture what is specific to this machine so that a fresh Kubuntu install plus a configuration restore gets back to a working state without having to reconstruct everything from memory.

This backup runs as root, targeting a dedicated Borg repository on the NAS, and schedules via the system-level anacron rather than a user session.

Prerequisites

Install borgmatic. On Kubuntu 24.04, the borgmatic package in the Ubuntu repositories may be significantly out of date. Use pipx for the latest version without conflicting with the system Python:

sudo apt install borgbackup pipx
sudo pipx install borgmatic --global

The --global flag installs borgmatic to /usr/local/bin/ making it available to all users including root. Verify:

sudo borgmatic --version

Borg preparation

Create the configuration and key directories for the root-level Borg installation:

sudo mkdir -p /etc/borg/{keys,ssh}
sudo mkdir -p /var/lib/borg/{cache,security}
sudo chmod 700 /etc/borg
sudo chmod 700 /etc/borg/keys
sudo chmod 700 /etc/borg/ssh

Generate a dedicated SSH key for the system backup. This key is separate from your personal SSH key and is used only for authenticating the Borg backup connection:

sudo ssh-keygen -t ed25519 \
    -C "BorgBackup-system@$(hostname -f)" \
    -f /etc/borg/ssh/id_ed25519 \
    -N ""
sudo chmod 0600 /etc/borg/ssh/id_ed25519

Display the public key to copy to the NAS:

sudo cat /etc/borg/ssh/id_ed25519.pub

Install this public key on the NAS backup server. The NAS section of this series covers configuring the server side, including the forced command that restricts this key to Borg operations only and limits it to this client’s repository path.

Verify SSH connectivity to the NAS before proceeding:

sudo ssh -i /etc/borg/ssh/id_ed25519 borg-backup@nas.yourdomain.net

The connection should succeed and then immediately close, since the forced command on the server terminates the session after the Borg handshake.

Generate a passphrase

The backup repository is encrypted with a repokey. Generate a strong passphrase and store it in KeePassXC before initialising any repository:

xkcdpass --numwords 7

Store this passphrase in KeePassXC under Infrastructure > Backup > System Backup Repository Passphrase. You will need it if you ever need to restore from scratch on a new machine.

Mail notification script

Create the notification script that borgmatic calls when something goes wrong:

sudo mkdir -p /etc/borgmatic
sudo tee /etc/borgmatic/notify.sh << 'EOF'
#!/usr/bin/env bash
#
# Notify user of borgmatic backup error.
# Called by borgmatic with: notify.sh "{configuration_filename}" "{repository}" "{error}" "{output}"
#

mail -s "Borgmatic System Backup Error on $(hostname -s)" root <<MAIL

Borgmatic system backup on $(hostname -f) failed.

Configuration file: $1
Repository: $2
Error message: $3

Command output:
$4

For more information, query the systemd journal:
  journalctl -u borgmatic.service

MAIL
EOF

sudo chmod 0700 /etc/borgmatic/notify.sh

This sends mail to root, which the Postfix null client aliases to your actual address.

Borgmatic configuration

Borgmatic 1.8+ uses a flat configuration schema. The older nested schema with location:, storage:, and retention: top-level keys still works but is deprecated. The configuration below uses the current schema.

Create /etc/borgmatic/system.yaml:

# /etc/borgmatic/system.yaml
# System configuration backup for desktop
# See: https://torsion.org/borgmatic/docs/reference/configuration/

# ============================================================
# What to back up
# ============================================================

source_directories:
    - /boot
    - /etc
    - /opt
    - /root
    - /usr/local
    - /var/lib

exclude_patterns:
    - '**/.cache'
    - '**/cache'
    - /dev
    - /lost+found
    - /mnt
    - /proc
    - /run
    - /snap
    - /sys
    - /var/lib/container
    - /var/lib/docker
    - /var/lib/lxcfs
    - /var/run
    - /var/tmp/*

exclude_caches: true

exclude_if_present:
    - .nobackup

# ============================================================
# Where to store backups
# ============================================================

repositories:
    - path: ssh://borg-backup@nas.yourdomain.net/volume1/BorgBackup/{hostname}-system
      label: local-nas
    - path: ssh://borg-backup@lighthouse-nas.yourdomain.net/volume1/BorgBackup/{hostname}-system
      label: remote-nas

encryption_passphrase: "your seven word diceware passphrase here"

ssh_command: ssh -i /etc/borg/ssh/id_ed25519 -o BatchMode=yes -o VerifyHostKeyDNS=yes -o StrictHostKeyChecking=yes

borg_base_directory: /var/lib/borg
borg_config_directory: /etc/borg
borg_cache_directory: /var/lib/borg/cache
borg_keys_directory: /etc/borg/keys

archive_name_format: '{hostname}-system-{now}'
compression: zstd

# ============================================================
# Retention policy
# ============================================================

keep_daily: 7
keep_weekly: 4
keep_monthly: 6
keep_yearly: 1

# ============================================================
# Consistency checks
# ============================================================

checks:
    - name: repository
      frequency: 2 weeks
    - name: archives
      frequency: 4 weeks

# ============================================================
# Before backup hooks
# ============================================================

before_backup:
    # Save list of installed packages
    - dpkg --get-selections > /etc/borgmatic/installed-packages.txt
    # Save list of manually installed packages
    - apt-mark showmanual > /etc/borgmatic/manually-installed-packages.txt
    # Save list of enabled apt repositories
    - apt-cache policy > /etc/borgmatic/apt-repos.txt
    # Save list of pipx packages
    - pipx list > /etc/borgmatic/pipx-packages.txt 2>/dev/null || true

# ============================================================
# Error hooks
# ============================================================

on_error:
    - /etc/borgmatic/notify.sh "{configuration_filename}" "{repository}" "{error}" "{output}"

The before_backup hooks generate package lists before each backup run, so the backup always captures the current state of installed software. These lists go into /etc/borgmatic/ which is included in the backup via /etc.

Secure and validate the configuration

The configuration file contains the encryption passphrase in plaintext. Lock it down:

sudo chmod 0600 /etc/borgmatic/system.yaml
sudo chown root:root /etc/borgmatic/system.yaml
sudo chmod 0600 /etc/borgmatic/notify.sh

Validate the configuration:

sudo borgmatic config validate --config /etc/borgmatic/system.yaml

No output means no errors. Fix any reported issues before continuing.

Initialise the repositories

Initialise both repositories. The repokey encryption mode stores the key in the repository itself, protected by the passphrase. This means the repository is self-contained and can be accessed from any machine with the passphrase, without needing to copy a keyfile.

sudo borgmatic init \
    --config /etc/borgmatic/system.yaml \
    --encryption repokey \
    --verbosity 1

Borgmatic will initialise both repositories in sequence. It will prompt to confirm the passphrase for each.

Export a backup of the repository key. Even though repokey embeds the key in the repository, keeping an export is good practice:

sudo borg key export \
    ssh://borg-backup@nas.yourdomain.net/volume1/BorgBackup/$(hostname -s)-system \
    /etc/borg/keys/system-repokey.export
sudo chmod 0400 /etc/borg/keys/system-repokey.export

Store a copy of this exported key in your offline safe storage alongside the GPG key backup.

Run an interactive backup test

Run the first backup manually with verbose output to confirm everything works:

sudo borgmatic create \
    --config /etc/borgmatic/system.yaml \
    --verbosity 1 \
    --list \
    --stats

The first run will take longer than subsequent runs as all data is transferred. Subsequent runs transfer only changed files.

List the archives to verify the backup was created:

sudo borgmatic list --config /etc/borgmatic/system.yaml

Test extracting a single file to confirm the archive is readable:

sudo borgmatic extract \
    --config /etc/borgmatic/system.yaml \
    --archive latest \
    --path etc/hostname \
    --destination /tmp/borg-test
sudo cat /tmp/borg-test/etc/hostname
sudo rm -rf /tmp/borg-test

Scheduling via anacron

The source material this series draws on uses a systemd timer for scheduling. This series uses anacron throughout for desktop tasks, and the system backup follows the same pattern. Create a system-level anacron job:

sudo tee /etc/cron.daily/borgmatic-system << 'EOF'
#!/usr/bin/env bash
# Daily system configuration backup via borgmatic

# Do not run on battery power
if command -v on_ac_power >/dev/null 2>&1; then
    on_ac_power || exit 0
fi

# Run borgmatic with syslog logging
/usr/local/bin/borgmatic create \
    --config /etc/borgmatic/system.yaml \
    --syslog-verbosity 1 \
    --stats

EOF

sudo chmod 0755 /etc/cron.daily/borgmatic-system

This runs daily via the system anacron, which fires on boot if a day has passed since the last run. The battery check prevents the backup from running when the machine is on battery power, which matches the ConditionACPower=true behaviour of the original systemd service.

For a weekly schedule instead (more appropriate for system configuration that changes less frequently than user data):

sudo mv /etc/cron.daily/borgmatic-system /etc/cron.weekly/borgmatic-system

Verify the script runs correctly:

sudo run-parts --test /etc/cron.weekly

Monitoring

Check the borgmatic log for recent backup results:

sudo journalctl -t borgmatic --since "7 days ago"

List all archives:

sudo borgmatic list --config /etc/borgmatic/system.yaml

Check repository integrity:

sudo borgmatic check --config /etc/borgmatic/system.yaml --verbosity 1

View backup statistics:

sudo borgmatic info --config /etc/borgmatic/system.yaml

The encryption passphrase in /etc/borgmatic/system.yaml is stored in plaintext. The file permissions (600, root-owned) are the minimum protection. Store the passphrase in KeePassXC and keep a copy of the exported repository key in offline safe storage. Without either the passphrase or an accessible repository, the backup cannot be decrypted.