Further to my previous post, I've been playing with borg backups, in particular, how to efficiently back up KVM machine images (or any virtual images managed by virsh). Once you have a bunch of different borg backups, each with it's own settings / configuration, it turns out you really need yet another tool to manage the borg!

Borgmatic!

I struggled for a little while with the prospect of not being able to parameterise these configurations files, but on the other hand if you parameterise with a common template, one change has the potential to stuff many backups, so I've sort of come to terms with something coders might consider not DRY. I suppose if I had a lot of machine images to backup, I could generate these configurations files on the fly, but I only have a few so static is good for now.

So for each machine I now have something line this;


location:
    source_directories:
      - /machine_images/

    repositories:
        - (borgbase id).repo.borgbase.com:repo

    one_file_system: true
    patterns:
        - 'R /'
        - '+ /machine_images/(machine instance).qcow2'
        - '- /machine_images/*'

storage:
    encryption_passphrase: "(borgbase repo passphrase)"
    checkpoint_interval: 1800
    compression: zstd,3
    ssh_command: 'ssh -i ~/.ssh/my_borgbase_key'

retention:
    keep_secondly: 60
    keep_minutely: 60
    keep_hourly: 24
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6
    keep_yearly: 1

consistency:
    checks:
        - repository
        - archives

hooks:
    before_backup:
            - virsh -c qemu:///system
                             snapshot-create-as --domain (machine instance)
                             --name backup.qcow2
                             --no-metadata
                             --atomic
                             --quiesce
                             --disk-only
                             --diskspec vda,snapshot=external

    after_backup:
            - echo "Complete.........................." &&
              virsh -c qemu:///system blockcommit (machine instance) vda --active --pivot &&
              mv /machine_images/(machine instance).backup.qcow2 /machine_images/(machine instance.backup.qcow2.old

Which goes into .config/borgmatic.d/(machine instance).yaml .. all that remains is to run borgmatic, which will run through all the available configuration files and run each backup in turn, so in cron, all you need is;

0 0 * * * borgmatic -nc --stats 2>&1 |mail -s "Borgmatic backup" (email)

And you're away .. borgmatic also provides a handy shortcut in terms of managing credentials your various backups .. if you access borg directly you'll need the repo name, passphrase and ssh key in order to actually "do" anything, whereas borgmatic knows about all these attributes, so you can do something as simple as;

borgmatic -c .config/borgmatic.d/(instance).yaml list

To get a list of backups in the configured repository, i.e. once you tell borgmatic which configuration file to use, you can just add standard borg commands. To be honest I looked at Borgmatic initially and thought it looked a little hairy, but having dug in a little and experienced what happens when you try to manage half a dozen sets of credentials by hand, it's certainly worth a thought.

Example Run

# borgmatic -c .config/borgmatic_local.d/sandbox.yaml --progress --stats
Domain snapshot backup.qcow2 created
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
Deleted data:                    0 B                  0 B                  0 B
All archives:              241.63 GB             24.77 GB              1.92 GB
                       Unique chunks         Total chunks
Chunk index:                    1516                39503
------------------------------------------------------------------------------
Archive name: host-2020-04-17T15:12:16.471438
Archive fingerprint: (fingerprint)
Time (start): Fri, 2020-04-17 15:12:17
Time (end):   Fri, 2020-04-17 15:12:58
Duration: 40.95 seconds
Number of files: 1
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               16.11 GB              1.89 GB              3.29 MB
All archives:              257.74 GB             26.67 GB              1.92 GB

                       Unique chunks         Total chunks
Chunk index:                    1530                42331
------------------------------------------------------------------------------
Complete..........................
Successfully pivoted

Oh, an afterthought, the use of "--quiesce" when taking a "virtsh snapshot" helps with disc consistency, however you will need to install the QEMU Guest agent inside each virtual machine .. or if you prefer, just remove the --quiesce option.

apt install qemu-guest-agent