Wrap Dangerous Commands in pre-post ZFS or Btrfs Snapshots

Diggin’ Dotfiles CollectionPermalink

It’s time to dig the dotfiles, and to dig in to them! This post is a part of a blog post collection called Diggin’ Dotfiles where I unearth some gems from my personal dotfiles repo erikw/dotfiles.

The following articles are a part of this series:

• Manage shell configurations across different systems
• Wrap Dangerous Commands in pre-post ZFS or BRTFS Snapshots
• Finger of god: using sudo with TouchID on macOS
• Generate permutation shell aliases
• Git hacks

Dangerous Commands in the Daily Life of a SysadminPermalink

We’re all the sysadmin of our machines nowadays and as we go on, we will often sway around with potentially very dangerous commands that has the potential to break the system.

The most common dangerous operation would probably be to install or upgrade some packages with a (systems) package manager, or maybe upgrade the OS itself even. Other commands are directly destructible and non-reversible, like for example dd(1). For those running an OS that is working with rolling releases like FreeBSD or Arch Linux, it pays off to keep up with the development and upgrade frequently but in small increments instead of seldom and in big jumps. This means, however, that executing dangerous commands becomes part of everyday life.

Does

$ sudo pacman -Syyu
$ # -or-
$ sudo freebsd-upgrade install

give you shivers? It should! But it must not…

Is there a way to make this safer? Yes!

Copy-on-write Snapshotting to the RescuePermalink

COW (Copy-on-write is a space-efficient storage management technique where copies are made in a soft fashion and only the diff from the source is recorded.
) is a feature that some file systems offer that lets you very easily make a whole (system) disk snapshot for free in terms of storage. Of course, nothing is for free, but the idea is that the current live data and a snapshot at some time in the past share data that are not modified. Put another way, the difference on disk between the live system and a snapshot is only those files that are different. Thus this has the potential to be quite storage efficient.

Two file systems that I’ve been using a lot over the last 10 years are Btrfs (A Linux native file system with features like copy-on-write and easy logical volume management.
) on my Arch (Probably the best Linux distribution you can get.
) Linux system and ZFS (A well honored filesystem and volume manager popular on BSD-derived operating systems.
) on my FreeBSD one. They both provide full file system snapshots and tools to view, restore, diff, etc. these snapshots with each other and the current live data.

snp & znp: Snapshot Wrappers for CommandsPermalink

Over time I whipped up two scripts for the respective file systems that basically

1. makes a pre-snapshot of the full system.
2. executes the supplied system command.
3. makes a post-snapshot of the full system.

Additionally the output of the command is logged to /var/local/log/ with a timestamp.

The script can be executed by just being a “prefix” to the normal command you would do:

$ sudo snp echo "not-so-dangerous-command"
$ sudo snp pacman -Syyu # a bit more dangerous command mitigated
$ sudo znp freebsd-upgrade install # quite serious...
$ # super dangerous if a space is added before * by mistake
$ sudo znp rm -rf abc*  

With this, we can in case of issues easily do a diff between these two snapshots and see what really happened on the file system level when those packages got upgraded or whatever the command did. Furthermore an issue might arise only after a few days and you don’t have access to the output of that command anymore. Then we just go to the log in /var/local/log/ to refresh the mind and the proceed to inspect the logged snapshot numbers listed in the logs.

I can tell you, over the course of the years these snapshots have saved me (and my systems) many times! I made it a habit to use these snapshots frequently, as they are cheap anyway. You should figure out a way to clean up snapshots by setting up a retention policy scheme as well. Additionally, if you’re getting into snapshotting, why not make snapshots all the time? For BSD there’s the excellent zfstools that lets you do exactly that: snapshots on a cron-based schedule.

snpPermalink

Hosted on Gist at erikw/snp.

znpPermalink

Hosted on Gist at erikw/znp.