Diggin’ Dotfiles Collection

It’s time to dig the dotfiles, and to dig in to them! This post is a part of a blog post collection called Diggin’ Dotfiles where I unearth some gems from my personal dotfiles repo erikw/dotfiles.

The following articles are a part of this series:

• Manage shell configurations across different systems
• Wrap Dangerous Commands in pre-post ZFS or BRTFS Snapshots
• Finger of god: using sudo with TouchID on macOS
• Generate permutation shell aliases

Dangerous Commands in the Daily Life of a Sysadmin

We’re all the sysadmin of our machines nowadays and as we go on, we will often sway around with potentially very dangerous commands that has the potential to break the system.

The most common dangerous operation would probably be to install or upgrade some packages with a (systems) package manager, or maybe upgrade the OS itself even. Other commands are directly destructible and non-reversible like for example dd(1). For those running an OS that is working with rolling releases like FreeBSD or Arch Linux, it pays of to keep up with the development and upgrade frequent but small instead of seldom and big. This means however, executing dangerous commands becomes part of everyday life.

Does

$ sudo pacman -Syyu
$ # -or-
$ sudo freebsd-upgrade install

give you shivers? It should! But it must not…

Is there a way to make this safer? Yes!

Copy-on-write Snappshotting to the Rescue

COW Copy-on-write is a space-efficient storage management technique where copies are made in a soft fashion and only the diff from the source is recorded.
is a feature that some file systems offer that let’s you very easily make a whole (system) disk snapshot for free in terms of storage. Of course nothing is for free, but the idea is that the current live data and a snapshot at some time in the past share data that are not modified. Put in another way, the difference on disk between the live and a snapshot is only those files that are different. Thus this has the potential to be quite storage efficient

Two file systems that I’ve been using a lot the last 10 years are Brtfs A Linux native file system with features like copy-on-write and easy logical volume management.
on my Arch Probably the best Linux distribution you can get.
Linux system and ZFS A well honored filesystem and volume manager popular on BSD-derived operating systems.
on my FreeBSD one. The both provide full file system snapshots and tools to view, restore, diff etc. these snapshots with each other and the current live data.

snp & znp: Snapshot Wrappers for Commands

Over time I whipped up two scripts for the respective file systems that basically

1. makes a pre-snapshot of the full system.
2. executes the supplied system command.
3. makes a post-snapshot of the full system.

Additionally the output of the command is logged to /var/local/log/ with a timestamp.

The script can be executed by just being a “prefix” to the normal command you would do:

$ sudo snp echo "not-so-dangerous-command"
$ sudo snp pacman -Syyu # a bit more dangerous command mitigated
$ sudo znp freebsd-upgrade install # quite serious...
$ # super dangerous if a space is added before * by mistake
$ sudo znp rm -rf abc*  

With this, we can in case of issues easily do a diff between these two snapshots and see what really happened on the file system level when those packages got upgraded or whatever the command did. Furthermore an issue might arise only after a few days and you don’t have access to the output of that command anymore. Then we just go to the log in /var/local/log/ to refresh the mind and the proceed to inspect the logged snapshot numbers listed in the logs.

I can tell you, over the course of the years these snapshots have saved me (and my systems) many times! I made it a habit to use these snapshots frequently as they are cheap anyways. You should figure out a way to clean up snapshots by setting up a retention policy scheme as well. Additionally if you’re getting in to snapshotting, why not make snapshots all the time? For BSD there’s the excellent zfstools that let’s you do exactly that, snapshots on a (cron) time schedule.

snp

Hosted on Gist at erikw/snp.

znp

Hosted on Gist at erikw/znp.