ZFS Dataset Structure

After a lot of experimenting, as of August 2nd 2017 I was using the following filesystem heirarchy for my ZFS datasets during system setup when using Arch.

Dataset Structure

I'll use a few variables to represent different locations in the pool for datasets.

  • SYS_ROOT=vault/sys - The location of any systems on the pool.

  • DATA_ROOT=vault/data - System shared data.

Boot Environments

For boot environments I use the following configuration. SYSTEM_NAME can be anything, I use the hostname.

${SYS_ROOT}/${SYSTEM_NAME}/ROOT/${BOOT_ENV}

For example, my current boot environment which will be mounted to /:

vault/sys/chin/ROOT/default

In this configuration it makes it easy to dual boot multiple systems off of a single ZFS pool. To create a new system just add a new dataset under vault/sys, and set it up as normal. This should even work dual booting Linux and FreeBSD.

Datasets

While only a dataset for / really needs creating, I create quite a few. This lets me backup and snapshot only datasets I find important.

Setup datasets. Set all besides / legacy, or use zfs management. I like using legacy for multi system setups using a shared pool, and zfs for single install systems.

Boot environment Dataset

The boot environment will be mounted to / and store everything that doesnt have it's own mounted dataset.

zfs create -o mountpoint=none ${SYS_ROOT}; \
zfs create -o mountpoint=none ${SYS_ROOT}/${SYSTEM_NAME}; \
zfs create -o mountpoint=none ${SYS_ROOT}/${SYSTEM_NAME}/ROOT; \
zfs create -o mountpoint=/ ${SYS_ROOT}/${SYSTEM_NAME}/ROOT/${BOOT_ENV}

canmount=off Datasets

Set /var, /var/lib and /usr to canmount=off meaning they're not mounted and are only there to create the directory structure. This will put their data in the boot environment dataset.' Their properties will be inherited.

zfs create -o canmount=off -o mountpoint=/var -o xattr=sa ${SYS_ROOT}/${SYSTEM_NAME}/var; \
zfs create -o canmount=off -o mountpoint=/var/lib ${SYS_ROOT}/${SYSTEM_NAME}/var/lib; \
zfs create -o canmount=off -o mountpoint=/var/lib/systemd ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/systemd; \
zfs create -o canmount=off -o mountpoint=/usr ${SYS_ROOT}/${SYSTEM_NAME}/usr

Regular Datasets

The other datasets will be independent from the boot environment and will not change between boot environments.

System Datasets

I keep some datasets like /var/cache's' dataset seperate to avoid having to snapshot and backup their data. I also keep /var/log 's' dataset seperate so the logs are always available as well as the datasets for my containers and VMs.

Turn on posixacls for systemd-journald's /var/log/journal dataset.

zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/systemd/coredump; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/log; \
zfs create -o mountpoint=legacy -o acltype=posixacl ${SYS_ROOT}/${SYSTEM_NAME}/var/log/journal; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/lxc; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/lxd; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/machines; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/libvirt; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/cache; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/usr/local

User Datasets

I create extensive user datasets, outside the boot environment.

zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local; \
zfs create -o mountpoint=/home/john/.local/share -o canmount=off ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local/share; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local/share/Steam; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john/config; \
zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john/cache

As of zfsonlinux 0.7.0 ZFS delegation using zfs allow works on linux. I delegate all datasets under ${SYS_ROOT}/${SYSTEM_NAME}/home/john to my user 'john' giving the abiity to snapshot and create datasets.

zfs allow john create,mount,mountpoint,snapshot ${SYS_ROOT}/${SYSTEM_NAME}/home/john

Checking permissions shows john's permissions.

zfs allow ${SYS_ROOT}/${SYSTEM_NAME}/home/john
---- Permissions on vault/sys/chin/home/john -------------------------
Local+Descendent permissions:
        user john create
[root@chin ~]# zfs allow john snapshot ${SYS_ROOT}/${SYSTEM_NAME}/home/john
[root@chin ~]# zfs allow ${SYS_ROOT}/${SYSTEM_NAME}/home/john
---- Permissions on vault/sys/chin/home/john -------------------------
Local+Descendent permissions:
        user john create,snapshot

Available options:

NAME             TYPE           NOTES
allow            subcommand     Must also have the permission that is
                                being allowed
clone            subcommand     Must also have the 'create' ability and
                                'mount'
                                ability in the origin file system
create           subcommand     Must also have the 'mount' ability
destroy          subcommand     Must also have the 'mount' ability
hold             subcommand     Allows adding a user hold to a snapshot
mount            subcommand     Allows mount/umount of ZFS datasets
promote          subcommand     Must also have the 'mount' and 'promote'
                                ability in the origin file system
receive          subcommand     Must also have the 'mount' and 'create'
                                ability
release          subcommand     Allows releasing a user hold which
                                might destroy the snapshot
rename           subcommand     Must also have the 'mount' and 'create'
                                ability in the new parent
rollback         subcommand
send             subcommand
share            subcommand     Allows sharing file systems over NFS or
                                SMB protocols
snapshot         subcommand
groupquota       other          Allows accessing any groupquota@...
                                property
groupused        other          Allows reading any groupused@... property
userprop         other          Allows changing any user property
userquota        other          Allows accessing any userquota@...
                                property
userused         other          Allows reading any userused@... property
aclinherit       property
aclmode          property
atime            property
canmount         property
casesensitivity  property
checksum         property
compression      property
copies           property
dedup            property
devices          property
exec             property
logbias          property
mlslabel         property
mountpoint       property
nbmand           property
normalization    property
primarycache     property
quota            property
readonly         property
recordsize       property
refquota         property
refreservation   property
reservation      property
secondarycache   property
setuid           property
shareiscsi       property
sharenfs         property
sharesmb         property
snapdir          property
utf8only         property
version          property
volblocksize     property
volsize          property
vscan            property
xattr            property
zoned            property

Data Datasets

I'll be mounting these under ${HOME}. They exist outside the different systems and are shared between them.

zfs create -o mountpoint=none ${DATA_ROOT}; \
zfs create -o mountpoint=legacy ${DATA_ROOT}/Books; \
zfs create -o mountpoint=legacy ${DATA_ROOT}/Computer; \
zfs create -o mountpoint=legacy ${DATA_ROOT}/Personal; \
zfs create -o mountpoint=legacy ${DATA_ROOT}/Pictures; \
zfs create -o mountpoint=legacy ${DATA_ROOT}/University; \
zfs create -o mountpoint=legacy ${DATA_ROOT}/Workspace; \
zfs create -o mountpoint=legacy ${DATA_ROOT}/Reference

Final Structure

So my system ends up as.

zfs list -o name | grep -E 'chin|data'

vault/data                                   768K   860G    96K  none
vault/data/Books                              96K   860G    96K  legacy
vault/data/Computer                           96K   860G    96K  legacy
vault/data/Personal                           96K   860G    96K  legacy
vault/data/Pictures                           96K   860G    96K  legacy
vault/data/Reference                          96K   860G    96K  legacy
vault/data/University                         96K   860G    96K  legacy
vault/data/Workspace                          96K   860G    96K  legacy
vault/sys/chin                              1.97M   860G    96K  none
vault/sys/chin/ROOT                          192K   860G    96K  none
vault/sys/chin/ROOT/default                   96K   860G    96K  /
vault/sys/chin/home                          672K   860G    96K  legacy
vault/sys/chin/home/john                     576K   860G    96K  legacy
vault/sys/chin/home/john/cache                96K   860G    96K  legacy
vault/sys/chin/home/john/config               96K   860G    96K  legacy
vault/sys/chin/home/john/local               288K   860G    96K  legacy
vault/sys/chin/home/john/local/share         192K   860G    96K  /home/john/.local/share
vault/sys/chin/home/john/local/share/Steam    96K   860G    96K  legacy
vault/sys/chin/usr                           192K   860G    96K  /usr
vault/sys/chin/usr/local                      96K   860G    96K  legacy
vault/sys/chin/var                           864K   860G    96K  /var
vault/sys/chin/var/cache                      96K   860G    96K  legacy
vault/sys/chin/var/lib                       576K   860G    96K  /var/lib
vault/sys/chin/var/lib/lxc                    96K   860G    96K  legacy
vault/sys/chin/var/lib/lxd                    96K   860G    96K  legacy
vault/sys/chin/var/lib/machines               96K   860G    96K  legacy
vault/sys/chin/var/lib/libvirt                96K   860G    96K  legacy
vault/sys/chin/var/lib/systemd               192K   860G    96K  /var/lib/systemd
vault/sys/chin/var/lib/systemd/coredump       96K   860G    96K  legacy
vault/sys/chin/var/log                        96K   860G    96K  legacy

Install Preperation

Using this structure datasets must be mounted in the correct order.

ZFS Setup

Import zpool and mount root dataset:

zpool import -d /dev/disk/by-id -R /mnt vault
mount -t zfs vault/sys/chin/ROOT/default /mnt

After dataset creation, create cachefile.

zpool set cachefile=/etc/zfs/zpool.cache vault
mkdir -p /mnt/etc/zfs && cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache

Mount system datasets:

mkdir -p /mnt/usr/local
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/usr/local /mnt/usr/local; \

mkdir -p /mnt/var/cache
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/cache /mnt/var/cache; \

mkdir -p /mnt/var/lib/{lxc,lxd,machines,libvirt,systemd/coredump} /mnt/var/log; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/lxc /mnt/var/lib/lxc; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/lxd /mnt/var/lib/lxd; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/machines /mnt/var/lib/machines; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/libvirt /mnt/var/lib/libvirt; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/systemd/coredump /mnt/var/lib/systemd/coredump; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/log /mnt/var/log; \
mkdir /mnt/var/log/journal; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/log/journal /mnt/var/log/journal

Mount home.

mkdir -p /mnt/home ; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home /mnt/home; \

mkdir -p /mnt/home/john; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john /mnt/home/john; \

mkdir -p /mnt/home/john/{.cache,.config,.local}; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john/cache /mnt/home/john/.cache; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john/config /mnt/home/john/.config; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local /mnt/home/john/.local; \

mkdir -p /mnt/home/john/.local/share/Steam; \
mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local/share/Steam /mnt/home/john/.local/share/Steam

Mount data:

mkdir -p /mnt/home/john/{Books,Computer,Personal,Pictures,Reference,University,Workspace}; \
mount -t zfs vault/data/Books /mnt/home/john/Books; \
mount -t zfs vault/data/Computer /mnt/home/john/Computer; \
mount -t zfs vault/data/Personal /mnt/home/john/Personal; \
mount -t zfs vault/data/Pictures /mnt/home/john/Pictures; \
mount -t zfs vault/data/Reference /mnt/home/john/Reference; \
mount -t zfs vault/data/University /mnt/home/john/University; \
mount -t zfs vault/data/Workspace /mnt/home/john/Workspace

Boot Setup

Create esp, (EF00) for regular install.

gdisk /dev/sdf
mkfs.fat -F32 /dev/sdf1
mount /dev/sdf1 /mnt/boot

I keep it at /mnt/efi instead, and bindmount kernel directory to /boot.

mkdir -p /mnt/mnt/efi
mount /dev/sdf1 /mnt/mnt/efi
mkdir -p /mnt/boot /mnt/mnt/efi/installs/chin
mount --bind /mnt/mnt/efi/installs/chin /mnt/boot

Swap

Create 32GiB partition and create swap.

mkswap /dev/sdf2
swapon /dev/sdf2

fstab Configuration

Create fstab, adding all currently mounted filesystems.

genfstab -U -p /mnt >> /mnt/etc/fstab

Get swap UUID and add to fstab.

lsblk -no UUID /dev/sdf2
UUID=4b00ce42-d400-4060-9329-622c420f367e none swap defaults 0 0

Now all partitions and datasets should be setup, check that the fstab looks correct.

PreviousGaming with WineNextRaspberry PI Secure VPN Torrentbox

Last updated