# README These notes are a collection of information and documentation I have collected on various subjects mostly related to system administration, Linux, and Unix/BSD. # Linux Section on Linux and related technology. # Ubiquity ``` sudo apt install docker-compose ``` `docker-compose.yml` ```yaml version: '3.7' services: unifi-db: image: mongo:4.4.29 container_name: unifi-db environment: MONGO_INITDB_ROOT_USERNAME: root MONGO_INITDB_ROOT_PASSWORD: GETYOUROWNPASSWORD MONGO_USER: unifi MONGO_PASS: GETYOUROWNPASSWORD MONGO_DBNAME: unifi MONGO_AUTHSOURCE: admin volumes: - /data/mongo:/data/db - ./init-mongo.sh:/docker-entrypoint-initdb.d/init-mongo.sh:ro restart: unless-stopped unifi-network-application: image: lscr.io/linuxserver/unifi-network-application:latest container_name: unifi-network-application environment: - PUID=1000 - PGID=1000 - TZ=Etc/UTC - MONGO_USER=unifi - MONGO_PASS=GETYOUROWNPASSWORD - MONGO_HOST=unifi-db - MONGO_PORT=27017 - MONGO_DBNAME=unifi - MONGO_AUTHSOURCE=admin - MEM_LIMIT=1024 #optional - MEM_STARTUP=1024 #optional - MONGO_TLS= #optional volumes: - /data/unifi:/config ports: - 8443:8443 - 3478:3478/udp - 10001:10001/udp - 8080:8080 - 1900:1900/udp #optional - 8843:8843 #optional - 8880:8880 #optional - 6789:6789 #optional - 5514:5514/udp #optional restart: unless-stopped ``` `init-mongo.sh` ``` #!/bin/bash mongo < /etc/modules-load.d/virtio.conf ``` Install the dependencies. * [libvirt](https://www.archlinux.org/packages/?name=libvirt) * KVM/QEMU * [qemu-headless](https://www.archlinux.org/packages/?name=qemu-headless) * Network * Nat/DHCP * [ebtables](https://www.archlinux.org/packages/?name=ebtables) * [dnsmasq](https://www.archlinux.org/packages/?name=dnsmasq) * bridged networking * [bridge-utils](https://www.archlinux.org/packages/?name=) * UEFI * [ovmf](https://www.archlinux.org/packages/?name=ovmf) * Frontends * [virt-manager](https://www.archlinux.org/packages/?name=virt-manager) * [virt-viewer](https://www.archlinux.org/packages/?name=virt-viewer) ```shell pacman -S libvirt qemu-headless ebtables dnsmasq bridge-utils virt-manager virt-viewer ovmf ``` ## ZVOL Backing Store I like to use ZFS ZVOL's as my backing store. ### Setup Create a dataset for ZVOLs: ```shell zfs create -o mountpoint=none vault/zvols ``` #### User ZVOLs Create a dataset for user 'john''s ZVOLs ```shell zfs create -o mountpoint=none vault/zvols/john ``` As of [zfsonlinux 0.7.0](https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.0) ZFS delegation using `zfs allow` works on linux. Delegate permissions giving the abiity to snapshot and create datasets. ```shell zfs allow john create,mount,mountpoint,snapshot vault/zvols/john ``` ### Create ZVOL To let guest do its own caching, use: * primarycache=metadata Create ZVOL for a new VM. Replace with name. Volumes still need to be created by root. ```shell zfs create -o mountpoint=none vault/zvols/john/libvirt zfs create -V 50G vault/zvols/john/libvirt/ -o primarycache=metadata -o compression=on ``` ## Authentication By default, anybody in the `wheel` group can authenticate with polkit as defined in `/etc/polkit-1/rules.d/50-default.rules` (see [Polkit#Administrator identities](https://wiki.archlinux.org/index.php/Polkit#Administrator_identities)). If you want passwordless authentication, as of libvirt 1.2.16, anyone in the `libvirt` group can access to the RW daemon socket by default. Create the group if it doesn't exist. ```shell groupadd libvirt ``` Add any users required to it. ```shell gpasswd -a john libvirt ``` Make sure to re-login after. ## System Service Enable libvirtd.service. ```shell systemctl enable --now libvirtd ``` To run only a user-session the daemon does not need to be enabled. ## Connect Test libvirt system-session: ```shell virsh -c qemu:///system ``` Test libvirt system user-session: ```shell virsh -c qemu:///session ``` ## UEFI Add the following to `/etc/libvirt/qemu.conf`. ```shell nano /etc/libvirt/qemu.conf ``` ```shell nvram = [ "/usr/share/ovmf/ovmf_code_x64.bin:/usr/share/ovmf/ovmf_vars_x64.bin" ] ``` I have found UEFI may not work if I haven't set the system user to `user = root` in `/etc/libvirt/qemu.conf`. and restart libvirtd ```shell systemctl restart libvirtd ``` ### User To use uefi as a user, note networking options are limited, move the nvram to a user readable location and add it to `~/.config/libvirt/qemu.conf`. ```shell cp -r /usr/share/ovmf /home/john/.config/libvirt/ovmf chown -R john:john /home/john/.config/libvirt/ovmf ``` Add the following to `/etc/libvirt/qemu.conf`. ```shell nano ~/.config/libvirt/qemu.conf ``` ```shell nvram = [ "/home/john/.config/libvirt/ovmf/ovmf_code_x64.bin:/home/john/.config/libvirt/ovmf/ovmf_vars_x64.bin" ] ``` ## Create Guest Use virsh or virt manager. ## Storage Select virtIO Network and storage for best performance. Select ZVOL raw device. Mine was `/dev/vault/zvols/john/libvirt/`. ### ZVOL Persistance If using a user session the block device might need to be changed to be owned by the user running the VM. Temporarily the device can be chown'd, but the owner will not live through reboot. For persistence [add a udev rule](https://github.com/johnramsden/docs/blob/gitbook/docs/operatingsystems/linux/distributions/archlinux/ramsdenj.com/2016/07/21/making-a-zvol-backed-virtualbox-vm-on-linux.html) by creating a new file `99-local-zvol.rules` in `/etc/udev/rules.d/` that contains the following (replacing the ZVOL path and user): ```shell # /etc/udev/rules.d/99-local-zvol.rules # Give persistant ownership of ZVOL to user KERNEL=="zd*" SUBSYSTEM=="block" ACTION=="add|change" PROGRAM="/lib/udev/zvol_id /dev/%k" RESULT=="vault/zvols/john/libvirt/win" OWNER="john" GROUP="john" MODE="0750" ``` Refresh the rules with `udevadm control --reload` ### VirtIO Install drivers: I [downloaded](https://fedoraproject.org/wiki/Windows_Virtio_Drivers#Direct_download) ISO and attached the drivers pre-install. At the "Where do you want to install Windows?" screen, select the option Load Drivers, uncheck the box for "Hide drivers that aren't compatible with this computer's hardware". Browse to the wanted driver(s) at: SCSI: "viostor\w10\amd64" Networking: "NetKVM\w10\amd64" ## Network To use another interface, don't configure anything on the host and select macvtap passthrough, and select the interface. Install then reboot. # Post Install Tasks First [setup AUR](https://github.com/johnramsden/docs/blob/gitbook/archlinux_aur_pacaur.html) ## Time Setup [time](https://wiki.archlinux.org/index.php/time) using [systemd-timesyncd](https://wiki.archlinux.org/index.php/Systemd-timesyncd). ```shell timedatectl set-ntp true timedatectl set-ntp 1 ``` ## Configure Reflector So you always have fresh mirrors, setup [reflector](https://www.archlinux.org/packages/?name=reflector). ```shell pacman -S reflector ``` Create service to select the 200 most recently synchronized HTTP or HTTPS mirrors, sort them by download speed, and overwrite the file `/etc/pacman.d/mirrorlist`. ```shell nano /etc/systemd/system/reflector.service ``` ```shell [Unit] Description=Pacman mirrorlist update [Service] Type=oneshot ExecStart=/usr/bin/reflector --latest 200 --protocol http --protocol https --sort rate --save /etc/pacman.d/mirrorlist ``` Create timer. ```shell nano /etc/systemd/system/reflector.timer ``` ```shell [Unit] Description=Run reflector weekly [Timer] OnCalendar=weekly RandomizedDelaySec=12h Persistent=true [Install] WantedBy=timers.target ``` That will run reflector weekly. ```shell systemctl enable --now reflector.timer ``` ## Configure SMTP I used to use ssmtp but since it's now unmaintained I've started using [Msmtp](https://wiki.archlinux.org/index.php/Msmtp). ```shell pacman -S msmtp msmtp-mta ``` Setup system default. ```shell cp /usr/share/doc/msmtp/msmtprc-system.example /etc/msmtprc ``` Example config file ```shell # msmtp system wide configuration file # A system wide configuration file with default account. defaults # The SMTP smarthost. host smtp.fastmail.com port 465 # Construct envelope-from addresses of the form "user@oursite.example". #auto_from on maildomain # Use TLS. tls on tls_starttls off # Activate server certificate verification tls_trust_file /etc/ssl/certs/ca-certificates.crt # Syslog logging with facility LOG_MAIL instead of the default LOG_USER. syslog LOG_MAIL aliases /etc/aliases # msmtp root account, inherit from 'default' account account default user from system@ # Terrible... # auth plain # password # or with passwordeval, # passwordeval "gpg --quiet --for-your-eyes-only --no-tty --decrypt ~/.msmtp-root.gpg" account root : default # password, see below ``` Set permissions. ```shell chmod 600 /etc/msmtprc ``` You can setup a [gpg encrypted passphrase](https://wiki.archlinux.org/index.php/Msmtp#Password_management) if using interactively. The other (not very good option) is setting with 'password' in config. Add aliases to `/etc/aliases`. ```shell root: root@ ``` If anything private is in /etc/msmtprc, secure the file [as shown](https://wiki.archlinux.org/index.php/SSMTP#Security) on the Arch wiki. Create an ssmtp group and set the owner of `/etc/msmtp` and the msmtp binary. ```shell groupadd msmtp chown :msmtp /etc/msmtprc chown :msmtp /usr/bin/msmtp ``` Make sure only root, and the msmtp group can access `msmtprc`, then et the SGID bit on the binary ```shell chmod 640 /etc/msmtprc chmod g+s /usr/bin/msmtp ``` Then add a pacman hook to always set the file permissions after the package has been upgraded: ```shell nano /usr/local/bin/msmtp-set-permissions ``` ```shell #!/bin/sh chown :msmtp /usr/bin/msmtp chmod g+s /usr/bin/msmtp ``` Make it executable: ```shell chmod u+x /usr/local/bin/msmtp-set-permissions ``` Now add the pacman hook: ```shell nano /usr/share/libalpm/hooks/msmtp-set-permissions.hook ``` ```shell [Trigger] Operation = Install Operation = Upgrade Type = Package Target = msmtp [Action] Description = Set msmtp permissions for security When = PostTransaction Exec = /usr/local/bin/msmtp-set-permissions ``` ### Test mail Send a test mail. ```shell echo "Text, more text." | /usr/bin/mail -s SUBJECT email@your.domain.com ``` ## ZFS Configuration I always set up snapshotting and replication as one of the first things I do on a new desktop. ### Enable Snapshots Install [zfs-auto-snapshot (AUR)](https://aur.archlinux.org/packages/zfs-auto-snapshot-git/) and setup snapshotting on all datasets. ```shell pacaur -S zfs-auto-snapshot-git systemctl enable --now zfs-auto-snapshot-daily.timer ``` Set all datasets to snapshot and disable any datasets that dont require snapshotting. ```shell for ds in $(zfs list -H -o name); do MP="$(zfs get -H -o value mountpoint $ds )"; if [ ${MP} == "legacy" ] || [ "${MP}" == "/" ]; then echo "${ds}: on"; zfs set com.sun:auto-snapshot=true ${ds}; else echo "${ds}: off"; zfs set com.sun:auto-snapshot=false ${ds}; fi; done ``` In one line: ```shell for ds in $(zfs list -H -o name); do MP="$(zfs get -H -o value mountpoint $ds )"; if [ ${MP} == "legacy" ] || [ "${MP}" == "/" ]; then echo "${ds}: on"; zfs set com.sun:auto-snapshot=true ${ds}; else echo "${ds}: off";zfs set com.sun:auto-snapshot=false ${ds}; fi; done ``` ### ZFS Replication With ZnapZend Install [ZnapZend (AUR)](https://aur.archlinux.org/packages/znapzend/) (it's a great tool, I maintain the AUR package). ```shell pacaur -S znapzend systemctl enable --now znapzend ``` Create a config for each dataset thet needs replicating, where SYSTEM will be a name for the dataset at `${POOL}/replication/${SYSTEM}` on the remote. Specify the remote user and IP as well. Here is a small script I use for my setup. The grep can be adjusted to exclude any datasets that are unwanted. ```shell #!/bin/sh REMOTE_POOL_ROOT="${1}" REMOTE_USER="${2}" REMOTE_IP="${3}" for ds in $(zfs list -H -o name | \ grep -E 'data/|default|john|usr/|var/|lib/' | \ grep -v cache); do echo "Creating: ${REMOTE_USER}@${REMOTE_IP}:${REMOTE_POOL_ROOT}/${ds}" # See ssh(1) for -tt # https://www.freebsd.org/cgi/man.cgi?query=ssh # In simple terms, force pseudo-terminal and pseudo tty ssh -tt ${REMOTE_USER}@${REMOTE_IP} \ "~/znap_check_dataset ${REMOTE_POOL_ROOT}/${ds}" znapzendzetup create --tsformat='%Y-%m-%d-%H%M%S' \ SRC '1d=>15min,7d=>1h,30d=>4h,90d=>1d' ${ds} \ DST:${REMOTE_IP} '1d=>15min,7d=>1h,30d=>4h,90d=>1d,1y=>1w,10y=>1month' \ "${REMOTE_USER}@${REMOTE_IP}:${REMOTE_POOL_ROOT}/${ds}" done ``` On remote I have a pre-znazendzetup script which makes sure the remote location exists. ```shell #!/bin/sh # Pre zapzendzetup script. Put in ~/znap_check_dataset on remote and run with ds="${1}" if [ "$(zfs list -H -o name "${ds}")" = "${ds}" ]; then echo "${ds} exists, running ZnapZend." else echo "Creating non-existant dataset ${ds}" zfs create -p "${ds}" zfs unmount "${ds}" echo "${ds} created, running ZnapZend." fi ``` I would then run, for chin on `replicator@`. ```shell ./znapcfg "tank/replication/chin" "replicator" "" ``` ### Scrub Setup a monthly scrub with a systemd unit and timercontaining the following. ```shell nano /usr/lib/systemd/system/zpool-scrub@.service ``` ```shell # /etc/systemd/system/zpool-scrub@.service [Unit] Description=Scrub ZFS Pool Requires=zfs.target After=zfs.target [Service] Type=oneshot ExecStartPre=-/usr/bin/zpool scrub -s %i ExecStart=/usr/bin/zpool scrub %i ``` ```shell nano /etc/systemd/system/zpool-scrub@.timer ``` ```shell [Unit] Description=Scrub ZFS pool weekly [Timer] OnCalendar=weekly Persistent=true [Install] WantedBy=timers.target ``` Enable for pool. ```shell systemctl enable --now zpool-scrub@vault.timer ``` ### Enable The ZFS Event Daemon If an SMTP or MTA is configured, setup [The ZFS Event Daemon (ZED)](https://ramsdenj.com/2016/08/29/arch-linux-on-zfs-part-3-followup.html#zed-the-zfs-event-daemon) ```shell nano /etc/zfs/zed.d/zed.rc ``` Ad an email and mail program and set verbosity. ```shell ZED_EMAIL_ADDR="root" ZED_EMAIL_PROG="mail" ZED_NOTIFY_VERBOSE=1 ``` Start and enable the daemon. ```shell systemctl enable --now zfs-zed.service ``` Start a scrub and check for an email. ```shell zpool scrub vault ``` ## Define Hostid [Define a hostid](https://ramsdenj.com/2016/06/23/arch-linux-on-zfs-part-2-installation.html#first-tasks) or problems arise at boot. ## smart Install [smartmontools](https://www.archlinux.org/packages/?name=smartmontools). ```shell pacman -S smartmontools ``` ### Tests Long or short tests can be run on a disk. A short test will check for device problems. The long test is just a short test plus complete disc surface examination. Long test example: ```shell smartctl -t long /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_152271401093 smartctl -t long /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_154501401266 smartctl -t long /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402487 smartctl -t long /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402657 smartctl -t long /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNSADA75563M ``` Veiw results: ```shell smartctl -H /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_152271401093 smartctl -H /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_154501401266 smartctl -H /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402487 smartctl -H /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402657 smartctl -H /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNSADA75563M ``` Or, veiw all test results. ```shell smartctl -l selftest /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_152271401093 smartctl -l selftest /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_154501401266 smartctl -l selftest /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402487 smartctl -l selftest /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402657 smartctl -l selftest /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNSADA75563M ``` Or detailed results. ```shell smartctl -a /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_152271401093 smartctl -a /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_154501401266 smartctl -a /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402487 smartctl -a /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402657 smartctl -a /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNSADA75563M ``` ### Daemon The smartd daemon can also run, periodically running tests and will send you a message if a problem occurs. Edit the configuration file at `/etc/smartd.conf`. ```shell nano /etc/smartd.conf ``` To check for all errors on a disk use the option `-a` after the disk ID. ```shell /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_152271401093 -a -m /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_154501401266 -a -m /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402487 -a -m /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402657 -a -m /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNSADA75563M -a -m ``` To test if your mail notification is working run a test, add `-m -M test` to the end of the config. This will run the test on the start of the daemon.: ```shell DEVICESCAN -m -M test ``` Start smartd: ```shell systemctl start smartd ``` My config looks like: ```shell /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_152271401093 -a -m /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_154501401266 -a -m /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402487 -a -m /dev/disk/by-id/ata-SanDisk_SDSSDXPS480G_164277402657 -a -m /dev/disk/by-id/ata-Samsung_SSD_840_EVO_250GB_S1DBNSADA75563M -a -m ``` ## nfs ```shell pacman -S nfs-utils systemctl enable --now rpcbind.service nfs-client.target remote-fs.target ``` ## Autofs Install [autofs](https://www.archlinux.org/packages/?name=autofs). ```shell pacman -S autofs ``` ```shell nano /etc/autofs/auto.master ``` Add or uncomment the following. ```shell /net -hosts --timeout=60 ``` Start and enable. ```shell systemctl enable --now autofs ``` ## User Cache I like to keep certain directories in tmpfs. It avoids extra writes to disk and can be faster since everything is stored in memory. ## Cleaning the cache I like periodically have my users cache directory cleaned. This can easily be done using tmpfiles.d. Create a new file in the `/etc/tmpfiles.d` directory. ```shell nano /etc/tmpfiles.d/home-cache.conf ``` Add a rule that will delete any file older than 10 days. ```shell # remove files in /home/john/.cache older than 10 days D /home/john/.cache 1755 john john 10d ``` # Package Management This section contains Arch Linux package management sections. # aurutils [Aurutils](https://github.com/AladW/aurutils) is an AUR helper. > aurutils is a collection of scripts to automate usage of the Arch User Repository, with different tasks such as package searching, update checks, or computing dependencies kept separate. > > The chosen approach for managing packages is local pacman repositories, rather than foreign (installed by "pacman -U") packages. ## Install Install from [aur package](https://aur.archlinux.org/packages/aurutils). [Add](https://wiki.archlinux.org/index.php/Pacman/Package_signing#Adding_unofficial_keys) developer key. ```shell $ gpg --recv-key 6BC26A17B9B7018A && gpg --lsign 6BC26A17B9B7018A ``` Optional dependencies: * aria2 (aria2-fast, aria2-git) (optional) – threaded downloads * devtools (devtools-git, devtools32-git) (optional) – systemd-nspawn support * expac (expac-git) (optional) – aursift script * parallel (parallel-rust) (optional) – threaded downloads * repose (repose-git) (optional) – repo-add alternative * vifm (vifm-git) (optional) – build file interaction I started with `aurutils devtools parallel vifm`. ## Setup Create a local repository config. *Adapted from man page* ```shell [root]# nano /etc/pacman.d/custom ``` ```shell [options] CacheDir = /var/cache/pacman/pkg CacheDir = /var/cache/pacman/custom CleanMethod = KeepCurrent [custom] SigLevel = Optional TrustAll Server = file:///var/cache/pacman/custom ``` Add config to the end of `/etc/pacman.conf`. ```shell [root]# nano /etc/pacman.conf ``` ```shell Include = /etc/pacman.d/custom ``` Create repository root and database: ```shell $ sudo install -d /var/cache/pacman/custom -o $USER $ repo-add /var/cache/pacman/custom/custom.db.tar ``` Sync repo. ```shell [root]# pacman -Syu ``` ## Build in chroot It's possible to build in a clean chroot (actually using systemd-nspawn container) with `makechrootpkg`. To install a package to the container use `pacman --root=/var/lib/aurbuild/x86_64/root -S zfs-linux`. ### Setup Container I set up a aurbuild root to be zfs dataset (optional). Could also use btrfs. ```shell [root]# zfs create /var/lib/aurbuild -o mountpoint=legacy [root]# mkdir /var/lib/aurbuild [root]# mount -t zfs /var/lib/aurbuild /var/lib/aurbuild ``` Add to fstab. ## Usage To get a list of all current existing AUR packages so that they can be migrated to `aurutils`, run `pacman -Qmq`. Attempt a build in a clean chroot of `google-chrome`. ```shell aursync -c google-chrome ``` Now it can be installed with `pacman -S google-chrome`. # Programming Languages This section describes Arch Linux programming language configuration and setup. # nodejs ### Ruby Arch Wiki: [ruby](https://wiki.archlinux.org/index.php/ruby) Install the [ruby](https://www.archlinux.org/packages/?name=ruby) package. Add ruby gem path to shell rc, ie `~/.zshrc`, or `~/.bashrc`. ```shell PATH="$(ruby -e 'print Gem.user_dir')/bin:${PATH}" ``` On Arch user gems will be installed to `~/.gem/ruby/` so they don't interact with anything installed by Pacman. #### Bundler Install bundler with. ```shell gem install bundler ``` Bundler by default installs gems system-wide. To change this default add the following to your shell rc. ```shell export GEM_HOME=$(ruby -e 'print Gem.user_dir') ``` Bundles can be installed explicitly at a certain location using: ```shell bundle install --path .bundle ``` This would install a bundle in the working directory inside of a .bundle directory. ### Node.js Arch Wiki: [nodejs](https://wiki.archlinux.org/index.php/Node.js) Install the [nodejs](https://www.archlinux.org/packages/?name=nodejs) package. To set up nodejs to store packages in working directories, add the following to your shell rc. ```shell export npm_config_prefix=${HOME}/.node_modules PATH="${HOME}/.node_modules/bin:${PATH}" ``` # ruby ### Ruby Arch Wiki: [ruby](https://wiki.archlinux.org/index.php/ruby) Install the [ruby](https://www.archlinux.org/packages/?name=ruby) package. Add ruby gem path to shell rc, ie `~/.zshrc`, or `~/.bashrc`. ```shell PATH="$(ruby -e 'print Gem.user_dir')/bin:${PATH}" ``` On Arch user gems will be installed to `~/.gem/ruby/` so they don't interact with anything installed by Pacman. #### Bundler Install bundler with. ```shell gem install bundler ``` Bundler by default installs gems system-wide. To change this default add the following to your shell rc. ```shell export GEM_HOME=$(ruby -e 'print Gem.user_dir') ``` Bundles can be installed explicitly at a certain location using: ```shell bundle install --path .bundle ``` This would install a bundle in the working directory inside of a .bundle directory. ### Node.js Arch Wiki: [nodejs](https://wiki.archlinux.org/index.php/Node.js) Install the [nodejs](https://www.archlinux.org/packages/?name=nodejs) package. To set up nodejs to store packages in working directories, add the following to your shell rc. ```shell export npm_config_prefix=${HOME}/.node_modules PATH="${HOME}/.node_modules/bin:${PATH}" ``` # Restore Installed Applications To list explicitly installed packages. ```shell pacman -Qqe > pkglist.txt ``` ## Regular repo Remove AUR packages from explicitly installed package list, and save to file `native.txt`. ```shell bash -c "comm -12 <(pacman -Slq | sort) <(sort pkglist.txt)" > native.txt ``` Install them on new system after selecting wanted packages. ```shell pacman -S - < native.txt ``` ## AUR List aur packages. Dont forget to edit. ```shell pacman -Qmq > aur.txt ``` On new system, install. ```shell pacaur -S - < aur.txt ``` # User Configuration Management To keep my home orgamized I use [vcsh](https://github.com/RichiH/vcsh/blob/master/doc/README.md) and [myrepos](http://myrepos.branchable.com/). ### Setup On a new system, install the requirements. ```shell aursync --update --temp --chroot myrepos vcsh ``` Clone an existing myrepos configuration from a users `$HOME`. ```shell vcsh clone git@github.com:johnramsden/mr.git ``` To clone a branch: Clone an existing myrepos configuration from a users `$HOME`. ```shell vcsh clone -b branch git@github.com:johnramsden/mr.git ``` Or the [vcsh template](https://github.com/RichiH/vcsh_mr_template) for a new setup. ```shell vcsh clone git@github.com:RichiH/vcsh_mr_template.git mr ``` It tracks the myrepos config. ```shell cat ~/.mrconfig ``` ```shell [DEFAULT] git_gc = git gc "$@" jobs = 5 include = cat ~/.config/mr/config.d/* ``` and myrepos template config. ```shell cat ~/.config/mr/available.d/mr.vcsh ``` ```shell [$HOME/.config/vcsh/repo.d/mr.git] checkout = vcsh clone git://github.com/RichiH/vcsh_mr_template.git mr ``` The templates are stored in `$HOME/.config/mr/available.d` and can be changed to point to your own myrepos config. ### Usage After adding a config for all your repos in `~/.config/mr/available.d/`, symlink the ones you want enabled to `~/.config/mr/config.d/`. To enable the mr config. ```shell cd ~/.config/mr/config.d ln -s ../available.d/mr.vcsh ``` Now, run `mr up` to clone the specified repos. ### Setup SSH To start ssh-agent with a systemd unit create `~/.config/systemd/user/ssh-agent.service`. ```shell nano ~/.config/systemd/user/ssh-agent.service ``` ```shell [Unit] Description=SSH key agent [Service] Type=simple Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket ExecStart=/usr/bin/ssh-agent -D -a $SSH_AUTH_SOCK [Install] WantedBy=default.target ``` Add `SSH_AUTH_SOCK DEFAULT="${XDG_RUNTIME_DIR}/ssh-agent.socket"` to `~/.pam_environment` Start and enable. ```shell systemctl --user enable --now ssh-agent ``` ### chezmoi cd \~/.local/share git clone --config transfer.fsckobjects=false --config receive.fsckobjects=false --config fetch.fsckobjects=false git://github.com/robbyrussell/oh-my-zsh.git 'oh-my-zsh' cd \~/.config/oh-my-zsh/custom/plugins git clone ':zsh-users/zsh-autosuggestions.git' 'zsh-autosuggestions' # User Namespaces Enable [user namespaces](https://wiki.archlinux.org/index.php/Linux_Containers#Enable_support_to_run_unprivileged_containers_.28optional.29) ## Requirements First enable the sysctl: ```shell echo 'sysctl kernel.unprivileged_userns_clone = 1' | tee /etc/sysctl.d/20-unprivileged_userns.conf ``` Reload sysctl's with `sysctl --system` ## User (G/U)IDs Setup LXC mappings in `/etc/lxc/default.conf`. ``` lxc.idmap = u 0 100000 65536 lxc.idmap = g 0 100000 65536 ``` Edit shadow files for g/uids ```shell cat /etc/subuid /etc/subgid ``` ``` root:100000:65536 john:165536:231072 root:100000:65536 john:165536:231072 ``` Now add changed mapping to userns containers. ## User Setup Setup directories. Similar paths: * /etc/lxc/lxc.conf => \~/.config/lxc/lxc.conf * /etc/lxc/default.conf => \~/.config/lxc/default.conf * /var/lib/lxc => \~/.local/share/lxc * /var/lib/lxcsnaps => \~/.local/share/lxcsnaps * /var/cache/lxc => \~/.cache/lxc Create zfs dataset for containers: ```shell mkdir ~/.local/share/lxc zfs create -o mountpoint=legacy vault/sys/wooly/home/john/local/share/lxc mount -t zfs vault/sys/wooly/home/john/local/share/lxc /home/john/.local/share/lxc chown -R john:john /home/john/.local/share/lxc ``` Add to fstab (double check it). ```shell genfstab -U / | grep /home/john/.local/share/lxc | tee --append /etc/fstab ``` Let user create up to 10 bridges. ```shell echo 'john veth lxcbr0 10' | tee --append /etc/lxc/lxc-usernet ``` *NOTE: May need to enable `haveged.service` (I got gpg errors without it).* ## Create Container ```shell lxc-create --template=download --name=tiger ``` ## References * * * # Gaming with Wine This post describes setting up wine for use on Arch Linux. ## Setup Install wine staging for much better performance. Optionally install [wine-mono](https://www.archlinux.org/packages/?name=wine_gecko) and [wine\_gecko](https://www.archlinux.org/packages/?name=wine-mono) see [wiki](https://wiki.archlinux.org/index.php/Wine#Installation) for info. ```shell pacman -S wine-staging wine-mono wine_gecko ``` ### Wine file associations To avoid having [wine file associations](https://wiki.archlinux.org/index.php/Wine#Unregister_existing_Wine_file_associations) unregister them. To prevent them set environment variable `WINEDLLOVERRIDES`. ```shell export WINEDLLOVERRIDES="winemenubuilder.exe=d" ``` # ZFS Dataset Structure After a lot of experimenting, as of August 2nd 2017 I was using the following filesystem heirarchy for my ZFS datasets during system setup when using Arch. ### Dataset Structure I'll use a few variables to represent different locations in the pool for datasets. * `SYS_ROOT=vault/sys` - The location of any systems on the pool. * `DATA_ROOT=vault/data` - System shared data. ### Boot Environments For boot environments I use the following configuration. SYSTEM\_NAME can be anything, I use the hostname. ```shell ${SYS_ROOT}/${SYSTEM_NAME}/ROOT/${BOOT_ENV} ``` For example, my current boot environment which will be mounted to `/`: ```shell vault/sys/chin/ROOT/default ``` In this configuration it makes it easy to dual boot multiple systems off of a single ZFS pool. To create a new system just add a new dataset under `vault/sys`, and set it up as normal. This should even work dual booting Linux and FreeBSD. ### Datasets While only a dataset for `/` really needs creating, I create quite a few. This lets me backup and snapshot only datasets I find important. Setup datasets. Set all besides `/` legacy, or use zfs management. I like using legacy for multi system setups using a shared pool, and zfs for single install systems. #### Boot environment Dataset The boot environment will be mounted to `/` and store everything that doesnt have it's own mounted dataset. ```shell zfs create -o mountpoint=none ${SYS_ROOT}; \ zfs create -o mountpoint=none ${SYS_ROOT}/${SYSTEM_NAME}; \ zfs create -o mountpoint=none ${SYS_ROOT}/${SYSTEM_NAME}/ROOT; \ zfs create -o mountpoint=/ ${SYS_ROOT}/${SYSTEM_NAME}/ROOT/${BOOT_ENV} ``` **canmount=off Datasets** Set `/var`, `/var/lib` and `/usr` to `canmount=off` meaning they're not mounted and are only there to create the directory structure. This will put their data in the boot environment dataset.' Their properties will be inherited. ```shell zfs create -o canmount=off -o mountpoint=/var -o xattr=sa ${SYS_ROOT}/${SYSTEM_NAME}/var; \ zfs create -o canmount=off -o mountpoint=/var/lib ${SYS_ROOT}/${SYSTEM_NAME}/var/lib; \ zfs create -o canmount=off -o mountpoint=/var/lib/systemd ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/systemd; \ zfs create -o canmount=off -o mountpoint=/usr ${SYS_ROOT}/${SYSTEM_NAME}/usr ``` #### Regular Datasets The other datasets will be independent from the boot environment and will not change between boot environments. **System Datasets** I keep some datasets like `/var/cache`'s' dataset seperate to avoid having to snapshot and backup their data. I also keep `/var/log` 's' dataset seperate so the logs are always available as well as the datasets for my containers and VMs. Turn on posixacls [for systemd-journald](https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html)'s /var/log/journal dataset. ```shell zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/systemd/coredump; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/log; \ zfs create -o mountpoint=legacy -o acltype=posixacl ${SYS_ROOT}/${SYSTEM_NAME}/var/log/journal; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/lxc; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/lxd; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/machines; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/libvirt; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/var/cache; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/usr/local ``` **User Datasets** I create extensive user datasets, outside the boot environment. ```shell zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local; \ zfs create -o mountpoint=/home/john/.local/share -o canmount=off ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local/share; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local/share/Steam; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john/config; \ zfs create -o mountpoint=legacy ${SYS_ROOT}/${SYSTEM_NAME}/home/john/cache ``` As of [zfsonlinux 0.7.0](https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.7.0) ZFS delegation using `zfs allow` works on linux. I delegate all datasets under `${SYS_ROOT}/${SYSTEM_NAME}/home/john` to my user 'john' giving the abiity to snapshot and create datasets. ```shell zfs allow john create,mount,mountpoint,snapshot ${SYS_ROOT}/${SYSTEM_NAME}/home/john ``` Checking permissions shows john's permissions. ```shell zfs allow ${SYS_ROOT}/${SYSTEM_NAME}/home/john ``` ```shell ---- Permissions on vault/sys/chin/home/john ------------------------- Local+Descendent permissions: user john create [root@chin ~]# zfs allow john snapshot ${SYS_ROOT}/${SYSTEM_NAME}/home/john [root@chin ~]# zfs allow ${SYS_ROOT}/${SYSTEM_NAME}/home/john ---- Permissions on vault/sys/chin/home/john ------------------------- Local+Descendent permissions: user john create,snapshot ``` Available options: ```shell NAME TYPE NOTES allow subcommand Must also have the permission that is being allowed clone subcommand Must also have the 'create' ability and 'mount' ability in the origin file system create subcommand Must also have the 'mount' ability destroy subcommand Must also have the 'mount' ability hold subcommand Allows adding a user hold to a snapshot mount subcommand Allows mount/umount of ZFS datasets promote subcommand Must also have the 'mount' and 'promote' ability in the origin file system receive subcommand Must also have the 'mount' and 'create' ability release subcommand Allows releasing a user hold which might destroy the snapshot rename subcommand Must also have the 'mount' and 'create' ability in the new parent rollback subcommand send subcommand share subcommand Allows sharing file systems over NFS or SMB protocols snapshot subcommand groupquota other Allows accessing any groupquota@... property groupused other Allows reading any groupused@... property userprop other Allows changing any user property userquota other Allows accessing any userquota@... property userused other Allows reading any userused@... property aclinherit property aclmode property atime property canmount property casesensitivity property checksum property compression property copies property dedup property devices property exec property logbias property mlslabel property mountpoint property nbmand property normalization property primarycache property quota property readonly property recordsize property refquota property refreservation property reservation property secondarycache property setuid property shareiscsi property sharenfs property sharesmb property snapdir property utf8only property version property volblocksize property volsize property vscan property xattr property zoned property ``` **Data Datasets** I'll be mounting these under `${HOME}`. They exist outside the different systems and are shared between them. ```shell zfs create -o mountpoint=none ${DATA_ROOT}; \ zfs create -o mountpoint=legacy ${DATA_ROOT}/Books; \ zfs create -o mountpoint=legacy ${DATA_ROOT}/Computer; \ zfs create -o mountpoint=legacy ${DATA_ROOT}/Personal; \ zfs create -o mountpoint=legacy ${DATA_ROOT}/Pictures; \ zfs create -o mountpoint=legacy ${DATA_ROOT}/University; \ zfs create -o mountpoint=legacy ${DATA_ROOT}/Workspace; \ zfs create -o mountpoint=legacy ${DATA_ROOT}/Reference ``` ### Final Structure So my system ends up as. ```shell zfs list -o name | grep -E 'chin|data' vault/data 768K 860G 96K none vault/data/Books 96K 860G 96K legacy vault/data/Computer 96K 860G 96K legacy vault/data/Personal 96K 860G 96K legacy vault/data/Pictures 96K 860G 96K legacy vault/data/Reference 96K 860G 96K legacy vault/data/University 96K 860G 96K legacy vault/data/Workspace 96K 860G 96K legacy vault/sys/chin 1.97M 860G 96K none vault/sys/chin/ROOT 192K 860G 96K none vault/sys/chin/ROOT/default 96K 860G 96K / vault/sys/chin/home 672K 860G 96K legacy vault/sys/chin/home/john 576K 860G 96K legacy vault/sys/chin/home/john/cache 96K 860G 96K legacy vault/sys/chin/home/john/config 96K 860G 96K legacy vault/sys/chin/home/john/local 288K 860G 96K legacy vault/sys/chin/home/john/local/share 192K 860G 96K /home/john/.local/share vault/sys/chin/home/john/local/share/Steam 96K 860G 96K legacy vault/sys/chin/usr 192K 860G 96K /usr vault/sys/chin/usr/local 96K 860G 96K legacy vault/sys/chin/var 864K 860G 96K /var vault/sys/chin/var/cache 96K 860G 96K legacy vault/sys/chin/var/lib 576K 860G 96K /var/lib vault/sys/chin/var/lib/lxc 96K 860G 96K legacy vault/sys/chin/var/lib/lxd 96K 860G 96K legacy vault/sys/chin/var/lib/machines 96K 860G 96K legacy vault/sys/chin/var/lib/libvirt 96K 860G 96K legacy vault/sys/chin/var/lib/systemd 192K 860G 96K /var/lib/systemd vault/sys/chin/var/lib/systemd/coredump 96K 860G 96K legacy vault/sys/chin/var/log 96K 860G 96K legacy ``` ### Install Preperation Using this structure datasets must be mounted in the correct order. #### ZFS Setup Import zpool and mount root dataset: ```shell zpool import -d /dev/disk/by-id -R /mnt vault mount -t zfs vault/sys/chin/ROOT/default /mnt ``` After dataset creation, create cachefile. ```shell zpool set cachefile=/etc/zfs/zpool.cache vault mkdir -p /mnt/etc/zfs && cp /etc/zfs/zpool.cache /mnt/etc/zfs/zpool.cache ``` Mount system datasets: ```shell mkdir -p /mnt/usr/local mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/usr/local /mnt/usr/local; \ mkdir -p /mnt/var/cache mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/cache /mnt/var/cache; \ mkdir -p /mnt/var/lib/{lxc,lxd,machines,libvirt,systemd/coredump} /mnt/var/log; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/lxc /mnt/var/lib/lxc; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/lxd /mnt/var/lib/lxd; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/machines /mnt/var/lib/machines; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/libvirt /mnt/var/lib/libvirt; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/lib/systemd/coredump /mnt/var/lib/systemd/coredump; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/log /mnt/var/log; \ mkdir /mnt/var/log/journal; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/var/log/journal /mnt/var/log/journal ``` Mount home. ```shell mkdir -p /mnt/home ; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home /mnt/home; \ mkdir -p /mnt/home/john; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john /mnt/home/john; \ mkdir -p /mnt/home/john/{.cache,.config,.local}; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john/cache /mnt/home/john/.cache; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john/config /mnt/home/john/.config; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local /mnt/home/john/.local; \ mkdir -p /mnt/home/john/.local/share/Steam; \ mount -t zfs ${SYS_ROOT}/${SYSTEM_NAME}/home/john/local/share/Steam /mnt/home/john/.local/share/Steam ``` Mount data: ```shell mkdir -p /mnt/home/john/{Books,Computer,Personal,Pictures,Reference,University,Workspace}; \ mount -t zfs vault/data/Books /mnt/home/john/Books; \ mount -t zfs vault/data/Computer /mnt/home/john/Computer; \ mount -t zfs vault/data/Personal /mnt/home/john/Personal; \ mount -t zfs vault/data/Pictures /mnt/home/john/Pictures; \ mount -t zfs vault/data/Reference /mnt/home/john/Reference; \ mount -t zfs vault/data/University /mnt/home/john/University; \ mount -t zfs vault/data/Workspace /mnt/home/john/Workspace ``` #### Boot Setup Create esp, (EF00) for regular install. ```shell gdisk /dev/sdf mkfs.fat -F32 /dev/sdf1 mount /dev/sdf1 /mnt/boot ``` I keep it at `/mnt/efi` instead, and [bindmount kernel directory to /boot](https://ramsdenj.com/2016/04/15/multi-boot-linux-with-one-boot-partition.html). ```shell mkdir -p /mnt/mnt/efi mount /dev/sdf1 /mnt/mnt/efi ``` ```shell mkdir -p /mnt/boot /mnt/mnt/efi/installs/chin mount --bind /mnt/mnt/efi/installs/chin /mnt/boot ``` #### Swap Create 32GiB partition and create swap. ```shell mkswap /dev/sdf2 swapon /dev/sdf2 ``` #### fstab Configuration Create fstab, adding all currently mounted filesystems. ```shell genfstab -U -p /mnt >> /mnt/etc/fstab ``` Get swap UUID and add to fstab. ```shell lsblk -no UUID /dev/sdf2 ``` ```shell UUID=4b00ce42-d400-4060-9329-622c420f367e none swap defaults 0 0 ``` Now all partitions and datasets should be setup, check that the fstab looks correct. # Raspberry PI Secure VPN Torrentbox ## Setup * Install [Arch Linux Arm](https://archlinuxarm.org/) * Install [base-devel](https://www.archlinux.org/packages/?sort=\&q=base-devel) I used the AUR manager [aurutils](https://aur.archlinux.org/packages/aurutils/)AUR to download and setup any AUR packages. ```shell mkdir -p ~/Downloads && cd ~/Downloads gpg --recv-key 6BC26A17B9B7018A && gpg --lsign 6BC26A17B9B7018A git clone https://aur.archlinux.org/aurutils.git cd aurutils && makepkg -si && cd .. && rm -rf aurutils ``` If used, setup repo for [aurutils](https://docs.ramsdenj.com/operatingsystems/linux/distributions/archlinux/packagemanagement/aurutils.html). ### OpenVPN ipv6 should be [disabled](https://wiki.archlinux.org/index.php/IPv6#Disable_IPv6) since PIA [doesn't support it](https://helpdesk.privateinternetaccess.com/hc/en-us/articles/232324908-Why-Do-You-Block-IPv6-). Add `ipv6.disable=1` to `/boot/cmdline.txt` and reboot. To check if it's disabled, see if you get an ipv6 address with `ip addr`. If disabled, `inet6` will not be present. Install [openvpn](https://www.archlinux.org/packages/?name=openvpn). ```shell pacman -S openvpn ``` In order to install openvpn, required scripts need to be [downloaded and renamed](https://wiki.archlinux.org/index.php/Private_Internet_Access#Manual), ```shell mkdir -p ~/Downloads/openvpn/certs && cd ~/Downloads/openvpn/certs curl http://www.privateinternetaccess.com/openvpn/openvpn-strong.zip --location --remote-name --remote-header-name unzip openvpn-strong.zip mv openvpn-strong.zip ~/Downloads/openvpn ``` * `-L`, `--location` * Follow re-direct if the server reports that the requested page has moved to a different location * `-O`, `--remote-name` * Write output to a local file named like the remote file we get. * `-J`, `--remote-header-name` * This option tells the -O, --remote-name option to use the server-specified filename. Replace all `.ovpn` extensions on the files downloaded with `.conf` and remove spaces in names. To view the renames first, run. ```shell for f in *.ovpn; do echo "${f} -->" " " "$(echo ${f} | sed -e 's/ //g' -e 's/.ovpn/.conf/')"; done ``` If you're happy do the rename. ```shell for f in *.ovpn; do mv "${f}" "$(echo ${f} | sed -e 's/ //g' -e 's/.ovpn/.conf/')"; done ``` Move the files to `/etc/openvpn/client`, which is where OpenVPN expects them to be. Make sure they're owned by `root`. ```shell install -D --owner=root --group=root ./* /etc/openvpn/client ``` ```shell ls -la /etc/openvpn/client total 188 drwxr-x--- 2 root network 4096 Feb 26 06:58 . drwxr-xr-x 4 root root 4096 Feb 26 06:46 .. -rwxr-xr-x 1 root root 297 Feb 26 06:58 AUMelbourne.conf -rwxr-xr-x 1 root root 291 Feb 26 06:58 Austria.conf -rwxr-xr-x 1 root root 287 Feb 26 06:58 AUSydney.conf -rwxr-xr-x 1 root root 291 Feb 26 06:58 Belgium.conf -rwxr-xr-x 1 root root 290 Feb 26 06:58 Brazil.conf -rwxr-xr-x 1 root root 286 Feb 26 06:58 CAMontreal.conf -rwxr-xr-x 1 root root 2719 Feb 26 06:58 ca.rsa.4096.crt -rwxr-xr-x 1 root root 294 Feb 26 06:58 CAToronto.conf -rwxr-xr-x 1 root root 296 Feb 26 06:58 CAVancouver.conf -rwxr-xr-x 1 root root 1214 Feb 26 06:58 crl.rsa.4096.pem -rwxr-xr-x 1 root root 289 Feb 26 06:58 ... ``` #### Configuration The above configs can be used as is, or a custom one can be used. They contain the following. For `/etc/openvpn/client/Netherlands.conf`: ```shell client dev tun proto udp remote nl.privateinternetaccess.com 1197 resolv-retry infinite nobind persist-key persist-tun cipher aes-256-cbc auth sha256 tls-client remote-cert-tls server auth-user-pass comp-lzo verb 1 reneg-sec 0 crl-verify crl.rsa.4096.pem ca ca.rsa.4096.crt disable-occ ``` Copy the config to a new file. ```shell [root]# cp /etc/openvpn/client/Netherlands.conf /etc/openvpn/client/custompivpn.conf ``` Edit the file, replace the server `remote nl.privateinternetaccess.com 1197` with the PIA servers you want to use. The servers are in the openvpn files. They can all be listed with a `grep` for `privateinternetaccess.com`. ```shell grep --no-filename privateinternetaccess.com /etc/openvpn/client/* remote aus-melbourne.privateinternetaccess.com 1197 remote austria.privateinternetaccess.com 1197 remote aus.privateinternetaccess.com 1197 remote belgium.privateinternetaccess.com 1197 remote brazil.privateinternetaccess.com 1197 remote ca.privateinternetaccess.com 1197 remote ca-toronto.privateinternetaccess.com 1197 remote ca-vancouver.privateinternetaccess.com 1197 remote czech.privateinternetaccess.com 1197 remote denmark.privateinternetaccess.com 1197 remote fi.privateinternetaccess.com 1197 remote france.privateinternetaccess.com 1197 remote germany.privateinternetaccess.com 1197 remote hk.privateinternetaccess.com 1197 remote in.privateinternetaccess.com 1197 remote ireland.privateinternetaccess.com 1197 remote israel.privateinternetaccess.com 1197 remote italy.privateinternetaccess.com 1197 remote japan.privateinternetaccess.com 1197 remote mexico.privateinternetaccess.com 1197 remote nl.privateinternetaccess.com 1197 remote nz.privateinternetaccess.com 1197 remote no.privateinternetaccess.com 1197 remote ro.privateinternetaccess.com 1197 remote sg.privateinternetaccess.com 1197 remote spain.privateinternetaccess.com 1197 remote sweden.privateinternetaccess.com 1197 remote swiss.privateinternetaccess.com 1197 remote turkey.privateinternetaccess.com 1197 remote uk-london.privateinternetaccess.com 1197 remote uk-manchester.privateinternetaccess.com 1197 remote uk-southampton.privateinternetaccess.com 1197 remote us-atlanta.privateinternetaccess.com 1197 remote us-california.privateinternetaccess.com 1197 remote us-chicago.privateinternetaccess.com 1197 remote us-east.privateinternetaccess.com 1197 remote us-florida.privateinternetaccess.com 1197 remote us-midwest.privateinternetaccess.com 1197 remote us-newyorkcity.privateinternetaccess.com 1197 remote us-seattle.privateinternetaccess.com 1197 remote us-siliconvalley.privateinternetaccess.com 1197 remote us-texas.privateinternetaccess.com 1197 remote us-west.privateinternetaccess.com 1197 ``` To use a random server from a list, `remote-random` can be used. Replace the single server in `/etc/openvpn/client/custompivpn.conf` with the list of servers you would like to use. After the list add `remote-random`. To auto-login to the vpn with your PIA user path, add your user and password to a file. Add the file path to the config after `auth-user-pass`, with the username on line one, and password on line two. I created `/etc/openvpn/pia_auth` ```shell touch /etc/openvpn/pia_auth chown root:root /etc/openvpn/pia_auth && chmod 660 /etc/openvpn/pia_auth ``` So as of now my config consists of the following: ```shell client dev tun proto udp remote nl.privateinternetaccess.com 1197 remote ca.privateinternetaccess.com 1197 remote ca-toronto.privateinternetaccess.com 1197 remote ca-vancouver.privateinternetaccess.com 1197 remote sweden.privateinternetaccess.com 1197 remote-random resolv-retry infinite nobind persist-key persist-tun cipher aes-256-cbc auth sha256 tls-client remote-cert-tls server auth-user-pass /etc/openvpn/pia_auth comp-lzo verb 1 reneg-sec 0 crl-verify crl.rsa.4096.pem ca ca.rsa.4096.crt disable-occ ``` If you're connected over SSH to your pi, connection to the pi will drop if openvpn is started. This is because the default gateway changes. To make local connections continue to be routed over the same interface that SSH was started on, add a new table using the `ip` command. ```shell ip rule add table 128 from ip route add table 128 to /24 dev ip route add table 128 default via ``` For me this look like the following since the IP address of my pi was `172.20.30.4`, and my interface was `eth0`. ```shell ip rule add table 128 from 172.20.30.4 ip route add table 128 to 172.20.30.0/24 dev eth0 ip route add table 128 default via 172.20.30.1 ``` I added these as an `ExecStartPre` to \`systemd-networkd. ```shell systemctl edit systemd-networkd ``` ```shell [Service] ExecStartPre=-/usr/bin/ip rule add table 128 from 172.20.30.4 ExecStartPre=-/usr/bin/ip route add table 128 to 172.20.30.0/24 dev eth0 ExecStartPre=-/usr/bin/ip route add table 128 default via 172.20.30.1 ``` Now open VPN can be started. A systemd unit exists that lets any client configurations be started from the directory where we put our configuration, so long as they end in `.conf`. If everything is setup correctly we should be able to start our VPN connection with `systemctl start openvpn-client@custompivpn`. I was concerned about losing connection and not being able to get back into my pi, so the first time I started the service in `tmux` with a five minute kill timer so that if I wasn't able to reconnect I knew that after 5 minutes the service would be stopped and I would be able to get back in. ```shell systemctl start openvpn-client@custompivpn; \ sleep 5m; \ systemctl stop openvpn-client@custompivpn ``` Fortunately everything was setup correctly so my connection wasn't dropped. To make sure that the VPN is working correctly, and that your IP is changing, check your IP address before and after the VPN is started with `curl -s checkip.dyndns.org`. After starting the VPN I got a swedish IP address meaning the VPN was working. ### Create User I'll be using the `media` user and group for everything torrent related. Create it. ```shell groupadd --gid 8675309 media useradd --system --shell /usr/bin/nologin --gid 8675309 --uid 8675309 media ``` ### Network Shares If mounting [NFS shares](https://wiki.archlinux.org/index.php/NFS#Installation) install the [nfs-utils](https://www.archlinux.org/packages/?name=nfs-utils) package. #### NFS Configuration ```shell pacman -S nfs-utils ``` Enable NFSv4 idmapping ```shell echo N > /sys/module/nfs/parameters/nfs4_disable_idmapping ``` Set permanent in `/etc/modprobe.d/nfsd.conf`. ```shell options nfsd nfs4_disable_idmapping=0 ``` [Optionally](https://wiki.archlinux.org/index.php/NFS#Client) start `nfs-client.target`. ### Mount Shares Add mounts to `/etc/fstab`. ```shell mkdir -p /media/Downloads/{Complete,Incomplete} /media/Torrents chown -R media:media /media/* mount lilan.ramsden.network:/mnt/tank/media/Downloads/Complete /media/Downloads/Complete mount lilan.ramsden.network:/mnt/tank/media/Downloads/Incomplete /media/Downloads/Incomplete mount lilan.ramsden.network:/mnt/tank/media/Torrents /media/Torrents ``` Generate fstab entries and copy paste nfs mounts into fstab. ```shell genfstab -U / ``` ### DNS Using `systemd-resolvd`, DNS can by dynamically updated when OpenVPN starts using the [update-systemd-resolved](https://github.com/jonathanio/update-systemd-resolved) script. Install from github or install the [openvpn-update-systemd-resolved](https://aur.archlinux.org/packages/openvpn-update-systemd-resolved/)AUR package. You can then add the following into your OpenVPN configuration file: ```shell script-security 2 setenv PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin up /etc/openvpn/scripts/update-systemd-resolved down /etc/openvpn/scripts/update-systemd-resolved down-pre ``` It will then follow `dhcp-option` commands set in OpenVPN. We can use PIA's DNS servers this way: ```shell dhcp-option DNS 209.222.18.222 dhcp-option DNS 209.222.18.218 ``` Now, after starting the OpenVPN you should see the following new lines in `/etc/resolv.conf`. ```shell nameserver 209.222.18.222 nameserver 209.222.18.218 ``` Start and anable OpenVPN. ```shell systemctl enable --now openvpn-client@custompivpn ``` My final config was the following. ```shell client dev tun proto udp remote nl.privateinternetaccess.com 1197 remote ca.privateinternetaccess.com 1197 remote ca-toronto.privateinternetaccess.com 1197 remote ca-vancouver.privateinternetaccess.com 1197 remote sweden.privateinternetaccess.com 1197 remote-random resolv-retry infinite nobind persist-key persist-tun cipher aes-256-cbc auth sha256 tls-client remote-cert-tls server auth-user-pass /etc/openvpn/pia_auth comp-lzo verb 1 reneg-sec 0 crl-verify crl.rsa.4096.pem ca ca.rsa.4096.crt disable-occ script-security 2 setenv PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin up /etc/openvpn/scripts/update-systemd-resolved down /etc/openvpn/scripts/update-systemd-resolved down-pre ``` ### iptables killswitch Enable ip forwarding, add the `net.ipv4.ip_forward=1` sysctl. ```shell echo 'net.ipv4.ip_forward=1' | tee '/etc/sysctl.d/90-openvpn-networking.conf' ``` Reload sysctls. ```shell sysctl --system ``` Create an [iptables](https://wiki.archlinux.org/index.php/Iptables) rules file in `/etc/iptables/iptables.rules`. Start with a filter table in the [iptables-restore](http://www.iptables.info/en/iptables-save-restore-rules.html) syntax. ```shell *filter ``` Drop all traffic by default. ```shell --policy INPUT DROP --policy FORWARD DROP --policy OUTPUT DROP ``` Start with input rules. Only allow established connections and SSH from LAN, (use your LAN subnet). ```shell --append INPUT --match conntrack --ctstate RELATED,ESTABLISHED --jump ACCEPT --append INPUT --protocol tcp --dport 22 --source 172.20.0.0/16 --jump ACCEPT --append INPUT --protocol tcp --dport 22 --source 127.0.0.0/8 --jump ACCEPT --append INPUT --protocol tcp --dport 22 --jump DROP ``` Open ports deluge needs. ```shell --append INPUT --protocol tcp --dport 56881:56889 --jump ACCEPT --append INPUT --protocol udp --dport 56881:56889 --jump ACCEPT ``` For remote access: ```shell --append INPUT --protocol tcp --dport 58846 --jump ACCEPT ``` Now output rules. Allow the loopback interface and ping. ```shell --append OUTPUT --out-interface lo --jump ACCEPT --append OUTPUT --out-interface tun0 --protocol icmp --jump ACCEPT ``` Allow LAN traffic (use your lan subnet). ```shell --append OUTPUT --destination 172.20.30.0/24 --jump ACCEPT ``` Allow PIA DNS servers. ```shell --append OUTPUT --destination 209.222.18.222 --jump ACCEPT --append OUTPUT --destination 209.222.18.218 --jump ACCEPT ``` Optionally allow your own DNS server. ```shell --append OUTPUT --destination 172.20.30.1 --jump ACCEPT ``` Allow the VPN port and the interface. ```shell --append OUTPUT --protocol udp --match udp --dport 1197 --jump ACCEPT --append OUTPUT --out-interface tun0 --jump ACCEPT ``` Finally commit the table. ```shell COMMIT ``` My final rules looks like the following: ```shell # /etc/iptables/iptables.rules # iptables rules for OpenVPN killswitch *filter --policy INPUT DROP --policy FORWARD DROP --policy OUTPUT DROP --append INPUT --match conntrack --ctstate RELATED,ESTABLISHED --jump ACCEPT --append INPUT --protocol tcp --dport 22 --source 172.20.0.0/16 --jump ACCEPT --append INPUT --protocol tcp --dport 22 --source 127.0.0.0/8 --jump ACCEPT --append INPUT --protocol tcp --dport 22 --jump DROP --append INPUT --protocol tcp --dport 56881:56889 --jump ACCEPT --append INPUT --protocol udp --dport 56881:56889 --jump ACCEPT --append INPUT --protocol tcp --dport 58846 --jump ACCEPT --append OUTPUT --out-interface lo --jump ACCEPT --append OUTPUT --out-interface tun0 --protocol icmp --jump ACCEPT --append OUTPUT --destination 172.20.30.0/24 --jump ACCEPT --append OUTPUT --destination 209.222.18.222 --jump ACCEPT --append OUTPUT --destination 209.222.18.218 --jump ACCEPT --append OUTPUT --destination 172.20.30.1 --jump ACCEPT --append OUTPUT --protocol udp --match udp --dport 1197 --jump ACCEPT --append OUTPUT --out-interface tun0 --jump ACCEPT COMMIT ``` Save the file. Test starting the VPN and firewall. ```shell systemctl start iptables openvpn-client@custompivpn; \ sleep 5m; \ systemctl stop iptables openvpn-client@custompivpn ``` Check they started successfully. ```shell systemctl status iptables openvpn-client@custompivpn ``` Try to ping google. ```shell ping google.com ``` Stop OpenVPN, and try again. Your connection should be blocked. ```shell ping google.com PING google.com (216.58.216.174) 56(84) bytes of data. ping: sendmsg: Operation not permitted ``` Start and enable the iptables service. ```shell systemctl enable --now iptables ``` You may also want to set up a regular firewall to block unwanted incoming traffic. The arch Wiki has a good reference for a [simple stateful firewall](https://wiki.archlinux.org/index.php/Simple_stateful_firewall) ### Deluge Now to setup the [deluge](https://wiki.archlinux.org/index.php/Deluge) service. Install [deluge](https://www.archlinux.org/packages/?sort=\&q=deluge) Start and enable the system service, which runs as deluge. ```shell systemctl enable --now deluged ``` To connect remotely, [create a user](https://wiki.archlinux.org/index.php/Deluge#Create_a_user) in `~deluge/.config/deluge/auth` with `USER:PASSWORD:PERMISSIONS` (10 is admin). For example: ```shell john:p422WoRd:10 ``` Stop deluge and set `"allow_remote": true` in `~deluge/.config/deluge/core.conf`. If `core.conf` doesn't exist, connect to the console. ```shell sudo -u deluge deluge-console ``` Now you should be able to connect to deluge from `:`, likely port 58846, while the VPN is off. Settings: * Network * Incoming Ports: * From: 56881 * To: 56889 * Outgoing Ports * Use random ports: yes * Network Extras * Peer Exchange: yes * DHT: yes * Encryption * Inbound: Forced * Outbound: Forced * Level: Full Stream * Encrypt entire stream: yes * Proxy * Peer * Type: Socksv5 W/ Auth * Username: * Password: * Host: I use proxy-nl.privateinternetaccess.com * Port: 1080 Add deluge user to media group: ```shell gpasswd -a deluge media ``` Using the proxy, check your ip is masked using an [IP checker torrent](http://btguard.com/BTGuard_Torrent_IP_Check.torrent). More info [here](https://wiki.btguard.com/index.php/CheckMyTorrentIP) (May need to restart deluge). ## References * [How To Create A VPN Killswitch Using Iptables on Linux](https://linuxconfig.org/how-to-create-a-vpn-killswitch-using-iptables-on-linux) * [iptables - save and restore](http://www.iptables.info/en/iptables-save-restore-rules.html) # NixOS This section is about the linux distribution NixOS. A specialized Linux distribution that uses a functional package manager called Nix for configuration and management of the NixOS Linux distribution. # Remotely Accessing Install Media If disaster occurs and it's necessary to remotely access the machine, boot into NixOS install media and access the machine using SSH. ### SSH Access By default it's possible to SSH into the install media as root. In order to log in create a password. ```shell passwd ``` Start SSH Daemon. ```shell systemctl start sshd ``` Check the IP address. ```shell ip addr ``` Now it should be possible to access the machine as root. *NOTE: SSH'ing into a machine as root is a terrible practice and is only done here temporarily.* # root on ZFS Install In order to import a ZFS pool, ZFS must be enabled in the NixOS configuration file. Make sure zfs is in `boot.supportedFilesystems`. ```shell { config, pkgs, ... }: { imports = [ ]; boot.supportedFilesystems = [ "zfs" ]; } ``` Rebuild NixOS and switch to the new configuration. ```shell nixos-rebuild switch ``` Check zfs is working,`modprobe zfs` should show no problems. ZFS should now work. #### Pool Create a new pool or mount an existing pool. **Creating a pool** Create a pool using the disk ID and set it to 4k block size as default with `ashift=12`. ```shell ls -la /dev/disk/by-id/ zpool create -f -o ashift=12 vault /dev/disk/by-id/${DISKS} ``` Export the pool after creation. ```shell zpool export ${POOLNAME} ``` **Using an Existing pool** The existing pool assigning it to be relative to `/mnt` with `-R`, the `-N` flag will tell ZFS not to mount any datasets. ```shell zpool import -N -d /dev/disk/by-id -R /mnt vault ``` ### Setup Datasets Mount all datasets partitions to /mnt. #### Filesystem ```shell NIX_ROOT=/mnt ZFS_ROOT_DATASET=vault/sys/atom ZFS_DATA_DATASET=vault/data ``` Setup datasets. Set all legacy. ```shell zfs create -o mountpoint=none vault/sys zfs create -o mountpoint=none ${ZFS_ROOT_DATASET} zfs create -o mountpoint=none ${ZFS_ROOT_DATASET}/ROOT zfs create -o mountpoint=legacy ${ZFS_ROOT_DATASET}/ROOT/default # Rest of datasets... ``` ### Mount Datasets Mount the datasets: ```shell mkdir ${NIX_ROOT}/nix; mount -t zfs ${ZFS_ROOT_DATASET}/ROOT/default ${NIX_ROOT} # Rest of datasets... ``` **Boot** Create a 512M esp, mount to /boot ```shell gdisk /dev/sdf Command (? for help): n Partition number (5-128, default 5): First sector (34-488397134, default = 225445888) or {+-}size{KMGTP}: Last sector (225445888-488397134, default = 488397134) or {+-}size{KMGTP}: +512 Hex code or GUID (L to show codes, Enter = 8300): ef00 Changed type of partition to 'EFI System' ``` Format boot and mount. ```shell mkfs.fat -F32 /dev/sdf1 mkdir ${NIX_ROOT}/boot mount /dev/sdf1 ${NIX_ROOT}/boot ``` **Swap** Create a partition of desired size. ```shell gdisk /dev/sdf Command (? for help): n Partition number (2-128, default 2): First sector (34-488397134, default = 2099200) or {+-}size{KMGTP}: Last sector (2099200-488397134, default = 488397134) or {+-}size{KMGTP}: +32G Current type is 'Linux filesystem' Hex code or GUID (L to show codes, Enter = 8300): 8200 Changed type of partition to 'Linux swap' ``` Enable swap. ```shell mkswap /dev/sdf2 swapon /dev/sdf2 ``` **Install** Setup config in `/mnt/etc/nixos` and install. ``` nixos-install --root /mnt ``` # systemd This section describes ways to use the systemd and it system and related software. # Network Bonding Going from a one interface setup, to two bonded: Before: ```shell nano /etc/systemd/network/25-wired.network ``` ```shell [Match] Name=eno1 [Network] Address=172.20.20.2/24 Gateway=172.20.20.1 ``` Create the [netdev](https://www.freedesktop.org/software/systemd/man/systemd.netdev.html) bond file `/etc/systemd/network/25-bond1.netdev`. ```shell nano /etc/systemd/network/25-bond1.netdev ``` ```shell [NetDev] Name=bond1 Kind=bond #default is "balance-rr" (round robin) [Bond] #Mode="balance-rr ``` Create network for bond. ```shell nano /etc/systemd/network/25-bond1.network ``` ```shell [Match] Name=bond1 [Network] Address=172.20.20.2/24 Gateway=172.20.20.1 ``` Select interfaces. ```shell nano /etc/systemd/network/20-eno1.network ``` ```shell [Match] Name=eno1 [Network] Bond=bond1 ``` ```shell nano /etc/systemd/network/25-enp5s0.network ``` ```shell [Match] Name=enp5s0 [Network] Bond=bond1 ``` Restart network: ```shell systemctl restart systemd-resolved systemd-networkd ``` Check if functional: ```shell networkctl ``` Before: ```shell IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 eno1 ether routable configured 3 enp5s0 ether off unmanaged 4 virbr0 ether no-carrier unmanaged 5 virbr0-nic ether off unmanaged 5 links listed. ``` After: ```shell IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 bond0 ether off unmanaged 3 bond1 ether routable configured 4 eno1 ether carrier configuring 5 enp5s0 ether no-carrier configuring 6 virbr0 ether no-carrier unmanaged 7 virbr0-nic ether off unmanaged 7 links listed. ``` **Note**: `systemd` automatically creates bond0, it can be ignored. Status: ```shell cat /proc/net/bonding/bond1 ``` ```shell cat /proc/net/bonding/bond1 john@chin Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: load balancing (round-robin) MII Status: up MII Polling Interval (ms): 0 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eno1 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 74:d0:2b:7d:2b:eb Slave queue ID: 0 Slave Interface: enp5s0 MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 00:1b:21:63:1f:4d Slave queue ID: 0 ``` **Note**: DNS using `systemd-resolved` config in `/etc/systemd/resolved.conf` ### References # Tuning The following sections include ways to tune Linux. # CPU Tuning ## intel\_pstate To disable turbo, set a udev rule to set `cpu/intel_pstate/no_turbo` to `1`: ```shell nano /etc/udev/rules.d/50-set_intel_pstate_no_turbo.rules ``` ```shell KERNEL=="cpu",RUN+="/bin/sh -c 'echo -n 1 > /sys/devices/system/cpu/intel_pstate/no_turbo'" ``` # Limits Set user defined limits by adding overrides to `/etc/systemd/system.conf` and `/etc/systemd/user.conf` ## Games Nofile may need increasing. Create an override and set it. For user: ```shell nano /etc/systemd/user.conf.d/nofile.conf ``` ```shell [Manager] DefaultLimitNOFILE=8192 ``` For system: ```shell cp /etc/systemd/user.conf.d/nofile.conf /etc/systemd/system.conf.d/nofile.conf ``` # Sysctls ### Inotify: Increase inotify max user watches: ```shell echo "fs.inotify.max_user_watches=524288" >> /etc/sysctl.d/40-max-user-watches.conf ``` ### Network Increase [netdev budget](https://access.redhat.com/sites/default/files/attachments/20150325_network_performance_tuning.pdf) for squeezed packets, and netdev\_max\_backlog. Add other [optimizations](https://wiki.archlinux.org/index.php/Sysctl#Networking) for better performance. ```shell [root]# nano /etc/sysctl.d/30-network-tuning.conf ``` ```shell # The maximum size of the receive queue. # The received frames will be stored in this queue after taking them from the ring buffer on the NIC. # Use high value for high speed cards to prevent loosing packets. net.core.netdev_max_backlog = 100000 net.core.netdev_budget=50000 # The upper limit on the value of the backlog parameter passed to the listen function. # Setting to higher values is only needed on a single highloaded server where new connection rate is high/bursty net.core.somaxconn = 16384 # The default and maximum amount for the receive/send socket memory # By default the Linux network stack is not configured for high speed large file transfer across WAN links. # This is done to save memory resources. # You can easily tune Linux network stack by increasing network buffers size for high-speed networks that connect server systems to handle more network packets. net.core.rmem_default = 1048576 net.core.wmem_default = 1048576 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.udp_rmem_min = 16384 net.ipv4.udp_wmem_min = 16384 # The maximum queue length of pending connections 'Waiting Acknowledgment' # In the event of a synflood DOS attack, this queue can fill up pretty quickly, at which point tcp_syncookies will kick in allowing your system to continue to respond to legitimate traffic, and allowing you to gain access to block malicious IPs. # If the server suffers from overloads at peak times, you may want to increase this value a little bit. net.ipv4.tcp_max_syn_backlog = 65536 ``` ### Reload Sysctls ```shell sysctl --system ``` ## References: # Network Reliability With iwlwifi My network keep dropping out using the iwlwifi driver, adding the following kernel parameters fixed the problem. To get rid of "*iwlwifi 0000:03:00.0: DMA: Out of SW-IOMMU space for 4096 bytes*" errors I increased the buffer size. ```shell swiotlb=32768 ``` I was getting the following PCI error. ```shell kernel: pcieport 0000:00:1b.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, id=00d8(Receiver ID) kernel: pcieport 0000:00:1b.0: device [8086:a167] error status/mask=00000001/00002000 kernel: pcieport 0000:00:1b.0: [ 0] Receiver Error (First) ``` Turning off active power management got rid of the error. ```shell pcie_aspm=off ``` # Surface Pro 4 Power Tuning ## Audio [Idle audio card](https://wiki.archlinux.org/index.php/Power_management#Audio) after one second: ```shell echo "options snd_hda_intel power_save=1" > /etc/modprobe.d/audio_powersave.conf ``` ## Kernel Tuning [Disable NMI watchdog](https://wiki.archlinux.org/index.php/Power_management#Disabling_NMI_watchdog). It can generate a lot of interrupts, causing a noticeable increase in power usage. ```shell echo "kernel.nmi_watchdog = 0" > /etc/sysctl.d/disable_watchdog.conf ``` ## PCI Enable [PCI Runtime Power Management](https://wiki.archlinux.org/index.php/Power_management#PCI_Runtime_Power_Management) ```shell echo "ACTION=="add", SUBSYSTEM=="pci", ATTR{power/control}="auto"" > /etc/udev/rules.d/pci_pm.rules ``` ## References * [Arch Wiki](https://wiki.archlinux.org) * [Powertop](https://wiki.archlinux.org/index.php/Powertop) * [Power Management](https://wiki.archlinux.org/index.php/Power_management#Power_management_with_systemd) # ZFS Arc Max on Linux Check stats with `arcstat.py` ```shell # arcstat.py -h Usage: arcstat.py [-hvx] [-f fields] [-o file] [-s string] [interval [count]] -h : Print this help message -v : List all possible field headers and definitions -x : Print extended stats -f : Specify specific fields to print (see -v) -o : Redirect output to the specified file -s : Override default field separator with custom character or string Examples: arcstat.py -o /tmp/a.log 2 10 arcstat.py -s "," -o /tmp/a.log 2 10 arcstat.py -v arcstat.py -f time,hit%,dh%,ph%,mh% 1 ``` Set arc max in `/etc/modprobe.d/zfs.conf`, defaults to 50% memory. For example, 48GiB: ```shell echo "options zfs zfs_arc_max=51539607552" > /etc/modprobe.d/zfs.conf ``` Rebuild kernel, then reboot. ```shell mkinitcpio -p linux ``` # TrueNAS TrueNAS tuning # Setup ## Mail Generate app password in email. # BSD Sections related to the various BSD operating systems. # FreeBSD The following sections are related to the FreeBSD operating system. # iocage I've been using the new python rewrite of the jail manager \[iocage],() for FreeBSD, and since it's still in rapid development there have been a lot of new changes and bug fixes. A [few of which](https://github.com/iocage/iocage/issues?utf8=%E2%9C%93\&q=is%3Aissue%20author%3Ajohnramsden) have been my own. Since releases aren't put out with every update I decided it would be a good idea to run from [the master branch](https://github.com/iocage/iocage). Since I didn't feel like polluting my system with user install python packages that weren't being managed by the package manager, I thought it would be a good idea to install them into a [virtual environment](https://docs.python.org/3/tutorial/venv.html). It seems to have worked well so far. ### Install From git in a virtualenv Create `/usr/local/opt/iocage`, install requirements and download source. ```shell pkg update && pkg upgrade && pkg install python36 libgit2 git # Or git-lite mkdir -p /usr/local/opt/iocage git clone --recursive https://github.com/iocage/iocage ``` Enter source directory, create a venv for the install.. ```shell cd iocage python3.6 -m venv venv ``` Enter the venv and install. ```shell source venv/bin/activate make install deactivate ``` Symlink the script to `/usr/local/bin/`. ```shell ln -s /usr/local/opt/iocage/iocage/venv/bin/iocage /usr/local/bin/iocage ``` And test: ```shell iocage --version ``` ```shell Version 0.9.9.2 RC ``` # Poudriere in a bhyve VM Setup iohyve: ```shell iohyve setup pool=tank iohyve setup net=igb1 iohyve setup kmod=1 ``` Fetch ISO: ```shell iohyve fetchiso ftp://ftp.freebsd.org/pub/FreeBSD/releases/ISO-IMAGES/11.0/FreeBSD-11.0-RELEASE-amd64-bootonly.iso iohyve deleteiso FreeBSD-11.0-RELEASE-amd64-bootonly.iso ``` Create guest with 20GiB HDD. ```shell iohyve create poudriere 20G iohyve set poudriere ram=8G cpu=4 ``` Install FreeBSD 11: ```shell iohyve install poudriere FreeBSD-11.0-RELEASE-amd64-bootonly.iso ``` Attach to console ```shell iohyve console poudriere ``` Exit and stop the installer when finished ```shell iohyve stop poudriere ``` Start the machine. ```shell iohyve start poudriere ``` ### In VM Update ```shell pkg update && pkg upgrade freebsd-update fetch install ``` #### Poudriere Install poudriere ```shell pkg install poudriere ``` Copy the config. ```shell cp /usr/local/etc/poudriere.conf.sample /usr/local/etc/poudriere.conf ``` #### Certs Setup SSL to sign ports: ```shell mkdir -p /usr/local/etc/ssl/{keys,certs} chmod 0600 /usr/local/etc/ssl/keys openssl genrsa -out /usr/local/etc/ssl/keys/poudriere.key 4096 openssl rsa -in /usr/local/etc/ssl/keys/poudriere.key -pubout -out /usr/local/etc/ssl/certs/poudriere.cert ``` ### NFS Start NFS ```shell sysrc nfs_client_enable=YES && service nfsclient start ``` Mount packages ```shell mount :/mnt/tank/data/poudriere/packages /usr/local/poudriere/data/packages ``` Add to fstab: ```shell :/mnt/tank/data/poudriere/packages /usr/local/poudriere/data/packages nfs rw 0 0 ``` Add locking: ```shell sysrc rpc_lockd_enable=YES && sysrc rpc_statd_enable=YES service lockd start && service statd start ``` ### Configuration Edit /usr/local/etc/poudriere.conf #### Configuration These were the settings I had uncommented: ```shell # Poudriere can optionally use ZFS for its ports/jail storage. For # ZFS define ZPOOL, otherwise set NO_ZFS=yes # #### ZFS # The pool where poudriere will create all the filesystems it needs # poudriere will use tank/${ZROOTFS} as its root # # You need at least 7GB of free space in this pool to have a working # poudriere. # ZPOOL=zroot # the host where to download sets for the jails setup # You can specify here a host or an IP # replace _PROTO_ by http or ftp # replace _CHANGE_THIS_ by the hostname of the mirrors where you want to fetch # by default: ftp://ftp.freebsd.org # # Also note that every protocols supported by fetch(1) are supported here, even # file:/// # Suggested: https://download.FreeBSD.org FREEBSD_HOST=https://download.FreeBSD.org # By default the jails have no /etc/resolv.conf, you will need to set # RESOLV_CONF to a file on your hosts system that will be copied has # /etc/resolv.conf for the jail, except if you don't need it (using an http # proxy for example) RESOLV_CONF=/etc/resolv.conf # The directory where poudriere will store jails and ports BASEFS=/usr/local/poudriere # Use portlint to check ports sanity USE_PORTLINT=no # Use tmpfs(5) # This can be a space-separated list of options: # wrkdir - Use tmpfs(5) for port building WRKDIRPREFIX # data - Use tmpfs(5) for poudriere cache/temp build data # localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing) # all - Run the entire build in memory, including builder jails. # yes - Only enables tmpfs(5) for wrkdir # no - Disable use of tmpfs(5) # EXAMPLE: USE_TMPFS="wrkdir data" USE_TMPFS=yes # How much memory to limit tmpfs size to for *each builder* in GiB # (default: none) TMPFS_LIMIT=2 # If set the given directory will be used for the distfiles # This allows to share the distfiles between jails and ports tree DISTFILES_CACHE=/usr/ports/distfiles # Automatic OPTION change detection # When bulk building packages, compare the options from kept packages to # the current options to be built. If they differ, the existing package # will be deleted and the port will be rebuilt. # Valid options: yes, no, verbose # verbose will display the old and new options CHECK_CHANGED_OPTIONS=verbose # Automatic Dependency change detection # When bulk building packages, compare the dependencies from kept packages to # the current dependencies for every port. If they differ, the existing package # will be deleted and the port will be rebuilt. This helps catch changes such # as DEFAULT_RUBY_VERSION, PERL_VERSION, WITHOUT_X11 that change dependencies # for many ports. # Valid options: yes, no CHECK_CHANGED_DEPS=yes # Path to the RSA key to sign the PKGNG repo with. See pkg-repo(8) PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key # URL where your POUDRIERE_DATA/logs are hosted # This will be used for giving URL hints to the HTML output when # scheduling and starting builds URL_BASE=http:/// # When using ATOMIC_PACKAGE_REPOSITORY, commit the packages if some # packages fail to build. Ignored ports are considered successful. # This can be set to 'no' to only commit the packages once no failures # are encountered. # Default: yes COMMIT_PACKAGES_ON_FAILURE=no # Define the building jail hostname to be used when building the packages # Some port/packages hardcode the hostname of the host during build time # This is a necessary setup for reproducible builds. BUILDER_HOSTNAME= ``` ### Create jail Create a new '11.1-RELEASE' jail with the name 'freebsd-11-amd64'. ```shell poudriere jail -c -j freebsd-11-amd64 -v 11.1-RELEASE ``` Setup ports tree: ```shell poudriere ports -c -p HEAD ``` Create pkg list(s) `/usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/iocage` ```shell sysutils/py3-iocage ``` Add to make.conf : /usr/local/etc/poudriere.d/freebsd-11-amd64-make.conf use py3.6 version of python3: ```shell DEFAULT_VERSIONS+= php=7.1 python3=3.6 ``` For my jails globally: `/usr/local/etc/poudriere.d/make.conf` No docs, X11 NLS or egs: ```shell OPTIONS_UNSET+= DOCS NLS X11 EXAMPLES ``` Set options: ```shell pkg install dialog4ports ``` ```shell poudriere options -j freebsd-11-amd64 -p HEAD -f /usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/iocage -f /usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/nextcloud ``` To update jail ```shell poudriere jail -u -j freebsd-11-amd64 ``` Update tree: ```shell poudriere ports -u -p HEAD ``` Start build(s): ```shell poudriere bulk -cj freebsd-11-amd64 -p HEAD -f /usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/iocage -f /usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/nextcloud ``` ### Web Server ```shell pkg install nginx && sysrc nginx_enable=YES ``` Remove all inside server in `/usr/local/etc/nginx/nginx.conf`, add: ``` server { listen 80 default; server_name server_domain_or_IP; root /usr/local/share/poudriere/html; location /data { alias /usr/local/poudriere/data/logs/bulk; autoindex on; } location /packages { root /usr/local/poudriere/data; autoindex on; } } ``` Edit mimetypes /usr/local/etc/nginx/mime.types, add log: ```shell text/plain txt log; ``` Check config and start nginx: ```shell service nginx configtest service nginx start ``` ### Repo Only server In jail, nullfs mount packages to same spot. Install nginx. ```shell server { listen 80 default; server_name pkgrepo.ramsden.network; root /usr/local/poudriere/data/packages; autoindex on; } ``` ### Clients Get cert: ```shell cat /usr/local/etc/ssl/certs/poudriere.cert ``` Save it on clients: ```shell mkdir -p /usr/local/etc/ssl/{keys,certs} ee /usr/local/etc/ssl/certs/poudriere.cert ``` ### Repo ```shell mkdir -p /usr/local/etc/pkg/repos ``` Define repo: ```shell ee /usr/local/etc/pkg/repos/freebsd.conf ``` Inside, use the name FreeBSD in order to match the default repository definition. Disable the repository by defining it like this: ```shell FreeBSD: { enabled: no } ``` Repo file at `/usr/local/etc/pkg/repos/poudriere.conf` If you want to mix your custom packages with those of the official repositories, your file should look something like this: ```shell poudriere: { url: "http://pkgrepo.ramsden.network/freebsd-11-amd64-HEAD/", mirror_type: "http", signature_type: "pubkey", pubkey: "/usr/local/etc/ssl/certs/poudriere.cert", enabled: yes, priority: 100 } ``` If you want to only use your compiled packages, your file should look something like this: ```shell poudriere: { url: "http://pkgrepo.ramsden.network/freebsd-11-amd64-HEAD/", mirror_type: "http", signature_type: "pubkey", pubkey: "/usr/local/etc/ssl/certs/poudriere.cert", enabled: yes } ``` Update: ```shell pkg update ``` Crontab: ```shell # Update tree at 3 0 3 * * * /usr/local/bin/poudriere ports -u -p HEAD >/dev/null 2>&1 # Jails at 3:30: 30 3 * * * /usr/local/bin/poudriere jail -u -j freebsd-11-amd64 # Build at 4 0 4 * * * poudriere bulk -cj freebsd-11-amd64 -p HEAD -f /usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/iocage -f /usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/nextcloud -f /usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/emby ``` ### Upgrade jails To upgrade releases, ie 11.0 to 11.1: ```shell /usr/local/bin/poudriere jail -u -t 11.1-RELEASE -j freebsd-11-amd64 ``` Or delete and re-create ```shell poudriere jail -d -j freebsd-11-amd64 poudriere jail -c -j freebsd-11-amd64 -v 11.1-RELEASE ``` Re-create ports tree: ```shell poudriere ports -d -p HEAD poudriere ports -c -p HEAD ``` ### Add new ports Add additional lists. for example, Emby: Add ports. ```shell ee /usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/emby ``` ```shell multimedia/ffmpeg graphics/ImageMagick ``` ### Poudriere options: ```shell poudriere options -j freebsd-11-amd64 -p HEAD -f /usr/local/etc/poudriere.d/portlists/freebsd-11-amd64/emby ``` For ffmpeg: * enable the ass subtitles option * enable the lame option * enable the opus subtitles option * enable the x265 subtitles option For ImageMagick * disable (unset) 16BIT\_PIXEL (to increase thumbnail generation performance) ## Reference * [DigitalOcean](https://www.digitalocean.com/community/tutorials/how-to-set-up-a-poudriere-build-system-to-create-packages-for-your-freebsd-servers) # FreeNAS This section has various topics related to the FreeNAS server operating system. # Copy SSH Keys off FreeNAS To copy ssh keys using `ssh-copy-id` off of FreeNAS an `ssh-agent` needs to be started . On FreeNAS run. ```shell sh eval `ssh-agent -s` ``` Then send any keys to a remote server. ```shell ssh-copy-id @ ``` # FreeNAS Service jails Manual setup of various services in FreeNAS jails. I have found manually set up services to be much more reliable then using FreeNAS' built in plugins. ## Deluge Setup of a jail for deluge server. ### FreeNAS Configuration #### User Use the media user from FreeNAS, It's important to check the UID and GID match up with the user's for any datasets shared with the jail. I have found the media user is usually already correctly matched. Create a dataset for deluge and mount to your desired location inside the jail. Mount the desired location inside the jail, I mounted mine to the `${HOME}/.config` directory of my deluge user. ### jail The following sections were done inside the jail. #### Install Deluge Install `deluge` or `deluge-cli` depending on what you want installed. Since this is a headless server I'm only installing the CLI version. ```shell pkg update && pkg upgrade pkg install deluge-cli ``` #### Init Script Setup `/etc/rc.conf` ```shell sysrc 'deluged_enable=YES' 'deluged_user=media' ``` #### Start Service ```shell service deluged start ``` ## Couchpotato Install [couchpotato](https://couchpota.to/#freebsd) freebsd version from git. ### FreeNAS UI Create database dataset couchpotato and mount to `/var/db/couchpotato`. ```shell pkg update && pkg upgrade ``` Install required tools ```shell pkg install python py27-sqlite3 fpc-libcurl docbook-xml git-lite ``` Use user media, clone to a temp repo in `/var/db`. ```shell cd /var/db git clone https://github.com/CouchPotato/CouchPotatoServer.git temp ``` Move the bare repo that was just cloned to the dataset we mounted earlier to `/var/db/couchpotato`. ```shell mv temp/.git couchpotato/ rm -rf temp ``` Switch to the `media` user and reset the repo to HEAD. ```shell su media cd couchpotato git reset --hard HEAD exit ``` As root, copy the startup script to `/usr/local/etc/rc.d` and make the startup script executable. ```shell cp couchpotato/init/freebsd /usr/local/etc/rc.d/couchpotato chmod 555 /usr/local/etc/rc.d/couchpotato ``` Read the options at the top of `/usr/local/etc/rc.d/couchpotato`. If not using the default install, specify options with startup flags. ```shell sysrc 'couchpotato_enable=YES' sysrc 'couchpotato_user=media' sysrc 'couchpotato_dir=/var/db/couchpotato' ``` Finally, start couchpotato. ```shell service couchpotato start ``` Restart the jail, open your browser and go to . ## Emby ### FreeNAS Create dataset, mount at `/var/db/emby` ### Jail In the jail, update all packages and install `emby-server`. ```shell pkg update && pkg upgrade pkg install emby-server ``` ### FFMpeg It's recommended to install ffmpeg from ports so that certain compile time options can be enabled. Update the FreeBSD ports tree ```shell portsnap fetch extract update ``` Remove the default ffmpeg package ```shell pkg delete -f ffmpeg ``` Reinstall FFMpeg from ports with lame option enabled ```shell cd /usr/ports/multimedia/ffmpeg && make config ``` * enable the lame option * enable the ass subtitles option * enable the opus subtitles option * enable the x265 subtitles option Compile and install. ```shell make install clean ``` ### ImageMagick It is recommended to recompile the graphics/ImageMagick package from ports with the following options . * disable (unset) 16BIT\_PIXEL (to increase thumbnail generation performance) Delete the imagemagick pkg. ```shell pkg delete -f imagemagick ``` Install from ports ```shell cd /usr/ports/graphics/ImageMagick && make config ``` * Disable the 16BIT\_PIXEL option ```shell make install clean ``` ## Emby Start Options Set the rc script executable. ```shell chmod 555 /usr/local/etc/rc.d/emby-server ``` Check the options. ```shell less /usr/local/etc/rc.d/emby-server ``` Set emby to start on boot and change the options based on setup. ```shell sysrc 'emby_server_enable=YES' sysrc 'emby_server_user=media' sysrc 'emby_server_group=media' sysrc 'emby_server_data_dir=/var/db/emby-server' ``` Start the emby service. ```shell service emby-server start ``` ## Pod ### In Jail Enter jail. ```shell jexec pod tcsh ``` Update. ```shell pkg update && pkg upgrade ``` ### Requirements ```shell pkg install bash libxslt wget curl ``` bash requires fdescfs(5) mounted on /dev/fd, add to boot tasks in FreeNAS UI. ```shell mount -t fdescfs fdesc /mnt/tank/jails/pod/dev/fd ``` ### Create User Create user 'pod'. ```shell adduser pod Username: pod Full name: Podcatcher Uid (Leave empty for default): Login group [pod]: Login group is pod. Invite pod into other groups? []: media Login class [default]: Shell (sh csh tcsh git-shell nologin) [sh]: bash Home directory [/home/pod]: Home directory permissions (Leave empty for default): Use password-based authentication? [yes]: Use an empty password? (yes/no) [no]: yes Lock out the account after creation? [no]: Username : pod Password : Full Name : Podcatcher Uid : 1001 Class : Groups : pod media Home : /home/pod Home Mode : Shell : /usr/local/bin/bash Locked : no OK? (yes/no): yes adduser: INFO: Successfully added (pod) to the user database. Add another user? (yes/no): no Goodbye! ``` ### Install bashpod Clone the script. ```shell su pod cd /home/pod git clone https://github.com/johnramsden/bashpod.git ``` ### FreeNAS Task In order to run from FreeNAS, create a new task that runs the bashpod script. ```shell jexec -U pod pod /usr/local/bin/bash -c "/home/pod/bashpod/bashpod.sh" ``` ## Sabnzbd ## FreeNAS Create dataset, mount at `/var/db/sabnzbd` ### Jail Enter jail. ```shell jexec sickrage tcsh ``` Update and install sabnzbd. ```shell pkg update && pkg upgrade && pkg install sabnzbdplus ``` ```shell sysrc 'sabnzbd_enable=YES' sysrc 'sabnzbd_user=media' sysrc 'sabnzbd_group=media' sysrc 'sabnzbd_conf_dir=/var/db/sabnzbd' ``` Restart jail Edit config in `/var/db/sabnzbd`, change host to `0.0.0.0` ## SickRage ### In Jail Enter jail. ```shell jexec sickrage tcsh ``` Update. ```shell pkg update && pkg upgrade ``` Install requirements. ```shell pkg install py27-sqlite3 ``` Install SickRage. ```shell cd /var/db git clone https://github.com/SickRage/SickRage.git temp mv temp/.git sickrage/ rm -rf temp chown -R media:media sickrage/ su media cd sickrage/ git reset --hard HEAD ls runscripts/ ``` Copy the startup script ```shell cp /var/db/sickrage/runscripts/init.freebsd /usr/local/etc/rc.d/sickrage ``` Make startup script executable ```shell chmod 555 /usr/local/etc/rc.d/sickrage ``` Add settings to rc.conf ```shell sysrc 'sickrage_enable=YES' sysrc 'sickrage_user=media' sysrc 'sickrage_group=media' sysrc 'sickrage_dir=/var/db/sickrage' ``` Start SickRage. ```shell service sickrage start ``` ## Syncthing ### Create User Syncthing On FreeNAS with ID `983`, `nologin` ### In Jail Enter jail. ```shell jexec syncthing tcsh ``` Update and install syncthing. ```shell pkg update && pkg upgrade && pkg install syncthing ``` Add the following to `rc.conf`: ```shell sysrc 'syncthing_enable=YES' sysrc 'syncthing_user=syncthing' sysrc 'syncthing_group=syncthing' sysrc 'syncthing_dir=/var/db/syncthing' ``` ### Configure Start syncthing as an initial test: service syncthing start Edit vim `/var/db/syncthing/config.xml` and change the IP address which the GUI will be accessible from. This will enable accessing the GUI from a remote computer: Before: ```
127.0.0.1:8384
; xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; ``` After: ```
0.0.0.0:8384
; xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; ``` Restart the service for changes to apply: ```shell service syncthing restart ``` Finally, access the GUI by pointing a browser to the server's address and port, ie `http://SERVER_URL:8384`. # iocage Service jails Creating jails on FreeNAS can [now be done](http://doc.freenas.org/11/jails.html#managing-iocage-jails) with [iocage](https://github.com/iocage). ## Iocage Setup ```shell iocage activate tank ``` iocage create --release 11.2-RELEASE --name rust\ boot=on vnet=on dhcp=on bpf=yes\ allow\_raw\_sockets="1"\ ip4\_addr="vnet1|172.20.40.42/24"\ interfaces="vnet1:bridge1"\ defaultrouter="172.20.40.1"\ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" # Couchpotato jail ### couchpotato jail Setup for couchpotato service jail with iocage. #### On FreeNAS Create jail: ```shell iocage create --release 11.1-RELEASE --name couchpotato \ boot="on" vnet=on bpf=on \ allow_raw_sockets="1" \ ip4_addr="vnet1|172.20.40.31/24" \ interfaces="vnet1:bridge1" \ defaultrouter="172.20.40.1" \ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" ``` On Freenas create datasets: * Datasets * Couchpotato Data * `tank/data/database/couchpotato` * Media * For all media `tank/media/Movies/...` * Downloads * For all Downloads `tank/media/Downloads/...` Create media user/group using uid from freenas: ```shell iocage exec couchpotato 'pw useradd -n media -u 8675309' ``` Nullfs mount datasets in jail: Couchpotato data: ```shell iocage exec couchpotato 'mkdir -p /var/db/couchpotato && chown media:media /var/db/couchpotato' iocage fstab --add couchpotato '/mnt/tank/data/database/couchpotato /var/db/couchpotato nullfs rw 0 0' ``` Downloads: ```shell iocage exec couchpotato 'mkdir -p /media/Downloads/Complete /media/Downloads/Incomplete && chown -R media:media /media/Downloads' iocage fstab --add couchpotato '/mnt/tank/media/Downloads/Complete /media/Downloads/Complete nullfs rw 0 0' && \ iocage fstab --add couchpotato '/mnt/tank/media/Downloads/Incomplete /media/Downloads/Incomplete nullfs rw 0 0' ``` Setup directories: ```shell iocage exec couchpotato 'mkdir -p /media/Movie/Movies /media/Movie/Sports && chown -R media:media /media' ``` Repeat for media: ```shell iocage fstab --add couchpotato '/mnt/tank/media/Movie/Movies /media/Movie/Movies nullfs rw 0 0' && \ iocage fstab --add couchpotato '/mnt/tank/media/Movie/Sports /media/Movie/Sports nullfs rw 0 0' ``` Check fstab: ```shell iocage fstab --list couchpotato ``` Start jail and enter. ```shell iocage console couchpotato ``` ### In Jail Install [couchpotato](https://couchpota.to/#freebsd) freebsd version from git. ```shell pkg update && pkg upgrade ``` Install required tools ```shell pkg install python py27-sqlite3 fpc-libcurl docbook-xml git-lite ``` Use user media, clone to a temp repo in `/var/db`. ```shell cd /var/db git clone https://github.com/CouchPotato/CouchPotatoServer.git temp ``` Move the bare repo that was just cloned to the dataset we mounted earlier to `/var/db/couchpotato`. ```shell mv temp/.git couchpotato/ rm -rf temp ``` Switch to the `media` user and reset the repo to HEAD. ```shell su media cd couchpotato git reset --hard HEAD exit ``` As root, copy the startup script to `/usr/local/etc/rc.d` and make the startup script executable. ```shell mkdir /usr/local/etc/rc.d cp /var/db/couchpotato/init/freebsd /usr/local/etc/rc.d/couchpotato chmod 555 /usr/local/etc/rc.d/couchpotato ``` Read the options at the top of `/usr/local/etc/rc.d/couchpotato`. If not using the default install, specify options with startup flags. ```shell sysrc 'couchpotato_enable=YES' 'couchpotato_user=media' 'couchpotato_dir=/var/db/couchpotato' ``` Finally, start couchpotato. ```shell service couchpotato start ``` Restart the jail, open your browser and go to . # Deluge jail Setup for Deluge service jail with iocage. ## On FreeNAS Create jail: ```shell iocage create --release 11.1-RELEASE --name deluge \ boot="on" vnet=on bpf=on \ allow_raw_sockets="1" \ ip4_addr="vnet1|172.20.40.35/24" \ interfaces="vnet1:bridge1" \ defaultrouter="172.20.40.1" \ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" ``` On Freenas create datasets: * Datasets * Deluge Data * `tank/data/database/deluge/` * Download Datasets * For Complete Torrents `tank/media/Downloads/Complete` * For Incomplete Torrents `tank/media/Downloads/Incomplete` * For Torrents `tank/media/Torrents` Create media user/group using uid from freenas: ```shell iocage exec deluge 'pw useradd -n media -u 8675309' ``` Nullfs mount datasets in jail: Deluge data: ```shell iocage exec deluge 'mkdir -p /home/media/.config /media/Downloads/Complete /media/Downloads/Incomplete /media/Torrents' && \ iocage exec deluge 'chown media:media /home/media/.config /media/Downloads/Complete /media/Downloads/Incomplete /media/Torrents' && \ iocage fstab --add deluge '/mnt/tank/data/database/deluge /home/media/.config nullfs rw 0 0' && \ iocage fstab --add deluge '/mnt/tank/media/Downloads/Complete /media/Downloads/Complete nullfs rw 0 0' && \ iocage fstab --add deluge '/mnt/tank/media/Downloads/Incomplete /media/Downloads/Incomplete nullfs rw 0 0' && \ iocage fstab --add deluge '/mnt/tank/media/Torrents /media/Torrents nullfs rw 0 0' ``` Check fstab: ```shell iocage fstab --list deluge ``` Start jail and enter. ```shell iocage console deluge ``` ### Install Deluge Install `deluge` or `deluge-cli` depending on what you want installed. Since this is a headless server I'm only installing the CLI version. ```shell pkg update && pkg upgrade && pkg install deluge-cli ``` ### Init Script Setup `/etc/rc.conf` ```shell sysrc 'deluged_enable=YES' 'deluged_user=media' ``` ### Start Service ```shell service deluged start ``` # Emby jail Setup for Emby service jail with iocage. ## On FreeNAS Create jail: ```shell iocage create --release 11.2-RELEASE --name emby \ boot=on vnet=on dhcp=on bpf=yes \ allow_raw_sockets="1" \ ip4_addr="vnet1|172.20.40.21/24" \ interfaces="vnet1:bridge1" \ defaultrouter="172.20.40.1" \ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" ``` On Freenas create datasets: * Datasets * Emby Data * `tank/data/database/emby/emby-server` * `tank/data/database/emby/media-metadata` * Media * For all media `tank/media/Series/...` Create media user/group using uid from freenas: ```shell iocage exec emby 'pw useradd -n media -u 8675309' ``` Nullfs mount datasets in jail: Emby data: ```shell iocage exec emby 'mkdir -p /var/db/emby-server /mnt/emby/media-metadata' iocage exec emby 'chown media:media /var/db/emby-server' iocage exec emby 'chown media:media /mnt/emby/media-metadata' iocage fstab --add emby '/mnt/tank/data/database/emby/emby-server /var/db/emby-server nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/data/database/emby/media-metadata /mnt/emby/media-metadata nullfs rw 0 0' ``` Setup directories: ```shell iocage exec emby 'mkdir -p /media/Series/Series /media/Series/Lectures /media/Series/Documentary /media/Series/Anime /media/Series/Animated /media/Series/Podcasts/Audio /media/Series/Podcasts/Video /media/Naddy /media/Movie/Movies /media/Movie/Sports /mnt/backups' iocage exec emby 'chown -R media:media /media && chown -R media:media /mnt/backups' ``` Repeat for media: ```shell iocage fstab --add emby '/mnt/tank/media/Series/Series /media/Series/Series nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/media/Series/Podcasts/Audio /media/Series/Podcasts/Audio nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/media/Series/Podcasts/Video /media/Series/Podcasts/Video nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/media/Series/Lectures /media/Series/Lectures nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/media/Series/Documentary /media/Series/Documentary nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/media/Series/Anime /media/Series/Anime nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/media/Series/Animated /media/Series/Animated nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/media/Naddy /media/Naddy nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/backups/Lilan/Emby /mnt/backups nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/media/Movie/Movies /media/Movie/Movies nullfs rw 0 0' iocage fstab --add emby '/mnt/tank/media/Movie/Sports /media/Movie/Sports nullfs rw 0 0' ``` Check fstab: ```shell iocage fstab --list emby ``` Start jail and enter. ```shell iocage start emby iocage console emby ``` ## Jail In the jail, update all packages and install `emby-server`. ```shell pkg update && pkg upgrade && pkg install emby-server ``` ### Package Options Its reccomended to change some package options. Either build a package with poudriere with these changes, or make these changes on the emby jails packages. #### FFMpeg It's recommended to install ffmpeg from ports so that certain compile time options can be enabled. Update the FreeBSD ports tree ```shell portsnap fetch extract update ``` Remove the default ffmpeg package ```shell pkg delete -f ffmpeg ``` Reinstall FFMpeg from ports with lame option enabled ```shell cd /usr/ports/multimedia/ffmpeg && make config ``` * enable the lame option * enable the ass subtitles option * enable the opus subtitles option * enable the x265 subtitles option Compile and install. ```shell make install clean ``` #### ImageMagick It is recommended to recompile the graphics/ImageMagick package from ports with the following options . * disable (unset) 16BIT\_PIXEL (to increase thumbnail generation performance) Delete the imagemagick pkg. ```shell pkg delete -f imagemagick ``` Install from ports ```shell cd /usr/ports/graphics/ImageMagick && make config ``` * Disable the 16BIT\_PIXEL option ```shell make install clean ``` ## Emby Start Options Set the rc script executable. ```shell chmod 555 /usr/local/etc/rc.d/emby-server ``` Check the options. ```shell less /usr/local/etc/rc.d/emby-server ``` Set emby to start on boot and change the options based on setup. ```shell sysrc 'emby_server_enable=YES' sysrc 'emby_server_user=media' && sysrc 'emby_server_group=media' sysrc 'emby_server_data_dir=/var/db/emby-server' ``` Start the emby service. ```shell service emby-server start ``` # Poudriere WebUI jail Setup for poudriere package server jail with iocage. ## On FreeNAS Create jail: ```shell iocage create --release 11.1-RELEASE --name pkgrepo \ boot="on" vnet=on bpf=on \ allow_raw_sockets="1" \ ip4_addr="vnet1|172.20.40.40/24" \ interfaces="vnet1:bridge1" \ defaultrouter="172.20.40.1" \ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" ``` Mount packages from host into jail with nullfs. ```shell iocage exec pkgrepo 'mkdir -p /usr/local/poudriere/data/packages' iocage fstab --add pkgrepo '/mnt/tank/data/poudriere/packages /usr/local/poudriere/data/packages nullfs rw 0 0' ``` Check fstab: ```shell iocage fstab --list pkgrepo ``` Start jail and enter. ```shell iocage start pkgrepo iocage console pkgrepo ``` ## Jail In the jail, update all packages. ```shell pkg update && pkg upgrade ``` ## Web Server ```shell pkg install nginx && sysrc nginx_enable=YES ``` Remove all inside server in `/usr/local/etc/nginx/nginx.conf`, add: Check config and start nginx: ```shell service nginx configtest service nginx start ``` In jail, nullfs mount packages to same spot. Install nginx. ```shell server { listen 80 default; server_name pkgrepo.ramsden.network; root /usr/local/poudriere/data/packages; autoindex on; } ``` # Podcatcher jail Setup for pocatcher jail with iocage using 'bashpod', previously 'mashpodder'. ## On FreeNAS Create jail: ```shell iocage create --release 11.1-RELEASE --name pod \ boot="on" vnet=on bpf=on \ allow_raw_sockets="1" \ ip4_addr="vnet1|172.20.40.34/24" \ interfaces="vnet1:bridge1" \ defaultrouter="172.20.40.1" \ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" ``` Create media user/group using uid from freenas: ```shell iocage exec pod 'pw useradd -n media -u 8675309' ``` Nullfs mount any datasets in jail: pod data, created on FreeNAS: ```shell iocage exec pod 'mkdir -p /var/db/pod' iocage exec pod 'chown media:media /var/db/pod' iocage fstab --add pod '/mnt/tank/data/database/pod /var/db/pod nullfs rw 0 0' ``` Setup directories for downloads: ```shell iocage exec pod 'mkdir -p /media/Downloads/Complete && chown -R media:media /media' iocage fstab --add pod '/mnt/tank/media/Downloads/Complete /media/Downloads/Complete nullfs rw 0 0' ``` Check fstab: ```shell iocage fstab --list pod ``` Enter jail. ```shell iocage console pod ``` ## In Jail Update. ```shell pkg update && pkg upgrade ``` ## Requirements ```shell pkg install bash libxslt wget curl git ``` ## Install bashpod Clone the script to tempdir and move it to our mounded directory. ```shell cd /var/db git clone https://github.com/johnramsden/bashpod.git temp mv temp/.git pod/ rm -rf temp chown -R media:media pod/ su media cd pod/ git reset --hard HEAD exit ``` Inside `/var/db/pod/bashpod.sh`, set `BASEDIR="/var/db/pod"`. Symlink the script to `/usr/local/bin/bashpod`. ```shell ln -s /var/db/pod/bashpod.sh /usr/local/bin/bashpod ``` ## FreeNAS Task In order to run from FreeNAS, create a new task that runs the bashpod script. ```shell iocage exec pod --jail_user media '/usr/local/bin/bashpod' ``` # Sabnzbd jail Setup for Emby service jail with iocage. ## On FreeNAS Create jail: ```shell iocage create --release 11.1-RELEASE --name sabnzbd \ boot="on" vnet=on bpf=on \ allow_raw_sockets="1" \ ip4_addr="vnet1|172.20.40.32/24" \ interfaces="vnet1:bridge1" \ defaultrouter="172.20.40.1" \ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" ``` On Freenas create datasets: * Datasets * Sabnzbd Data * `tank/data/database/sabnzbd` * Downloads * For all downloads `tank/media/Downloads/...` Create media user/group using uid from freenas: ```shell iocage exec sabnzbd 'pw useradd -n media -u 8675309' ``` Nullfs mount datasets in jail: Sabnzbd data: ```shell iocage exec sabnzbd 'mkdir -p /var/db/sabnzbd' && \ iocage exec sabnzbd 'chown media:media /var/db/sabnzbd' && \ iocage fstab --add sabnzbd '/mnt/tank/data/database/sabnzbd /var/db/sabnzbd nullfs rw 0 0' ``` Downloads: ```shell iocage exec sabnzbd 'mkdir -p /media/Downloads/Complete /media/Downloads/Incomplete && chown -R media:media /media' iocage fstab --add sabnzbd '/mnt/tank/media/Downloads/Complete /media/Downloads/Complete nullfs rw 0 0' && \ iocage fstab --add sabnzbd '/mnt/tank/media/Downloads/Incomplete /media/Downloads/Incomplete nullfs rw 0 0' ``` Check fstab: ```shell iocage fstab --list sabnzbd ``` Start jail and enter. ```shell iocage console sabnzbd ``` ## Jail Update and install sabnzbd. ```shell pkg update && pkg upgrade && pkg install sabnzbdplus ``` ```shell sysrc 'sabnzbd_enable=YES' 'sabnzbd_user=media' 'sabnzbd_group=media' 'sabnzbd_conf_dir=/var/db/sabnzbd' ``` Restart jail Edit config in `/var/db/sabnzbd`, change host to `0.0.0.0` # Sickrage jail # Syncthing jail Setup for Syncthing service jail with iocage. ## On FreeNAS Create jail: ```shell iocage create --release 11.1-RELEASE --name syncthing \ boot="on" vnet=on bpf=on \ allow_raw_sockets="1" \ ip4_addr="vnet1|172.20.40.33/24" \ interfaces="vnet1:bridge1" \ defaultrouter="172.20.40.1" \ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" ``` Create user Syncthing on FreeNAS with ID `983`, `nologin` to match the user in the jail. On Freenas create datasets: * Datasets * Syncthing Data * `tank/data/syncthing` Nullfs mount datasets in jail: Syncthing data: ```shell iocage exec syncthing 'mkdir -p /mnt/syncthing/data' iocage fstab --add syncthing '/mnt/tank/data/syncthing/sync /mnt/syncthing/data nullfs rw 0 0' ``` Start jail and enter. ```shell iocage start syncthing iocage console syncthing ``` ## Jail In the jail, update all packages and install `syncthing`. ```shell pkg update && pkg upgrade pkg install syncthing ca_root_nss ``` Enable the service on boot. ```shell sysrc 'syncthing_enable=YES' sysrc 'syncthing_user=syncthing' && sysrc 'syncthing_group=syncthing' sysrc 'syncthing_home=/var/db/syncthing' ``` Start the syncthing service. ```shell service syncthing start ``` ## Configure Start syncthing as an initial test: ```shell service syncthing restart ``` Edit `/var/db/syncthing/config.xml` and change the IP address which the GUI will be accessible from. This will enable accessing the GUI from a remote computer: Before: ```
127.0.0.1:8384
; xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; ``` After: ```
0.0.0.0:8384
; xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; ``` Restart the service for changes to apply: ```shell service syncthing restart ``` Finally, access the GUI by pointing a browser to the server's address and port, ie `http://SERVER_URL:8384`. # Duplicity jail Setup for Duplicity service jail with iocage. ## On FreeNAS Create jail: ```shell iocage create --release 11.1-RELEASE --name duplicity \ boot="on" vnet=on bpf=on \ allow_raw_sockets="1" \ ip4_addr="vnet1|172.20.40.41/24" \ interfaces="vnet1:bridge1" \ defaultrouter="172.20.40.1" \ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" ``` Create user on FreeNAS with ID `983`, `nologin` to match the user in the jail. Nullfs mount datasets to backup in jail: Duplicity data: ```shell iocage exec duplicity 'mkdir -p /mnt/duplicity/data' iocage fstab --add duplicity '/mnt/tank/data/syncthing/sync /mnt/duplicity/data nullfs rw 0 0' ``` Start jail and enter. ```shell iocage console duplicity ``` ## Jail In the jail, update all packages and install `duplicity` and `py27-boto`. ```shell pkg update && pkg upgrade pkg install duplicity py27-boto ``` Create a user with uid `983` to match mounted data. ```shell pw useradd -n duplicity -u 983 ``` Add script `/usr/local/scripts/duplicitybak`, put secrets in `/usr/local/scripts/duplicitybak.auth`. ```shell #!/bin/sh # on freebsd install duplicity, py27-boto # Place auth variables: PASSPHRASE, GS_ACCESS_KEY_ID, GS_SECRET_ACCESS_KEY . "/usr/local/scripts/duplicitybak.auth" # Folders to backup BACKUP_DATA_REGEXP='Workspace|Computer|Personal|Pictures|University' BACKUP_ROOT="/mnt/duplicity/data" # GS configuration variables GS_BUCKET="johnramsdenbackup" # Remove files older than 60 days from GS duplicity remove-older-than 60D --force gs://${GS_BUCKET} # Sync everything to GS duplicity --include-regexp "${BACKUP_DATA_REGEXP}" \ --exclude='**' \ ${BACKUP_ROOT} gs://${GS_BUCKET} # Cleanup failures duplicity cleanup --force gs://${GS_BUCKET} unset PASSPHRASE unset GS_ACCESS_KEY_ID unset GS_SECRET_ACCESS_KEY ``` Secrets in `/usr/local/scripts/duplicitybak.auth`: ```shell # Create password to use for symetric GPG encryption export PASSPHRASE="" # Create GS bucket, https://console.cloud.google.com/storage/ # enable interoperable access, get keys export GS_ACCESS_KEY_ID="" export GS_SECRET_ACCESS_KEY="" ``` Set executable: ```shell chmod +x /usr/local/scripts/duplicitybak ``` Now I can be run from a crontab outside of the jail: ```shell iocage exec duplicity /usr/local/scripts/duplicitybak ``` # Lets Encrypt jail ### Lets Encrypt jail Setup for letsencrypt service jail with iocage. #### On FreeNAS Create jail: ```shell iocage create --release 11.1-RELEASE --name letsencrypt \ boot="on" vnet=on bpf=on \ allow_raw_sockets="1" \ ip4_addr="vnet1|172.20.40.38/24" \ interfaces="vnet1:bridge1" \ defaultrouter="172.20.40.1" \ resolver="search ramsden.network;nameserver 172.20.40.1;nameserver 8.8.8.8" ``` **Datasets** On FreeNAS create user and group acme, GID/UID 169. In web ui create mount datasets: * letsencrypt * letsencrypt Data * mountpoint: `/var/db/acme/` \* `/mnt/tank/data/database/letsencrypt/acme` * certs * mountpoints: `/mnt/certs/` \* `couchpotato.ramsden.network` * `/mnt/certs/couchpotato.ramsden.network` * `/mnt/tank/data/database/letsencrypt/certs/couchpotato.ramsden.network` \* `emby.ramsden.network` * `/mnt/certs/emby.ramsden.network` * `/mnt/tank/data/database/letsencrypt/certs/emby.ramsden.network` \* `lilan.ramsden.network` * `/mnt/certs/lilan.ramsden.network` * `/mnt/tank/data/database/letsencrypt/certs/lilan.ramsden.network` \* `sabnzbd.ramsden.network` * `/mnt/certs/sabnzbd.ramsden.network` * `/mnt/tank/data/database/letsencrypt/certs/sabnzbd.ramsden.network` \* `sickrage.ramsden.network` * `/mnt/certs/sabnzbd.ramsden.network` * `/mnt/tank/data/database/letsencrypt/certs/sabnzbd.ramsden.network` \* `syncthing.ramsden.network` * `/mnt/certs/syncthing.ramsden.network` * `/mnt/tank/data/database/letsencrypt/certs/syncthing.ramsden.network` Have the acme user own thedataset`tank/data/database/letsencrypt/acme`. Mount `/mnt/tank/data/database/letsencrypt/acme` to `/var/db/acme/` Mount the certs under `/var/db/acme/certs/` Nullfs mount datasets in jail: letsencrypt data: ```shell iocage exec letsencrypt 'mkdir -p /var/db/acme' iocage fstab --add letsencrypt '/mnt/tank/data/database/letsencrypt/acme /var/db/acme nullfs rw 0 0' ``` Setup directories for certs: ```shell iocage exec letsencrypt 'mkdir -p /mnt/certs/couchpotato.ramsden.network /mnt/certs/emby.ramsden.network /mnt/certs/lilan.ramsden.network /mnt/certs/sabnzbd.ramsden.network /mnt/certs/sickrage.ramsden.network /mnt/certs/syncthing.ramsden.network' ``` Mount the directories: ```shell iocage fstab --add letsencrypt '/mnt/tank/data/database/letsencrypt/certs/couchpotato.ramsden.network /mnt/certs/couchpotato.ramsden.network nullfs rw 0 0' iocage fstab --add letsencrypt '/mnt/tank/data/database/letsencrypt/certs/emby.ramsden.network /mnt/certs/emby.ramsden.network nullfs rw 0 0' iocage fstab --add letsencrypt '/mnt/tank/data/database/letsencrypt/certs/lilan.ramsden.network /mnt/certs/lilan.ramsden.network nullfs rw 0 0' iocage fstab --add letsencrypt '/mnt/tank/data/database/letsencrypt/certs/sabnzbd.ramsden.network /mnt/certs/sabnzbd.ramsden.network nullfs rw 0 0' iocage fstab --add letsencrypt '/mnt/tank/data/database/letsencrypt/certs/sickrage.ramsden.network /mnt/certs/sickrage.ramsden.network nullfs rw 0 0' iocage fstab --add letsencrypt '/mnt/tank/data/database/letsencrypt/certs/syncthing.ramsden.network /mnt/certs/syncthing.ramsden.network nullfs rw 0 0' ``` Check fstab: ```shell iocage fstab --list letsencrypt ``` Start jail and enter. ```shell iocage console letsencrypt ``` #### Jail In the jail, update all packages and install `acme.sh`. ```shell pkg update && pkg upgrade pkg install acme.sh ``` Switch to the ‘acme’ user which renews the certificate on a cron job add configuration. ```shell su - acme ``` Issue cert ```shell export CF_Email="****************" export CF_Key="****************" acme.sh --issue --dns dns_cf -d emby.ramsden.network ``` Add acme to le in FreeNAS and jail. ```shell pw groupadd -n le -g 2000 && pw groupmod le -m acme ``` chown certs dir in freenas to acme:le recursively. ### Set Install Location Now, to set the install location for the certificates use the installcert command, for example: ```shell acme.sh --installcert -d lilan.ramsden.network \ --certpath /mnt/certs/lilan.ramsden.network/Lilan_s_LetsEncrypt_Certificate.crt \ --keypath /mnt/certs/lilan.ramsden.network/Lilan_s_LetsEncrypt_Certificate.key ``` Cert deploy location: /etc/certificates ### Various Services Various Services need their certificates installed two different locations, and some of them need some changes. There are a few that I make changes to from the default. #### Emby Emby needs pks file, to convert cert key cert and ca are needed Set deploy location ```shell ACME_BIN="~/.acme.sh/acme.sh" SERVER="emby.ramsden.network" CERT_DEPLOY_DIR="/mnt/certs" # Certs CERT="${SERVER}.cer" KEY="${SERVER}.key" CA="ca.cer" PKCS="${SERVER}.pfx" # Set deploy location: acme.sh --installcert -d "${SERVER}" \ --certpath "${CERT_DEPLOY_DIR}/${SERVER}/${CERT}" \ --keypath "${CERT_DEPLOY_DIR}/${SERVER}/${KEY}" \ --capath "${CERT_DEPLOY_DIR}/${SERVER}/${CA}" # Convert to pkcs openssl pkcs12 -export -out ${CERT_DEPLOY_DIR}/${SERVER}/${PKCS} \ -inkey ${CERT_DEPLOY_DIR}/${SERVER}/${KEY} \ -in ${CERT_DEPLOY_DIR}/${SERVER}/${CERT} \ -certfile ${CERT_DEPLOY_DIR}/${SERVER}/${CA} \ -passout pass: ``` Install directory in jail: /var/db/emby-server/ssl #### Cron: Crontab from freenas: You probably want to renew starts on a crontab so they get done every month. I use the following script to renew my various services: ```shell #!/bin/sh # letsencrypt Jail le_jail="letsencrypt" le_user="acme" cert_db="/mnt/tank/data/database/letsencrypt/certs" jail_db="/mnt/tank/data/database" # Cloudflare account export CF_Email="" export CF_Key="" ############# MAIN CODE ############# convert_pkcs(){ server="${1}" pass="${2}" out_name="${3}" key="${4}" cert="${5}" ca="${6}" echo echo "Generating pkcs for ${server}" echo "to ${cert_db}/${server}/${out_name}" openssl pkcs12 -export -out "${cert_db}/${server}/${out_name}" \ -inkey ${cert_db}/${server}/${key} \ -in ${cert_db}/${server}/${cert} \ -certfile ${cert_db}/${server}/${ca} \ -passout ${pass} } # Install to jail, locations relative to jail db # eg # deploy "emby/ssl" "letsencrypt/certs" "media" "media" "660" deploy(){ server="${1}" deploy_location="${2}" owner="${3}" group="${4}" perms="${5}" echo echo "Installing certs for: ${server}" echo "with deploy location: ${deploy_location}" # Install certs to {} find "${cert_db}/${server}/" -type f \ -exec install -b -m ${perms} \ -o ${owner} -g ${group} {} ${deploy_location} \; } # Run acme in jail to check if certs need renewing, if so renew iocage exec --jail_user ${le_user} ${le_jail} /bin/sh -c \ 'acme.sh --cron --force --home "/var/db/acme/.acme.sh"' # convert emby's key to pkcs convert_pkcs "emby.ramsden.network" "pass:" \ "emby.ramsden.network.pfx" \ "emby.ramsden.network.key" \ "emby.ramsden.network.cer" \ "ca.cer" # deploy emby's certs deploy "emby.ramsden.network" \ "${jail_db}/emby/emby-server/ssl/" \ "media" "media" "770" # Restart emby iocage exec emby /bin/sh -c 'service emby-server restart' # deploy lilan's certs? Saved in /etc/certificates #install -b -B ".old-`date +%Y-%m-%d-%H:%M:%S`" -m 400 -o root -g wheel \ #/mnt/tank/data/database/letsencrypt/certs/lilan.ramsden.network/Lilan_s_LetsEncrypt_Certificate.key \ #/etc/certificates #deploy "lilan.ramsden.network" \ # "/etc/certificates" \ # "root" "wheel" "400" echo echo "Finished deploying keys" ``` # Wrong Version jail If a different version of FreeNAS is running from the version of a jail. For example, the FreeBSD version FreeNAS is based upon is 11.0, but a jail is based on 10.3, the following error can occur when attempting to install something from ports in a jail. > make: "/usr/ports/Mk/bsd.port.mk" line 1177: UNAME\_r (11.0-STABLE) and OSVERSION (1003000) do not agree on major version number There are two solutions to this problem, the non-permanent option is to set an environment variable in the current shell with `setenv UNAME_r 10.3-RELEASE`. The permanent solution is to set the environment variable by editing `/etc/login.conf`, and adding the jail's version. ```shell default:\ :setenv=UNAME_r=10.3-RELEASE:\ ``` Reset the database. ```shell cap_mkdb /etc/login.conf ``` Exit and enter shell and everything should work. # pfSense This section contains articles related to the FreeBSD based pfSense router operating system. # Sending Specific Traffic Through OpenVPN Using Private Internet Access, follow instructions on [client support](https://www.privateinternetaccess.com/pages/client-support/pfsense). ### Certificate Authority Download cert from VPN provider. For PIA it's located at Navigate to System -> Cert Manager -> CAs. Add a new CA for PIA with above cert in 'Certificate data' field. ### OpenVPN Navigate to VPN -> OpenVPN -> Clients. Add client with the following settings. * Server host, choose from [available hosts](https://www.privateinternetaccess.com/pages/network/): nl.privateinternetaccess.com * Protocol: UDP * Server port: 1198 * Server hostname resolution: Ensure that "Infinitely resolve server" is checked. * User Authentication Settings: Fill the Username and Password fields with your PIA username and password. * TLS Authentication: Ensure "Enable authentication of TLS packets" is disabled. * Peer Certificate Authority: Select the PIA CA we setup. * Client Certificate: None (Username and/or Password required) * Encryption Algorithm: AES-128-CBC (128-bit). * Auth digest algorithm: SHA1 (160-bit). * Compression: Enabled with Adaptive Compression. * Disable IPv6: Ensure "Don't forward IPv6 traffic" is checked. * Custom options: Copy and paste the following into the custom options textbox. 'route-nopull' prevents the VPN client from creating a standard rule that forces ALL traffic through the VPN connection.: persist-key; persist-tun; remote-cert-tls server; reneg-sec 0; route-nopull; Navigate to Status -> OpenVPN. Check if status up. ### Create Interface Create a new interface, Interface -> Assign, and select the OpenVPN connection, enable. Create Alias OpenVPNHosts for IP's to send through OpenVPN. Check System: Advanced: Miscellaneous, "Skip rules when gateway is down": By default, when a rule has a specific gateway set, and this gateway is down, rule is created and traffic is sent to default gateway.This option overrides that behavior and the rule is not created when gateway is down. ### Mappings 1. Navigate to Firewall -> NAT -> Outbound. 2. Set the Mode under General Logging Options to "Manual Outbound NAT rule generation (AON)", and click Save. 3. Under the Mappings section, click the duplicate (dual-page) icon on the right for the first rule shown in the list. 4. Set Interface to "OpenVPN" and click Save at the bottom. 5. For interfaces to use the VPN, repeat the last two steps for all remaining rule shown under Mappings, until every rule has a duplicate for OpenVPN. ### Rules Add a firewall rule for interface(s) being sent through VPN. Protocol: Any Source: Single Host, OpenVPNHosts Advanced: Set Gateway to VPN. Add rule to block if VPN down Protocol: Any Source: Single Host, OpenVPNHosts Advanced: Set Gateway to VPN. ### Check IP Check IP on CLI with: ```shell curl -s checkip.dyndns.org ``` ### References * [Tunneling Specific Traffic over a VPN with pfSense - Muffin's Lab](https://blog.monstermuffin.org/tunneling-specific-traffic-over-a-vpn-with-pfsense/) * [Routing SOME traffic / static IPs through OpenVPN (over PIA) - pfsense forum](https://forum.pfsense.org/index.php?topic=72902.0) # Desktop and Userspace Things that may be done by a user on the desktop, that are platform specific (on UNIX-based systems) and do not belong in any other section, are in this section. # Gaming Section containing various gaming related information. # Grim Dawn Install `winetricks` and `wine-staging` ## Grim Dawn Create individual bottle, enable `staging -> CSMT`: ```shell env WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/grimdawn winecfg ``` Install requirements: ```shell env WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/grimdawn winetricks \ vcrun2010 vcrun2012 xact xinput d3dx9 ``` Download Windows Installer and run: ```shell env WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/grimdawn wine \ ${HOME}/.local/share/wine/grimdawn/drive_c/users/john/Downloads/SteamSetup.exe ``` After installing, install Grim Dawn in steam. If errors occur with `steamwebui`, run `winecfg` and set ONLY steam to run in XP mode. If errors stull occur use `-no-cef-sandbox`. Now grim dawn should start with: ```shell env WINEDEBUG=-all WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/grimdawn wine \ "${HOME}/.local/share/wine/grimdawn/drive_c/Program Files/Steam/steamapps/common/Grim Dawn/Grim Dawn.exe" -no-cef-sandbox ``` Create a desktop file in linux to start Grim Dawn. ```shell nano "${HOME}/.local/share/applications/Grim Dawn.desktop" ``` ```shell [Desktop Entry] Exec=env WINEDEBUG=-all WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/grimdawn wine "${HOME}/.local/share/wine/grimdawn/drive_c/Program Files/Steam/steamapps/common/Grim Dawn/Grim Dawn.exe" -no-cef-sandbox GenericName=Dark fantasy ARPG with fast paced combat and massive exploration. Icon=${HOME}/.local/share/wine/grimdawn/drive_c/Program Files/Steam/steam/games/889a02bbebe088f7bd4f011ae641732481b1b3d6.ico Name=Grim Dawn NoDisplay=false Path[$e]= StartupNotify=true Terminal=0 ``` # Path of Exile The following describes how to setup Path of Exile. Prerequisites (Arch only): * [Gaming with Wine](https://github.com/johnramsden/docs/blob/gitbook/operatingsystems/linux/distributions/archlinux/wine.html) ## Dataset Create a ZFS dataset for wine bottle. ```shell zfs create -o mountpoint=legacy vault/sys/$(hostname)/home/john/local/share/wine ``` Add to fstab: ```shell vault/sys/chin/home/john/local/share/wine /home/john/.local/share/wine zfs rw,relatime,xattr,noacl 0 0 ``` Mount it ```shell mkdir /home/john/.local/share/wine mount -a ``` ## Configuration Always use `env WINEPREFIX=${HOME}/.local/share/wine/` when creating bottles. Otherwise wine defaults to `~/.wine`. Install dependencies: ```shell pacman -S mpg123 lib32-gst-plugins-base-libs pulseaudio-alsa lib32-libpulse lib32-alsa-plugins lib32-libldap lib32-openal ``` ```shell pacaur -S ttf-ms-fonts ttf-tahoma ``` To create a 32bit bottle use `WINEARCH=win32`. ```shell env WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/pathofexile winecfg ``` Enable CSMT, optionallenable emulate virtual desktop and change DPI. ## Tuning Get videocard RAM: ``` echo $"VRAM: "$(($(grep -P -o -i "(?<=memory:).*(?=kbytes)" /var/log/Xorg.0.log) / 1024))$" Mb" ``` Set in regedit. Copy the number. ```shell env WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/pathofexile wine regedit ``` Go to `HKEY_CURRENT_USER>Software>Wine` * Add key "Direct3D" * Add new string to Direct3D folder. Right click>New String, type "VideoMemorySize", add "VideoMemorySize" string, use video memory number ## With Installer Install dependencies ```shell env WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/pathofexile winetricks -q glsl=disabled directx9 usp10 msls31 ``` Download and execute installer. ```shell cd ${HOME}/.local/share/wine/pathofexile wget https://www.pathofexile.com/downloads/PathOfExileInstaller.exe env WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/pathofexile wine ${HOME}/.local/share/wine/pathofexile/PathOfExileInstaller.exe ``` Run game launcher. ```shell env WINEDEBUG=-all WINEARCH=win32 WINEPREFIX=${HOME}/.local/share/wine/pathofexile wine "${HOME}/.local/share/wine/pathofexile/drive_c/Program Files/Grinding Gear Games/Path of Exile/PathOfExile.exe" dbox -no-dwrite -noasync ``` # Internet Section containing various internet related information. # Re-authenticate IRC Nickname If the password isn't entered in the allowed time in IRC, a user can be blocked from signing in, and have their name temporarily changed to guest. > You have 30 seconds to identify to your nickname before it is changed. > > You failed to identify in time for the nickname `` You are now known as Guest59588 If that occurs, doing the following can allow attempting to sign in again. Identify your nick (replacing nick and password of course). ```shell /quote NickServ identify $nick $password ``` Turn off enforce. ```shell /quote NickServ set enforce OFF ``` Release your nick from services: ```shell /quote NickServ release $nick $password ``` Login again. ```shell /nick $nick ``` Turn enforce back on ```shell /quote NickServ set enforce on ``` # Lightdm VNC Connection with Password To connect to a session being started with lightdm, install a vnc server such as tightvnc. Run `vncpasswd` as the user who will connect. Add the following to `/etc/lightdm/lightdm.conf`. Where `john` is replaced by the user. ```shell [VNCServer] command=/usr/bin/Xvnc -rfbauth /home/john/.vnc/passwd enabled=true port=5900 width=1920 height=1080 depth=16 ``` # Media This section contains ways to accomplish various media related tasks. # Convert Audio to Video Using [find](https://github.com/johnramsden/docs/blob/gitbook/shell_find.html), loop over all mp3s and convert them to mp4 videos; applying an image which will form the video's backdrop. ```shell find -name "*.mp3" -exec ffmpeg -loop 1 -i ${imagepath} -i {} -c:a aac -strict experimental -b:a 192k -shortest {}.mp4 \; ``` # Convert Text to Speech To convert a plain text file to an mp3, use the [espeak](http://espeak.sourceforge.net/) speech synthesizer and [ffmpeg](https://www.ffmpeg.org/) to save the audio as an mp3. ```shell espeak -f Book.txt --stdout | ffmpeg -i - -ab 192k -y AudioBook.mp3 ``` To convert a pdf to an mp3, use [pdftotext](http://www.foolabs.com/xpdf/home.html). ```shell pdftotext Book.pdf - | espeak --stdout | ffmpeg -i - -ab 192k -y AudioBook.mp3 ``` # System Administration System administration and related topics. # Security This section focuses on security-related topics. # GPG Subkeys List keys to get your key: ```shell gpg --list-keys ``` Edit key: ```shell gpg --edit-key ``` At prompt, add a new subkey, select signing or encrypting, keysize, and expiry: ```shell gpg> addkey Please select what kind of key you want: (3) DSA (sign only) (4) RSA (sign only) (5) Elgamal (encrypt only) (6) RSA (encrypt only) Your selection? 4 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 4096 Requested keysize is 4096 bits Please specify how long the key should be valid. 0 = key does not expire = key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 2y Key expires at Wed 04 Sep 2019 10:51:34 PM PDT Is this correct? (y/N) y Really create? (y/N) y ``` Repeat for encrypting key if you need one. ### Exporting the Subkey(s) Get your new subkey's ID you want to export. ```shell gpg --list-keys --with-subkey-fingerprint ``` Export the subkey, keeping the `!`, can list multiple keys: ```shell gpg -a --export-secret-subkeys ! [ !] > temp_directory/subkey.gpg ``` To change the passphrase, import the key into a temporary folder. ```shell mkdir temp_directory/gpg gpg --homedir temp_directory/gpg --import temp_directory/subkey.gpg ``` Edit the key, and change the passphrase. ```shell gpg --homedir temp_directory/gpg --edit-key ``` ```shell > passwd > save ``` Note: You will get a warning "error changing passphrase", but it can be ignored. Now export again as the new, altered subkey. ```shell gpg --homedir temp_directory/gpg -a --export-secret-subkeys [subkey id]! > temp_directory/subkey.altpass.gpg ``` ### Importing The Subkey(s) Now, on a new system, the subkeys can be imported: ```shell gpg --import subkey.altpass.gpg ``` Checking `gpg --list-secret-keys` will show a `#` after sec, meaning the master key isn't present: On new, subkey only system: ```shell /home/john/.gnupg/pubring.kbx ----------------------------- sec# rsa4096 2017-05-17 [SC] uid [ unknown] John Ramsden () uid [ unknown] John Ramsden () ssb rsa4096 2017-09-05 [S] [expires: 2019-09-05] ssb rsa4096 2017-09-05 [E] [expires: 2019-09-05] ``` References: * [Arch Wiki - GnuPG](https://wiki.archlinux.org/index.php/GnuPG#Edit_your_key) * [Debian - Subkeys](https://wiki.debian.org/Subkeys) * [void.gr](https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/) # SSH Signing Keys # Shell Scripting This section conatins a compilation of useful commands or small shell scripts. # dd The following page contains uses for the `dd` command. ### Image a Disk Using `dd`, with disk sdX, create a gzipped image: ``` dd if=/dev/sdX conv=sync,noerror bs=64K | gzip -c > /path/to/backup.img.gz ``` If fat32, split into volumes: ``` dd if=/dev/sdX conv=sync,noerror bs=64K | gzip -c | split -a3 -b2G - /path/to/backup.img.gz ``` Save the drive geometry. ``` fdisk -l /dev/sdX > /path/to/list_fdisk.info ``` To restore a system: ``` gunzip -c /path/to/backup.img.gz | dd of=/dev/sdX ``` Or, if it's been split: ``` cat /path/to/backup.img.gz* | gunzip -c | dd of=/dev/sdX ``` # find This is a compilation of useful ways to use the `find` command. ### Delete from Extension Delete all files with the 'jpeg' extension. ``` find . -type f -name '*.jpeg' -delete ``` # rsync ## Progress To get better progress statistics, force rsync to calculate files before transfer using `--no-i-r` and `--info=progress2` For example: ``` rsync -r --info=progress2 src dest ``` ## NTFS problems I've had issues with transfers to NTFS drives crashing. Not transferring permissions and groups as well as using specific ownership after transfer seems to resolve the issue. ``` rsync -r --no-p --no-g --chmod=ugo=rwX src dest ``` # vim Line select by keyboard: * `V` Indent: select, then `>>` Copy: * `v`, select, `y` copy * `p` to paste Cut: * `v`, select, `d` cut * `p` to paste # Ceph This section focuses on [Ceph](https://ceph.io/en/) Install on local via [microceph](https://github.com/canonical/microceph) ### Paper [Ceph: A Scalable, High-Performance Distributed File System](https://ceph.io/assets/pdfs/weil-ceph-osdi06.pdf) ### Terms * RADOS: * OSD (object storage devices): Combine a CPU, network interface, and local cache with an underlying disk or RAID * OSDs replace the traditional block-level interface with one in which clients can read or write byte ranges to much larger (and often variably sized) named objects, distributing low-level block allocation decisions to the devices themselves * Clients typically interact with a metadata server (MDS) to perform metadata operations (open, rename), while communicating directly with OSDs to perform file I/O (reads and writes), significantly improving overall scalability ### System Overview Three main components: * client - exposes "near-POSIX" file system interface to a host or process * cluster of OSDs, which collectively stores all data and metadata * metadata server cluster - manages the namespace (file names and directories) while coordinating security, consistency and coherence Goals: Scalability, performance, and reliability # ZFS Topics related to administration of ZFS. # Mirrors Creating a mirrored ZFS pool is easy. ### Two disk mirror To create a single two disk mirror: ```shell zpool create -f -o ashift=12 vault mirror \ ata-SanDisk_SDSSDXPS480G_152271401093 \ ata-SanDisk_SDSSDXPS480G_154501401266 ``` ### Four disk mirror - RAID10 To create a RAID10 style pool, create multiple mirrors. As many mirrors as desired can be added. ```shell zpool create -f -o ashift=12 vault \ mirror \ ata-SanDisk_SDSSDXPS480G_152271401093 \ ata-SanDisk_SDSSDXPS480G_154501401266 \ mirror \ ata-SanDisk_SDSSDXPS480G_164277402487 \ ata-SanDisk_SDSSDXPS480G_164277402657 ``` This created the following. ```shell zpool status pool: vault state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM vault ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ata-SanDisk_SDSSDXPS480G_152271401093 ONLINE 0 0 0 ata-SanDisk_SDSSDXPS480G_154501401266 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ata-SanDisk_SDSSDXPS480G_164277402487 ONLINE 0 0 0 ata-SanDisk_SDSSDXPS480G_164277402657 ONLINE 0 0 0 errors: No known data errors ``` # Certifications Content from cert studying. # CKA [Certified Kubernetes Administrator](https://www.cncf.io/certification/cka/) [Exam Curriculum (Topics)](https://github.com/cncf/curriculum) [Candidate Handbook](https://www.cncf.io/certification/candidate-handbook) [Exam Tips](http://training.linuxfoundation.org/go//Important-Tips-CKA-CKAD) # Core-Concepts ## Cluster Architecture * Kubelet listens for commanda (on each node) * Kube proxy manages communication between workers (on each node) ### Containers CRI - lets different solutions for running containers work (containerd etc) Imagespec - how container images are setup Runtimespec - how containers run ### ContainerD For debugging `ctr` official tool Alt tool: `nerdctl` - more user friendly, similar to `docker` cli `crictl` works across all CRI runtimes, good for debugging Very similar to `docker` ### etcd * KV store * 2 main APIs (v2, and v3), significant API change * All k8s changes modify etcd ### Components * kube-apiserver * Who you talk to with `kubectl` * Only think that talks to `etcd` * either * process with settings in systemd service * or pod with settings in `/etc/kubernetes/manifests/kube-apiserver.yaml` (kubeadm) * kube-scheduler * Schedules pods on workers, updates etcd * decides which pod goes where based on requirements * kubelet * Makes changes on worker * does EVERYTHING on node, communicates with api-server * Need to run on worker as service * Controller-Manager (brain of k8s) * Manages controllers (processes that monitor status of components, nodes etc) * Controllers are inside Controller-Manager process * kube-proxy * Deals with communications * Internal IPs can change on nodes, we use services instead of pod IPs * kube-proxy runs on each node and creates rules based on services so pod is accessible ### Pods * We can create pods with `yaml` * Several keys required in yaml Required: ```yaml apiVersion: kind: metadata: spec: ``` Typical pod values: ```yaml apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app: myapp spec: containers: - name: nginx-container image: nginx ``` ```shell kubectl create -f $FILE.yaml kubectl describe myapp-pod ``` For viewing state: ```shell kubectl describe pod webapp kubectl get pod webapp -o yaml ``` Checking where pod is located: ```shell kubectl get pods -o wide ``` ## ReplicaSets * A controller * Lets is run multiple pods for HA * Enforces number of pods * Also used for load scaling * Controller with balance pods across multiple nodes ReplicaSet replaces depreciated Replication Controller Depreciated **Replication Controller**: Create: ```yaml apiVersion: v1 kind: ReplicationController metadata: name: myapp-rc labels: app: myapp type: front-end spec: template: metadata: name: myapp-pod labels: app: myapp spec: containers: - name: nginx-container image: nginx replicas: 3 ``` So spec.template is children ```shell kubectl create -f $FILE.yml kubectl get replicationcontroller ``` **ReplicaSet:** selector is main difference, its required and takes children labels Create: ```yaml apiVersion: apps/v1 kind: ReplicaSet metadata: name: myapp-replicaset labels: app: myapp type: front-end spec: template: metadata: name: myapp-pod labels: app: myapp spec: containers: - name: nginx-container image: nginx replicas: 3 selector: matchLabels: type: front-end ``` ```shell kubectl create -f $FILE.yml kubectl get replicaset ``` ReplicaSet monitors and keeps pods up based on labels and selectors. ## Scaling Several options for scaling. ```shell kubectl replace -f $FILE.yml # With updated replicas kubectl scale --replicas=6 -f $DEFINITION.yml kubectl scale --replicas=6 replicaset myapp-replicaset # By name ``` ## Deployments Used for rolling updates and scaling. Deployments are a superset of other objects like ReplicaSet Compared to ReplicaSet only `kind: Deployment` needs changing: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: myapp-replicaset labels: app: myapp type: front-end spec: template: metadata: name: myapp-pod labels: app: myapp spec: containers: - name: nginx-container image: nginx replicas: 3 selector: matchLabels: type: front-end ``` ```shell kubectl create -f $FILE.yml kubectl get deployments kubectl get all # show all (pods, replicasets, deployments) ``` ## Creating YAML in CKA Using the `kubectl run` command can help in generating a YAML template. And sometimes, you can even get away with just the `kubectl run` command without having to create a YAML file at all. For example, if you were asked to create a pod or deployment with a specific name and image, you can simply run the `kubectl run` command. * [Conventions](https://kubernetes.io/docs/reference/kubectl/conventions/) Create an NGINX Pod ```shell kubectl run nginx --image=nginx ``` Generate POD Manifest YAML file (`-o yaml`). Don’t create it(–dry-run) ```shell kubectl run nginx --image=nginx --dry-run=client -o yaml ``` Create a deployment ```shell kubectl create deployment --image=nginx nginx ``` Generate Deployment YAML file (`-o yaml`). Don’t create it(`--dry-run`) ```shell kubectl create deployment --image=nginx nginx --dry-run=client -o yaml ``` Generate Deployment YAML file (`-o yaml`). Don’t create it (`--dry-run`) and save it to a file. ```shell kubectl create deployment --image=nginx nginx --dry-run=client -o yaml > nginx-deployment.yaml ``` Make necessary changes to the file (for example, adding more replicas) and then create the deployment. ```shell kubectl create -f nginx-deployment.yaml ``` OR In k8s version 1.19+, we can specify the –replicas option to create a deployment with 4 replicas. ```shell kubectl create deployment --image=nginx nginx --replicas=4 --dry-run=client -o yaml > nginx-deployment.yaml ``` ## Services Help with establishing connections. Pods are on private net, we need to expose services within them Service is an object that: * NodePort: forwards ports from node to pod * ClusterIP: Creates virtual IP for internal communication * LoadBalance: Distributes traffic ### NodePort * TargetPort: pod port * Port: port for Service to Pod * NodePort: port on Node ![NodePort](/files/By8Ge0CF8qiVrk8P5H4S) ```yaml apiVersion: v1 kind: Service metadata: name: myapp-service spec: type: NodePort ports: - targetPort: 80 # Required port: 80 # If unset will be same as port nodePort: 30008 # If unset will be random 30000-32767 selector: # Matching Pod labels app: myapp type: front-end ``` ```shell kubectl create -f $FILE.yml kubectl create service nodeport redis-service --dry-run=client --tcp=6379:6379 -o yaml kubectl get services curl https://$NODE_IP:30008 ``` For multiple Pods the service matches all matching labels and load balances When Pods are on different nodes the service spans them all, and you can use any node IP. ### ClusterIP When many clusters of Pods need to talk between various services we use ClusterIP: ![ClusterIP](/files/aputTsoq5r7xDAAH3Mmh) ```yaml apiVersion: v1 kind: Service metadata: name: back-end spec: type: ClusterIP ports: - targetPort: 80 # Port where backend exposed port: 80 # Port where service exposed selector: # Matching Pod labels app: myapp type: back-end ``` ```shell kubectl create -f $FILE.yml kubectl create service clusterip redis-service --dry-run=client --tcp=6379:6379 -o yaml kubectl get services ``` ## LoadBalancer Lets us use ONE ip for app. Uses native cloud provider LB. If unsupported reverts to NodePort. Same config as NodePort. ```yaml apiVersion: v1 kind: Service metadata: name: myapp-service spec: type: LoadBalancer ports: - targetPort: 80 # Required port: 80 # If unset will be same as port nodePort: 30008 # If unset will be random 30000-32767 selector: # Matching Pod labels app: myapp type: front-end ``` ## Namespaces Allows grouping resources. Default namespace is `default`. kubernetes shas a few for the system: * kube-system * kube-public Can set quotas per namespace. If you are connecting to external namespaces outside of your own you need to append the namespace: EG: For `db-service` in `dev` namespace: * `db-service.dev.svc.cluster.local` This DNS entry is added by default. ![Namespaces-DNS](/files/JmY5mPqra4TDpSZQMIyp) ```shell kubectl get pods --namespace=$NAMESPACE kubectl create -f --namespace=$NAMESPACE ``` Can put namespace in `metadata` ```yaml apiVersion: v1 kind: Pod metadata: namespace: dev name: myapp-pod labels: app: myapp spec: containers: - name: nginx-container image: nginx ``` To create a Namespace: ```shell kubectl create namespace $NAMESPACE ``` or ```yaml apiVersion: v1 kind: Namespace metadata: name: dev ``` We can switch namespace: ```shell kubectl config set-context $(kubectl config current-context) --namespace=$NAMESPACE ``` All namespaces: ```shell kubectl get pods --all-namespaces ``` For quotas: ```yaml apiVersion: v1 kind: ResourceQuota metadata: namespace: dev name: compute-quota spec: hard: pods: "10" requests.cpu: "4" requests.memory: 5Gi limits.cpu: "10" limits.memory: 10Gi ``` ## Imperative vs Declerative Imperative: ```shell kubectl run --image=nginx nginx kubectl create deployment --image=nginx nginx kubectl expose deployment nginx --port 80 kubectl edit deployment nginx kubectl edit deployment nginx --replicas=5 kubectl set image deployment nginx nginx=nginx:1.18 kubectl create -f $FILE.yml kubectl replace -f $FILE.yml kubectl delete -f $FILE.yml ``` Declerative: Use kubectl and describe state: ```shell kubectl apply -f FILE.yml ``` `apply` modifies state to match file. Declerative is best practice. For apply: * Edit original yaml * `kubectl apply -f $TARGET` * Can `kubectl apply -f $DIRECTORY` ## Apply 3 States: * Local file * Last applied * Live object * If object doesnt exist, apply creates * The initial state is stored * On next change we compare differences with "last applied" * Intelligently updates live configuration Dont mix apply and imperative. # Scheduling ## Manual Scheduling To manually schedule at creation - `nodeName`: ```yaml apiVersion: v1 kind: Pod metadata: name: nginx labels: name: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 8080 nodeName: node02 ``` Or create a binding object: ```yaml apiVersion: v1 kind: Binding metadata: name: nginx target: apiVersion: v1 kind: Node name: node02 ``` ## Labels and Selectors Filter via selectors Labels in metadata Can use: ```shell kubectl get pods --selector app=nginx ``` ## Taints and Tolerations * Taint: Tell pod "dont schedule here" * We taint nodes * Toleration: "You can schedule here even with taint" * Tolerate taint=xyz ```shell kubectl taint nodes kubectl taint nodes key=value:taint-effect ``` Taint effect defines what would happen to the pods if they do not tolerate the taint. * NoSchedule * PreferNoSchedule: Best effort * NoExecute: Happens to nodes on existing nodes * Once taint takes effect, existing node evicts pod unless meets NoEvict ```yaml apiVersion: v1 kind: Pod metadata: name: myapp-pod spec: containers: - name: nginx-container image: nginx tolerations: - key: "app" operator: "Equal" value: "blue" effect: "NoSchedule" ``` Master nodes have NoSchedule ## Node Selectors We can add `nodeSelectors` to a pod, which will help with scheduling: ```yaml apiVersion: v1 kind: Pod metadata: name: myapp-pod spec: containers: - name: data-processor image: data-processor nodeSelector: size: Large ``` To label nodes: ```shell kubectl label nodes = kubectl label nodes node-1 size=Large ``` ## Node Affinity ```yaml apiVersion: v1 kind: Pod metadata: name: myapp-pod spec: containers: - name: data-processor image: data-processor affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: size operator: In values: - Large - Medium ``` Other options: ```yaml nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: size operator: NotIn values: - Small ``` ```yaml nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: size operator: Exists ``` Available * `requiredDuringSchedulingIgnoredDuringExecution` * `preferredDuringSchedulingIgnoredDuringExecution` ## Resource Requirements * Can specify requirements with `resource.requests` * Can specify limits with `resource.limits` ```yaml apiVersion: v1 kind: Pod metadata: name: simple-webapp-color labels: name: simple-webapp-color spec: containers: - name: simple-webapp-color image: simple-webapp-color ports: - containerPort: 8080 resources: requests: memory: "1Gi" cpu: "1" limits: memory: "2Gi" cpu: "2" ``` Defaults is no limit, no requirements. * If no request, but we have limit, request = limit * Should atleast set `requests` to avoid starting a pod. If pod uses too much RAM during usage, we will OOM kill. We can set defaults for a namespace with `LimitRange`: We can also set `ResourceQuota` request and limit for a namespace. You cant adjust limits on pod without deletion, you can on deployment. Deployment will re-create. ## DaemonSets Run one copy of pod on every node in cluster. Matadata very similar to `ReplicaSet`: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: monitoring-daemon labels: app: nginx spec: selector: matchLabels: app: monitoring-agent template: metadata: labels: app: monitoring-agent spec: containers: - name: monitoring-agent image: monitoring-agent ``` Under the hood uses affinity. ## Static Pods Kubelet can read from `/etc/kubernetes/manifests` instead of talking to `kube-api` We can only use pods, no complex deployments. Check `--pod-manifest-path` or (`--kubeconfig` for `staticPodPath:`) We can view these by listing containers: * `crictl ps` * `nerdctl ps` * `docker ps` Cluster is aware of static pods, but we can't edit them outside manifests. Kubeadm sets up some services this way. ## Multiple Schedulers We can add custom schedulers. ```yaml apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration profiles: - schedulerName: my-scheduler ``` If using process, name should match systemd service which points at `yaml` config with `--config` If scheduler in pod, simply deploy as normal pod/deployment: [Configure Multiple Schedulers](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-multiple-schedulers/) On pod creation, direct pod to use custom scheduler: ```yaml apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - image: nginx name: nginx schedulerName: my-custom-scheduler ``` ```shell kubectl get events -o wide kubectl logs my-custom-scheduler -n kube-system ``` ## Scheduler Profiles Scheduling has various stages, each can have associated plugins: * Scheduling queue * Filtering * Scoring * Binding To customize plugins for each phase we have extension points We can set multiple profiles for one scheduler binary: ```yaml apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration profiles: - schedulerName: my-scheduler plugins: score: disabled: [] enabled: [] ``` # Logging ## Monitoring Can have one metrics server per cluster (built-in in memory) ```shell kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml ``` * Will poll and record metrics ```shell kubectl top node kubectl top pod ``` or Use alternatives like prometheus etc ## Application Logs ``` kubectl logs -f PODNAME [ CONTAINERNAME ] ``` # Application Lifecycle Management ## Rolling updates and Rollbacks Deployments trigger "rollouts", marking new "revisions" ```shell kubectl rollout status deployment/myapp-deployment kubectl rollout history deployment/myapp-deployment ``` Deployments rolling cause no downtime due to rolling strategy. Modify yaml, then apply, causing new rollout and revision. Upgrades in deployments create new replicaset and remove pods from old, add to new Useful: ```shell kubectl create -f deployment-definition.yaml kubectl get deployments kubectl apply -f deployment-definition.yaml kubectl set image deployment/myapp-deployment nginx=nginx:1.9.1 kubectl rollout status deployment/myapp-deployment kubectl rollout history deployment/myapp-deployment kubectl rollout undo deployment/myapp-deployment ``` ## Commands and Arguments Override commands and arguments via: ```yaml apiVersion: v1 kind: Pod metadata: name: ubuntu-sleeper-pod spec: containers: - name: ubuntu-sleeper image: ubuntu-sleeper command: [ "sleep2.0" ] args: [ "10" ] ``` ## Environment Variables ```yaml apiVersion: v1 kind: Pod metadata: name: simple-webapp-color spec: containers: - name: simple-webapp-color image: simple-webapp-color ports: - containerPort: 8080 env: - name: APP_COLOR value: pink ``` ## Configmaps Lets is define kv pairs Imperative: ```shell kubectl create configmap CONFIG_NAME --from-literal=KEY=VALUE kubectl create configmap app-config \ --from-literal=APP_COLOR=blue \ --from-literal=APP_MODE=prod ``` File: ConfigMap: ```` APP_COLOR: blue APP_MODE: prod ```_ ```shell kubectl create configmap CONFIG_NAME --from-file=CONFIG_FILE kubectl create configmap app-config --from-file=app_config.properties ```` Declarative: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: app-config data: APP_COLOR: blue APP_MODE: prod ``` ```yaml apiVersion: v1 kind: Pod metadata: name: simple-webapp-color spec: containers: - name: simple-webapp-color image: simple-webapp-color ports: - containerPort: 8080 envFrom: - configMapRef: name: app-config ``` ```shell kubectl get configmaps kubectl get configmap CONFIG_MAP kubectl describe configmap CONFIG_MAP ``` ## Secrets Same as ConfigMap, but encoded (NOT ENCRYPTED). ```shell kubectl create secret ``` We encode as `base64` in yaml: ```shell echo -n VALUE | base64 echo -n ENOCED_VALUE | base64 --decode ``` Secret: ``` APP_COLOR: BASE64_ENCODED APP_MODE: BASE64_ENCODED ``` ```yaml apiVersion: v1 kind: Secret metadata: name: app-config data: APP_COLOR: blue APP_MODE: prod ``` ```shell kubectl get secrets kubectl get secret CONFIG_MAP kubectl describe secret generic CONFIG_MAP ``` Just like configmap use envFrom: ```yaml apiVersion: v1 kind: Pod metadata: name: simple-webapp-color spec: containers: - name: simple-webapp-color image: simple-webapp-color ports: - containerPort: 8080 envFrom: - secretKeyRef: name: app-config ``` Can use `EncryptionConfiguration` to encrypt secrets at rest (stall accessible by users with access to pods) ## Encrypting We can encrypt at rest in `etcd` We can query `etcd` with `etcdctl`: [Encrypting Confidential Data at Rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/) Check if `--encryption-provider-config` set in `kube-apiserver`: (kubeadm): ```shell less /etc/kubernetes/manifests/kube-apiserver.yaml ``` Create `EncryptionConfiguration` (see docs), and pass via `--encryption-provider-config` ## Init Containers If you only wish to run something at initialization in a multi-container pod, use an `initContainer`, they work just like regular containers but exit. ```yaml apiVersion: v1 kind: Pod metadata: name: myapp-pod labels: app: myapp spec: containers: - name: myapp-container image: busybox:1.28 command: ['sh', '-c', 'echo The app is running! && sleep 3600'] initContainers: - name: init-myservice image: busybox command: ['sh', '-c', 'git clone ;'] ``` `initContainer`s must run to completion before the other container start. They run in sequential order. # Cluster Maintenance "drain" node and move pods: ```shell kubectl drain node-1 ``` This "cordons" a node, to uncordon: ```shell kubectl uncordon node-1 ``` cordon marks unschedulable but leaves existing nodes: ```shell kubectl cordon node-1 ``` ## Cluster Upgrade Introduction Components should be somewhat in synch. kube-apiserver is main component, the controller manager and the kube scheduler should be less than or equal to the version, and be a maximum of one lower inversion. The kubelet and kube proxy should be a maximum of two versions lower than the API server and should not be greater than the version of the API server. `kubectl` should be +-1 k8s supports last 3 minor versions. Upgrades do master first (pods stay up meanwhile) Nex we do workers, can do all at once or one node at a time. Alternatively create new nodes with higher version and remove old We need to upgrade `kubeadm` first with `apt`. Then `kubelet` with `apt` Upg master: ```shell kubeadm upgrade plan apt upgrade -y kubeadm=VERSION kubectl get nodes apt upgrade -y kubelet=VERSION systemctl restart kubelet kubectl get nodes ``` Upg workers: ```shell kubectl drain NODE apt upgrade -y kubeadm=VERSION kubectl get nodes apt upgrade -y kubelet=VERSION systemctl restart kubelet kubeadm upgrade node config --kubelet-version VERSION kubectl uncordon NODE ``` [Upgrading kubeadm clusters](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) ## Backup and Restore Can save all yaml for cluster via: ```shell kubectl get all --all-namespaces -o yaml > all-deploy-services.yaml ``` Can backup `etcd` via: ```shell ETCDCTL_API=3 etcdctl snapshot save snapshot.db ``` To restore: ```shell ETCDCTL_API=3 etcdctl snapshot restore snapshot.db --data-dir=NEW_ETCD_DIR ``` [Operating etcd clusters for Kubernetes](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/) Usually etcd is a static pod, so if we want to edit, edit manifests. Look at pod: ```shell kubectl describe ETCD_POD ``` Find ip, `trusted-ca-file`, `key-file` and `cert-file`, test via: ```shell ETCDCTL_API=3 etcdctl --endpoints IP_ADDR:2379 \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ member list ``` Snapshot to `/opt/snapshot-pre-boot.db`: ```shell ETCDCTL_API=3 etcdctl --endpoints IP_ADDR:2379 \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ snapshot save /opt/snapshot-pre-boot.db ``` Restore to `/etcd-backup`: ```shell ETCDCTL_API=3 etcdctl --endpoints IP_ADDR:2379 \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --data-dir=/etcd-backup \ snapshot restore /opt/snapshot-pre-boot.db ``` We will edit static pod. And point the etcd-data hostpath to new data directory. ## Multi-Cluster List all: ```shell kubectl config get-clusters ``` Swap: ```shell kubectl config use-context CLUSTER ``` # Security ## Authentication * Users (humans) * Service Accounts (Machines) User access is via `kube-apiserver` File-based: * Static password file * Static token file Simple, but insecure Static pass file: ```csv password123,user1,u0001 password123,user2,u0002 ``` Add to `kube-apiserver` command (likely in pod): ``` --basic-auth-file=/tmp/users/user-details.csv ``` Next create `Role` and `RoleBinding` ## TLS Asymmetric Encryption: Public "lock" Private "key" Lock a resource - eg `.ssh/authorized_keys` We need a way to securely transfer keys to the server so we can "unlock" We encrypt the private key before sending using the servers public key. Then we send and server can get the key We need to "certify" server is who it says it is. We use a CA for signing with a CSR ## TLS in Kubernetes Thee kinds of certificate we will consider: * Root * Client * Server Naming conventions: Usually Public keys are `.crt` and `.pem` Private keys in `.key` and `-key.pem` On k8s all servers and clients need client or server certificates (depending on function). kube-api and etcd need server cert, rest client We need a CA for creating these certs. ## Generating Certificates If using openssl: ### Setup CA * Generate Keys ```shell openssl genrsa -out ca.key 2048 ``` * Generate CSR ```shell openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr ``` * Sign certificates ```shell openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt ``` ### Generating Client Certificates #### Admin User Certificates * Generate Keys ```shell openssl genrsa -out admin.key 2048 ``` * Generate CSR (CN just for logs) ```shell openssl req -new -key admin.key -subj "/CN=kube-admin" -out admin.csr ``` * Sign certificates ```shell openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt ``` * Certificate with admin privilages ```shell openssl req -new -key admin.key -subj "/CN=kube-admin/O=system:masters" -out admin.csr ``` Viewing certs: ```shell openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout ``` [Cert info](https://github.com/mmumshad/kubernetes-the-hard-way/tree/master/tools) ## Certificate API Rather than manually signing new certs we can use API. Use can request a cert signed. EG, jane requests: ```shell openssl genrsa -out jane.key 2048 openssl req -new -key jane.key -subj "/CN=jane" -out jane.csr ``` ```yaml apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: jane spec: groups: - system:authenticated usages: - digital signature - key encipherment - server auth request: ``` Then send to `kubectl` \`base64 encode in file: ```shell cat jane.csr | base64 -w 0 # single line kubectl create -f jane.yaml ``` ```shell kubectl get csr kubectl certificate approve jane kubectl get csr jane -o yaml echo "" |base64 --decode ``` All this is completed by `controller-manager` ## KubeConfig We need to use cert with `kubectl` on every call. Rathen than CLI, we put in KubeConfig: Default: `$HOME/.kube/config` Three sections: * Clusters * Contexts * Users Context groups Cluster and Context. EG: user: `Admin`, cluster `AWS`, context: `Admin@AWS` `current-context` is default. Current config: ```shell kubectl config view ``` ```shell kubectl config use-context user@cluster ``` We can put namespaces in context if we want. [Configure Access to Multiple Clusters](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) ## API Groups The api has multiple API groups. Core functionality is in `/api` (`/api/v1`) (secrets, pods, svc etc..) Named groups are more heirarchical (in future API changes here) Groups are shown in docs EG: `/apis/apps/v1/{deployments, replicasets, statefulsets}` Can use `curl` to see groups: ```shell curl http://localhost:6443 -k \ --key=admin.key \ --cert=admin.crt \ --cacert=ca.crt ``` Can use: ```shell kubectl proxy ``` To avoid needing to specify certs (sets up listener with auth) ## Authorization *What can I do with access?* We create accounts, then authorize certain things. Usually done via namespaces. ### Authorization types **Node Authorizer:** Used by `kubelets` User in this group by adding cert `system:node` prefix on cert **ABAC Authorizer:** Associate user with a permission: eg "view pod" Requires restarting API server after perm change. **RBAC Authorizer:** Assign perms to a role Associate users with the role. **Webhook:** Outsource to third party (eg Open Policy Agent) **AlwaysAllow:** The default. **AlwaysDeny:** Does what is says. If you specify multiple `--authorization-mode=Node,RBAC` It tries all in order then allows if no match. ## RBAC Create `kind: Role` with `apiVersion: rbac.authorization.k8s.io/v1` ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: developer rules: - apiGroups: [""] # "" indicates the core API group resources: ["pods"] verbs: ["get", "list", "update", "delete", "create"] - apiGroups: [""] resources: ["ConfigMap"] verbs: ["create"] ``` Now create `kind: RoleBinding` and link user to role ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: devuser-developer-binding subjects: - kind: User name: dev-user # "name" is case sensitive apiGroup: rbac.authorization.k8s.io roleRef: kind: Role name: developer apiGroup: rbac.authorization.k8s.io ``` ```shell kubectl get roles kubectl get rolebindings kubectl describe role developer kubectl describe rolebinding devuser-developer-binding ``` Can check `can-i`: ```shell kubectl auth can-i create deployments kubectl auth can-i delete nodes kubectl auth can-i create deployments --as dev-user kubectl auth can-i create pods --as dev-user kubectl auth can-i create pods --as dev-user --namespace test ``` When in a namespace we can further restrict via `resourceNames` * [Using RBAC Authorization](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) ## Cluster Roles Some resources are cluster-wide We can do cluster-wide roles via `ClusterRole`, `ClusterRoleBinding` Very similar to `Role` Can view api groups via: ```shell kubectl api-resources ``` ## Service Accounts Used for machine access (eg an application) to `kube-api` ```shell kubectl create serviceaccount ``` Creates a token used to connect. Token is a `secret`, use: ```shell kubectl describe secret ``` Token can be used as a "Bearer" eg curl. Can use RBAC with the service account. If hosting an app on K8S we can expose the secret as a volume. By default the default namespace secret is exposed (only basic k8s access). `>=` v1.22 `TokenRequestAPI` creates token with expiry, bound to a pod. See projected volume mount. `>=` v1.24 service accounts dont create tokens by default. And have expiry. To use legacy (no binding, no expiry) use [ServiceAccount token Secrets](https://kubernetes.io/docs/concepts/configuration/secret/#serviceaccount-token-secrets) ## Image Security In `image` names, it has implicit account `library` on dockerhub if not specified. `image: REGISTRY/USER/NAME` We can use a private registry too. We need to pass auth to CRI. ```shell kubectl create secret docker-registry regcred \ --docker-server=private-registry.io \ --docker-username=registry-user \ --docker-password=registry-password \ --docker-email=registry-user@org.com ``` ```yaml apiVersion: v1 kind: Pod metadata: name: nginx-pod spec: containers: - name: nginx image: private-registry.io/apps/internal-app imagePullSecrets: - name: regcred ``` ## Security Contexts We can set capabilities at container granularity. ```yaml apiVersion: v1 kind: Pod metadata: name: web-pod spec: containers: - name: ubuntu image: ubuntu command: ["sleep", "3600"] securityContext: runAsUser: 1000 capabilities: add: ["MAC_ADMIN"] ``` securitycontext can be at pod granularity, but no capabilities. * [Configure a Security Context for a Pod or Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) ## Network Policies Ingress: inbound Egress: outbound By defaults k8s has `AllAllow` on communication between pods. We implement a network policy to restrict traffic. We use labels and selectors for policies. ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: db-policy spec: podSelector: matchLabels: role: db policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: role: api-pod ports: - protocol: TCP port: 3306 ``` Not all solutions support policies. If there are multiple items in `from:`, matching 1 allows. # Storage ## Volumes Allows for persistent data. Use `spec.containers[*].volumeMounts` and `spec.volumes`: [Volumes](https://kubernetes.io/docs/concepts/storage/volumes/) With basic `hostPath`, data is stored directly on EACH node, not shared. Various volume types exist we can use. ## PersistentVolumes Lets us store data centrally in a pool. We then claim the data with a persistent volume claim (PVC) ```yaml kind: PersistentVolume apiVersion: v1 metadata: name: pv-vol1 spec: accessModes: [ "ReadWriteOnce" ] capacity: storage: 1Gi hostPath: path: /tmp/data ``` [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) ## Persistent Volume Claims Looks for matching claims. We can select one ourselves with labels and selectors. PV and PVC are one to one If one cannot be matched with the required resources the PVC will stay in pending state ```yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: myclaim spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi ``` Once PVC deleted we can choose to automatically delete the underlying PV, retain it, or recycle it. [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#claims-as-volumes%5C) ## Storage Classes Provisioner dynamically provisions when we need storage. ```yaml sc-definition.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: google-storage provisioner: kubernetes.io/gce-pd ``` Creates a pv for us automatically when we create a claim if we associate it: ```yaml pvc-definition.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: myclaim spec: accessModes: [ "ReadWriteOnce" ] storageClassName: google-storage resources: requests: storage: 500Mi ``` * [Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/) # Networking ## Routing, Switching, Gateways To find gateway: ```shell route # or ip route list ``` To add entries into the routing table. Where 2nd ip is gateway ```shell ip route add 192.168.1.0/24 via 192.168.2.1 ``` If forwarding between machines required for communication without a router: `/proc/sys/net/ipv4/ip_forward` must be enabled: ```shell echo 1 > /proc/sys/net/ipv4/ip_forward ``` To persist: ``` # /etc/sysctl.conf net.ipv4.ip_forward=1 ``` ## DNS In `/etc/resolv.conf` set DNS server: ``` nameserver 192.168.1.100 ``` We can set order for `/etc/hosts` or DNS server in: `/etc/nsswitch.conf`: ``` hosts: files dns ``` We can use `nslookup`, `dig` to query DNS servers: ``` nslookup google.ca dig google.ca ``` ## Network Namespaces Lets us have isolated routing and arp tables along with virtual interfaces. ```shell ip netns add b ip netns list ``` Run in ns: ```shell ip netns exec red ip link # or ip -n res link ``` To connect namespaces we can use a virtual pair (or pipe): To create a virtual cable ```shell ip link add veth-red type veth peer name veth-blue ``` To attach with the network namespaces ```shell ip link set veth-red netns red ip link set veth-blue netns blue ``` To add an IP address ```shell ip -n red addr add 192.168.15.1/24 dev veth-red ip -n blue addr add 192.168.15.2/24 dev veth-blue ``` To set up `ns` interfaces ```shell ip -n red link set veth-red up ip -n blue link set veth-blue up ``` Check the connectivity ```shell ip netns exec red ping 192.168.15.2 ``` When we have many NS, we create a switch (bridge) *** Putting this all together we can have the bridge reach the external network by talking to our host as the Gateway, and have connections go back in to the private network by implementing NAT on our host via: ```shell iptables -t nat -A PREROUTING --dport 80 --to-destination 192.168.15.2:80 -j DNAT ``` ## Pod Networking The rules of kubernetes pod networking are that: * every pod should have an IP address * every pod should be able to communicate with every other pod in the same node * every pod should be able to communicate with every other pod on other nodes without NAT We create a bridge on each node for the containers. Each bridge has a private subnet. To allow cross-node communications we add routes between nodes or use a router. See `kube-controller-manager` `--cluster-cidr=` for pod range. ## CNI in Kubernetes We specify CNI plugin on container runtime in `/etc/cni/net.d`, bins in `/opt/cni/bin` Kubernetes networking Solutions will typically install agents on every node (DaemonSet) along with bridges and then deal with peer-to-peer communication ## IP Address Managements (IPAM) Who assigns IPs to containers. CNI plugin manages the IP management. ## Service Networking Pods communicate via services and each gets a cluster-wide IP. kube-proxy watches for service creation, and creates one. This is done by each node setting up forwarding rules on each node. proxy-mode defines how forwarding rules are created on kube-proxy. `service-cluster-ip-range` defines ip range for services. ```shell ps -aux | grep kube-apiserver ``` ``` --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub -- service-cluster-ip-range=10.96.0.0/12 ``` ## DNS in Kubernetes Whenever we create services they get a DNS entry so any pod can access. If in same namespace, can use just service name, eg 'service' In different namespace add namespace suffix, eg 'service.default'. Full domain is 'service.default.svc' or FQDN 'service.default.svc.cluster.local' By default pods do not get entry, but we can enable DNS enties for them, thry get entry: 'IP-WITH-DASHES.namespace.pod' eg '10-244-2-5.default.pod' ## CoreDNS Config in `/etc/coredns/Corefile` ``` .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload } ``` ```shell kubectl get configmap -n kube-system ``` Kubelet configures DNS server for pods by setting `nameserver` in `/etc/resolv.conf` `resolv.conf` also contains a search query to allow PARTIAL FQDN: ``` search default.svc.cluster.local svc.cluster.local cluster.local ``` ## Ingress Controllers Native internal loadbalancing. Not deployed by default. GCE, nginx maintained by k8s (currently) Create an ingress service account, and service. Create a deployment with: ```yaml kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration ``` ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: ingress-controller spec: replicas: 1 selector: matchLabels: name: nginx-ingress template: metadata: labels: name: nginx-ingress spec: serviceAccountName: ingress-serviceaccount containers: - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 ``` To configure ingress, create an ingress-resource: ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-wear spec: backend: serviceName: wear-service servicePort: 80 ``` ```shell kubectl get ingress ``` To define rules for paths: ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-wear-watch spec: rules: - http: paths: - path: /wear backend: serviceName: wear-service servicePort: 80 - path: /watch backend: serviceName: watch-service servicePort: 80 ``` For domain name rules: ```yaml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-wear-watch spec: rules: - host: wear.my-online-store.com http: paths: - backend: serviceName: wear-service servicePort: 80 - host: watch.my-online-store.com http: paths: - backend: serviceName: watch-service servicePort: 80 ``` # Install Kubernetes with kubeadm * [Installing kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) * [Creating a cluster with kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) # JSON PATH Query language for json ## Dictionaries `$` is root element. ```json { "car": { "price": "$25,000" } } ``` Query price via `$.car.price`: ```json [ "$25,000" ] ``` ## Lists ```json [ 1, 2, 3 ] ``` Query via `$[INDEX]`: `$[0]` is `[1]` can do multiple: `[0,2]` is `[1,3]` Can slice via `[START:END_EXCLUSIVE]` Can step via `[START:END_EXCLUSIVE:STEP]` Can get last via `[-1]` or `[-1:]` ## Criteria More complex queries ```json [ 1, 2, 3, 4 ] ``` `?()` for query. `@` each item in list So, get items `> 2`: ``` $[ $( @ > 2) ]` ``` Result: `[3,4]` Can use `==`, `!=`, `in`, `nin` Can check strings via `?(@.SOME_KEY == "val")` ## Wildcards We can also use wildcards, eg all matches: ``` $[*].price ``` Or with Dictionaries: ``` $*.car ``` ## JSON PATH in Kubernetes ```shell kubectl get pods -o=jsonpath='{ .item[0].spec }' ``` `$` is added if not there. We can combine queries for multiple results: ```shell kubectl get pods -o=jsonpath='{ .item[0].spec }{ .items[*].metadata }' ``` We can prettify via `{ "\n" }` between queries. We can loop with a `{range .items[*]}{ ... }{ end }`, and access internal elements inside. Can also create custom columns via `-o=custom-columns=COL_NAME:JSON_PATH` # Software Design Contains information on the design of software and software algorithms. # SQL Notes on queries & optimization. Mostly MySQL focused. # Query Optimization Resources: * [SCaLE - Query Optimization 101 in MySQL](https://www.youtube.com/watch?v=3pu7hoR1HbU) ## Indexes Always use indexes. They allow faster look up of columns. The lookups can be done with a b-tree rather than a full scan leading to O(logn) instead of O(n). ## Primary Keys A way to uniquely identify a row. It must be unique, and there can only be one primary key per table. It can consist of multiple columns. All primary keys are indexes. ## EXPLAIN `EXPLAIN` can be used to describe the query optimization plan. [Ref](https://dev.mysql.com/doc/refman/8.0/en/explain-output.html) A few key things to look out for: `type` The type is important for determining the access. Some common ones are `ref`, specifying index lookups. `all` specifying full scan (normally bad). `range` specifying range query ### EXPLAIN FORMAT=TREE Shows better sequence of query. ### EXPLAIN ANALYZE Executes query and reports back statistics. # DSA ## Subarrays It's possible to get number of subarrays ENDING at a fixed location, starting from a non fixed location: ``` [0, 1, 2, 3] ``` Subarrays ending at `i=3`, starting `i=1` There are `1,2,3`, `2,3`, `3` - 3 total. Compute via `end - start + 1` # Trees Time: * Usually `O(n)` (visit each node) * `O(n*k)` where k is work at each node Space: * `O(n)` - Straight line, all on stack * `O(longn)` - balanced tree Depth: `O(logn)` ## DFS Pre-order: Operate on node on way down. (Logic before - pre) ```python def preorder_dfs(node): if not node: return print(node.val) preorder_dfs(node.left) preorder_dfs(node.right) return ``` In an increasing tree (root < child) this gives us increasing order. In-order: Order on the way up. (logic in middle - in) In BST translates to in order. Do left, then print val, then right. ```python def inorder_dfs(node): if not node: return inorder_dfs(node.left) print(node.val) inorder_dfs(node.right) return ``` Post-order: Go to leaves before anything else. (logic after - post) ```python def postorder_dfs(node): if not node: return postorder_dfs(node.left) postorder_dfs(node.right) print(node.val) return ``` ## BFS `O(V+E)` (balanced tree) The final level in a perfect binary tree has n/2 nodes, so BFS uses `O(n)` space To traverse levels, use: ```python def largestValues(self, root: Optional[TreeNode]) -> List[int]: if root is None: return [] q = deque([root]) lrg = [] while q: lvl = len(q) for _ in range(lvl): # Traverses level n = q.popleft() if n.left is not None: q.append(n.left) if n.right is not None: q.append(n.right) return lrg ``` ## Trie A trie (pronounced “try”) is a tree-based data structure that stores strings efficiently by sharing common prefixes. Also called a prefix tree, a trie enables fast string search, insertion, and deletion operations in O(L) time, where L is the string length. ```python class TrieNode: def __init__(self): self.children = {} self.is_end_of_word = False class Trie: def __init__(self): self.root = TrieNode() def insert(self, word): current = self.root for char in word: if char not in current.children: # For each char, go down tree inserting if not present current.children[char] = TrieNode() # Add each char as a child of current current = current.children[char] current.is_end_of_word = True def search(self, word): current = self.root for char in word: if char not in current.children: return False current = current.children[char] return current.is_end_of_word # Example usage trie = Trie() trie.insert("GLOBAL") print(trie.search("GLOBE")) # False print(trie.search("GLOBES")) # False ``` # Graphs ## Shortest Path for Weighted Graph ### Dijkstra’s `O((V + E) log V)` with a binary heap. (Only works with pos weights) [Video](https://youtu.be/EFg3u_E6eHU?si=KAFs7dVMTXp919W3) Basic idea: * Determine shortest path from src to dest * To do this, determine shortest path to EVERY node along the way Approach: * Mark every node as distance=inf except src (0) Main steps after: * In map, update distances to neighbors (if shorter), mark as visited * Next, travel to next shortest node away (use prio queue). Mark visited. * Repeat Use a visited set and skip in pq if already visited EG: ``` [A] / \ 1/ \10 / \ [B] ----1-- [C] \ / 6\ /4 \ / [D] ``` ```python import heapq def dijkstra(graph, start): # graph is in the form: {node: [(neighbor, weight), ...], ...} distances = {node: float('inf') for node in graph} distances[start] = 0 visited = set() # Priority queue: (distance_from_start, node) pq = [(0, start)] while pq: current_distance, current_node = heapq.heappop(pq) # Skip if already visited if current_node in visited: continue visited.add(current_node) # Relax edges for neighbor, weight in graph[current_node]: distance = current_distance + weight if distance < distances[neighbor]: distances[neighbor] = distance heapq.heappush(pq, (distance, neighbor)) return distances # Example graph graph = { 'A': [('B', 1), ('C', 10)], 'B': [('C', 1), ('D', 6)], 'C': [('D', 4)], 'D': [] } print(dijkstra(graph, 'A')) ``` ### Bellman-Ford Algorithm Use if we have negative path weights. DP strategy. *** Intuition: Bellman–Ford finds the shortest path from a source node by: Trying all edges again and again (up to |V|−1 times), and updating a node’s distance whenever we find a cheaper way to get there through another node. It's like saying: “If I already know the shortest way to get to A, and there’s an edge from A → B, then maybe I now also know a better way to get to B.” Because the longest possible simple path in a graph with V nodes has V−1 edges — so if we relax all edges V−1 times, we’re guaranteed to propagate all minimal costs forward. *** [Bellman Ford](https://youtu.be/FtN3BYH2Zes?si=ady6gKsfmVi5RB88) `O(V * E)` Theorem 1: In a "graph with no negative-weight cycles? with N vertices, the shortest path between any two vertices has at most N-1 edges. A negative weight cycle is one that results in a negative path after the cycle is completed. For example, a positive weight cycle could be ``` A - 1 -> C - 2 -> B - -1 -> A ``` ![Neg weights](/files/oAdvgY65wn9ETaHdeM4O) The cost from A to B is +2, going back to A we subtract one resulting in a weight greater than zero (2). The reason the shortest path has at most N-1 edges is that because there are no negative weight cycles, starting over from the original cycle we'll just increase the cost or make it the same. So, basic idea - "relax" edges N-1 times (N is num vertices). Relaxation refers to trying a connection and seeing if it reduces the cost to get to that vertex. If it does we reduce it accordingly and that is relaxation Initially mark the cost to get to a vertex as Infinity for all vertex Loop over all edges in any order. For an edge between vertex u and v. ``` if dist[u] + cost(u,v) < dist[v]: dist[v] = dist[u] + cost(u,v) ``` Full example: ```python def bellman_ford(n, edges, source): INF = float('inf') dist = [INF] * n dist[source] = 0 # Relax all edges (V - 1) times for _ in range(n - 1): for u, v, w in edges: if dist[u] != INF and dist[u] + w < dist[v]: dist[v] = dist[u] + w # Optional: Detect negative-weight cycle for u, v, w in edges: if dist[u] != INF and dist[u] + w < dist[v]: raise ValueError("Graph contains a negative-weight cycle") return dist # Graph: # A → B (4) # A → C (2) # C → B (1) # B → D (2) # C → D (5) # Convert node names to indices: A=0, B=1, C=2, D=3 edges = [ (0, 1, 4), # A → B (0, 2, 2), # A → C (2, 1, 1), # C → B (1, 3, 2), # B → D (2, 3, 5), # C → D ] n = 4 source = 0 # A distances = bellman_ford(n, edges, source) print("Shortest distances from A:") print(f"A: {distances[0]}") print(f"B: {distances[1]}") print(f"C: {distances[2]}") print(f"D: {distances[3]}") ``` Note: We can limit the maximum number of edges traversed to see the shortest path with an additional restriction of how many edges we can traverse. For example if we wanted to traverse a maximum of K edges we would use that instead of N-1 If we do this we must use a temp variable! This is because otherwise we update additional times in the cycle: ```python def bellman_ford(n, edges, source): INF = float('inf') dist = [INF] * n dist[source] = 0 # Relax all edges (V - 1) times for _ in range(n - 1): curr = dist[:] for u, v, w in edges: # dist holds the best distances found using ≤ i edges. # curr should hold the best distances using ≤ i+1 edges. if dist[u] != INF and dist[u] + w < curr[v]: # Keep start location same (old), use new curr curr[v] = dist[u] + w dist = curr # Optional: Detect negative-weight cycle for u, v, w in edges: if dist[u] != INF and dist[u] + w < dist[v]: raise ValueError("Graph contains a negative-weight cycle") return dist ``` ## Topological Sort [Video (Kahn's)](https://www.youtube.com/watch?v=cIBFEhD77b4) [Leetcode Explore Card Kahn's Algorithm](https://leetcode.com/explore/learn/card/graph/623/kahns-algorithm-for-topological-sorting/3886/) Lets us model dependencies with a graph. `O(V+E)` Gives us an order we can traverse to path through graph from most dependant to least. ![](/files/KYO2JlUtB8h1DVTG9a51) There can be multiple orderings. ### Kahn's Algorithm In-degree for node represents how many must come before If a node’s in-degree is 0, it means no prerequisites Repeatedly remove nodes without dependencies from the graph and add them to the topological ordering. As we remove them from the graph, we removed their outgoing edges, and new nodes without dependencies become free. We repeat the process until we have looked at every node or we have found a cycle Use a queue to maintain zero dependencies nodes. [Example](https://leetcode.com/problems/course-schedule/description/) ```python from collections import defaultdict, deque class Solution: def canFinish(self, numCourses: int, prerequisites: List[List[int]]) -> bool: if not prerequisites: return True adj = defaultdict(list) in_deg = defaultdict(int) # Create adjacency list and in degree map for parent, child in prerequisites: adj[child].append(parent) in_deg[parent] += 1 if parent not in in_deg: in_deg[parent] = 0 # Account for ALL courses, even if not in prereq for v in range(0, numCourses): if v not in in_deg: in_deg[v] = 0 # Create queue for zero degree items q = deque() for k, v in in_deg.items(): if v == 0: q.append(k) # ordering order = [] # iterate over queue while it has items while q: node = q.popleft() order.append(node) for nei in adj[node]: in_deg[nei] -= 1 if in_deg[nei] == 0: q.append(nei) # cycle detected if len(order) != numCourses: return False return True ``` Or, a more conventional example: ```python from collections import deque, defaultdict from typing import Dict, Iterable, List, Tuple, Hashable def kahn_toposort(edges: Iterable[Tuple[Hashable, Hashable]]) -> List[Hashable]: """ Perform topological sort using Kahn's algorithm. edges: iterable of (u, v) meaning a directed edge u -> v Returns a list of nodes in topological order. Raises ValueError if a cycle is detected. """ # Build adjacency list and indegree counts adj: Dict[Hashable, List[Hashable]] = defaultdict(list) indeg: Dict[Hashable, int] = defaultdict(int) nodes = set() for u, v in edges: adj[u].append(v) indeg[v] += 1 nodes.add(u); nodes.add(v) # Include isolated nodes (if you have an external node list, merge it into 'nodes') for u in list(nodes): indeg.setdefault(u, 0) # Queue of all nodes with no incoming edges q = deque([n for n in nodes if indeg[n] == 0]) order: List[Hashable] = [] while q: u = q.popleft() order.append(u) for v in adj[u]: indeg[v] -= 1 if indeg[v] == 0: q.append(v) if len(order) != len(nodes): # Some nodes were never removed ⇒ cycle exists raise ValueError("Graph has at least one cycle; topological order does not exist.") return order # --- Example usage --- # DAG example: # A → C → D # ↘︎ B → ↗︎ edges = [ ("A", "C"), ("A", "B"), ("B", "D"), ("C", "D"), ] order = kahn_toposort(edges) print("Topological order (DAG):", order) # Possible output (one of many valid orders): # Topological order (DAG): ['A', 'B', 'C', 'D'] or ['A', 'C', 'B', 'D'] # Cycle example: 1 → 2 → 3 → 1 cyclic_edges = [(1, 2), (2, 3), (3, 1)] try: kahn_toposort(cyclic_edges) except ValueError as e: print("Cycle example:", e) # Output: # Cycle example: Graph has at least one cycle; topological order does not exist. ``` # Binary Search Classic binary search, find a number (no dupes): ```python def search(self, nums: List[int], target: int) -> int: left = 0 right = len(nums) while left < right: mid = (left + right) // 2 if nums[mid] == target: return mid if nums[mid] < target: left = mid+1 else: right = mid # Left would be the insertion point return left ``` If we have dupes, we can find insertion point. This find leftmost index if dupes. ```python def binary_search(arr, target): left = 0 right = len(arr) while left < right: mid = (left + right) // 2 if arr[mid] >= target: right = mid else: left = mid + 1 return left ``` With bisect: ```python bisect.bisect_left(a, x) ``` To find directly after: ```python def binary_search(arr, target): left = 0 right = len(arr) while left < right: mid = (left + right) // 2 if arr[mid] > target: right = mid else: left = mid + 1 return left ``` Finds insertion point to retain sorted order ```python bisect.bisect_right(a, x) ``` Similar to bisect\_left(), but returns an insertion point which comes after (to the right of) any existing entries of x in a. --- [Next Page](/llms-full.txt/1)