Skip to content

Boot server

Netsoc servers run Alpine Linux as their base OS. This is loaded over the network from the boot server and runs from RAM. Packages are installed from the internet. Configuration is downloaded over HTTP from the boot server and overlayed on the base system.

Main components

These instructions assume a working Arch Linux installation (and should be run as root unless otherwise specified).

Make sure packages are up to date with pacman -Syu (reboot if kernel was upgraded). Once all of the sections below are completed, reboot.

To get started, clone the infrastructure repo into /var/lib/infrastructure, ensuring it's owned by the unprivileged user (assumed to be netsoc) and is world-readable. This can be done by running (as netsoc):

sudo install -dm 755 -o netsoc -g netsoc /var/lib/infrastructure
git clone git@github.com:netsoc/infrastructure.git /var/lib/infrastructure

Any time a step is given to symlink a configuration file out of this repo, the provided inline configuration matches 1:1 with what is actually deployed on the current boot server!

Warning

When making changes to infrastructure repo-based config files, be sure to commit and push them! Make sure to pull new external changes that are made too. Failing to keep in sync with the upstream repo will cause the backup script to fail!

dnsmasq

Set up dnsmasq, the DNS and DHCP server

  1. Install dnsmasq
  2. Replace /etc/dnsmasq.conf with a symlink to config/dnsmasq.conf (i.e. ln -sf /var/lib/infrastructure/boot/config/dnsmasq.conf /etc/dnsmasq.conf). Current live configuration:

    # Interface for DHCP and DNS
    interface=lan
    # Bind only to the LAN interface
    bind-interfaces
    
    # Port for DNS server
    port=53
    domain=netsoc.internal
    # Append full domain to hosts from /etc/hosts
    expand-hosts
    no-resolv
    # Upstream DNS servers
    server=1.1.1.1
    server=8.8.8.8
    
    dhcp-range=10.69.10.1,10.69.20.254,12h
    # Static leases
    dhcp-host=84:8f:69:e0:d4:0f,napalm,10.69.1.1
    dhcp-host=84:8f:69:e0:d4:17,napalm-idrac,10.69.10.1
    dhcp-host=0c:9d:92:85:72:5b,spoon,10.69.1.2
    dhcp-host=00:26:b9:50:af:4f,cube,10.69.1.3
    dhcp-host=00:26:b9:50:af:51,cube-idrac,10.69.10.3
    dhcp-host=40:a8:f0:30:3a:d4,gandalf,10.69.1.4
    dhcp-host=50:65:f3:69:b4:dc,gandalf-ilo,10.69.10.4
    
    dhcp-option=option:router,10.69.0.1
    dhcp-option=option:dns-server,10.69.0.1
    
    enable-tftp
    tftp-root=/srv/tftp
    
    dhcp-boot=ipxe.efi
    
    # If a client is using BIOS, send them the BIOS variant of iPXE
    dhcp-match=set:bios,option:client-arch,0
    dhcp-boot=tag:bios,ipxe.kpxe
    
    # When a client is using iPXE (detected by DHCP option 175), we want to give
    # them the iPXE script
    dhcp-match=set:ipxe,175
    dhcp-boot=tag:ipxe,http://shoe.netsoc.internal/boot.ipxe
    

    This configuration sets up

    • A forwarding DNS server
    • DHCP server (with static leases, add a new dhcp-host line for each new server that should get the same IP)
    • DNS resolution for clients by hostname (*.netsoc.internal)
    • TFTP server for loading iPXE over PXE (and then chain loading to the boot script over HTTP)
  3. Create the TFTP directory /srv/tftp

  4. Replace /etc/hosts with a symlink to boot/config/hosts. Current live configuration:

    10.69.0.1 shoe.netsoc.internal shoe
    192.168.69.10 nintendo.netsoc.internal
    
  5. Enable dnsmasq (systemctl enable dnsmasq)

Network interfaces

  1. Install netctl
  2. Remove any existing network configuration

  3. Create a symlink to boot/config/netctl/mgmt at /etc/netctl/mgmt. Current live configuration:

    Description='Netsoc management VLAN'
    Interface=eth0
    Connection=ethernet
    IP=static
    Address="192.168.69.1/24"
    

    This sets up the mgmt interface with a static IP address. Make sure to replace eth0 with the name of the ethernet interface!

  4. Enable the mgmt config (netctl enable mgmt)

    Warning

    If the configuration ever changes, be sure to netctl re-enable it!

  5. Create a symlink to boot/config/netctl/lan at /etc/netctl/lan. Current live configuration:

    Description='VLAN 69 Netsoc LAN'
    Interface=lan
    Connection=vlan
    BindsToInterfaces=eth0
    VLANID=69
    IP=static
    Address="10.69.0.1/16"
    

    This sets up the lan interface with a static IP address. Make sure to replace eth0 with the name of the ethernet interface!

  6. Enable the lan config (netctl enable lan)

  7. Create a symlink to boot/config/netctl/wan at /etc/netctl/wan. Current live configuration:

    Description='VLAN 420 public TCD network'
    Interface=wan
    Connection=vlan
    BindsToInterfaces=eth0
    VLANID=420
    IP=static
    Address="134.226.83.42/24"
    Gateway="134.226.83.1"
    

    This sets up the wan interface with a static IP address. Make sure to replace eth0 with the name of the ethernet interface and use the desired public IP!

  8. Enable the wan config (netctl enable wan)

  9. Ensure systemd-resolved is stopped and disabled (systemctl disable --now systemd-resolved)
  10. Replace /etc/resolv.conf with a symlink to boot/config/resolv.conf. Current live configuration:

    nameserver 127.0.0.1
    domain netsoc.internal
    

nginx

  1. Install nginx
  2. Replace /etc/nginx/nginx.conf with a symlink to boot/config/nginx.conf. Current live configuration:

    user http;
    worker_processes 1;
    
    events {
      worker_connections 1024;
    }
    
    http {
      include mime.types;
      default_type application/octet-stream;
      sendfile on;
    
      server {
        listen 80 default_server;
        server_name _;
    
        location / {
          root /srv/http;
          index index.html;
          autoindex on;
        }
    
        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
          root /usr/share/nginx/html;
        }
      }
    }
    
  3. Enable nginx (systemctl enable nginx)

  4. Create the apk overlay directory /srv/http/apkovl

iPXE

iPXE is an advanced bootloader designed for use with network booting. This is used to boot Alpine over the network. The version used on Netsoc is the current revision of the submodule in boot/ipxe (built from source).

To update and build iPXE:

  1. Clone this repo and then iPXE: git submodule update --init
  2. Update to the latest version:

    git -C boot/ipxe pull
    git commit -am "Update iPXE version"
    
  3. Build the latest EFI binary: make -C boot/ipxe/src -j$(nproc) bin-x86_64-efi/ipxe.efi bin/unionly.kpxe

  4. Copy boot/ipxe/src/bin-x86_64-efi/ipxe.efi (for UEFI boot) and boot/ipxe/src/bin/undionly.kpxe (for BIOS) to the boot server (/srv/tftp/ipxe.efi, /srv/tftp/ipxe.kpxe)
  5. Create a symlink to boot/config/boot.ipxe at /srv/http/boot.ipxe (the boot script). Current live configuration:

    #!ipxe
    # Based on https://boot.alpinelinux.org/boot.ipxe
    
    set mirror http://dl-cdn.alpinelinux.org/alpine
    set branch v3.13
    set version 3.13.2
    set flavor lts
    set arch x86_64
    
    set console tty0
    set cmdline modules=loop,squashfs BOOTIF=01-${net0/mac:hexhyp} ip=dhcp apkovl=http://shoe.netsoc.internal/apkovl/{MAC}.tar.gz ssh_key=http://shoe.netsoc.internal/netsoc.pub
    set default_cmdline default
    set title Netsoc network boot
    iseq ${manufacturer} QEMU && set flavor virt && set console ttyS0 ||
    
    :menu
    set space:hex 20:20
    set space ${space:string}
    menu ${title}
    item --gap Boot options
    item flavor ${space} Kernel flavor [ ${flavor} ]
    item console ${space} Set console [ ${console} ]
    item cmdline ${space} Linux cmdline [ ${default_cmdline} ]
    item --gap Booting
    item --default boot ${space} Boot with above settings
    item --gap Utilities
    item shell ${space} iPXE Shell
    item exit ${space} Exit iPXE
    item reboot ${space} Reboot system
    item poweroff ${space} Shut down system
    choose --timeout 5000 item
    goto ${item}
    
    :flavor
    menu ${title}
    item lts Linux lts
    item virt Linux virt
    choose flavor || goto shell
    goto menu
    
    :console
    menu ${title}
    item tty0 Console on tty0
    item ttyS0 Console on ttyS0
    item ttyS1 Console on ttyS1
    item ttyAMA0 Console on ttyAMA0
    item custom Enter custom console
    choose console || goto menu
    iseq ${console} custom && goto custom_console ||
    goto menu
    
    :custom_console
    clear console
    echo -n Enter console:${space} && read console
    goto menu
    
    :cmdline
    echo -n Enter extra cmdline options:${space} && read cmdline
    set default_cmdline modified
    goto menu
    
    :boot
    isset ${console} && set console console=${console} ||
    set img-url ${mirror}/${branch}/releases/${arch}/netboot-${version}
    set repo-url ${mirror}/${branch}/main
    set modloop-url ${img-url}/modloop-${flavor}
    imgfree
    kernel ${img-url}/vmlinuz-${flavor} initrd=/initramfs-${flavor} ${cmdline} alpine_repo=${repo-url} modloop=${modloop-url} ${console}
    initrd ${img-url}/initramfs-${flavor}
    boot
    goto exit
    
    :shell
    echo Type "exit" to return to menu.
    shell
    goto menu
    
    :reboot
    reboot
    
    :poweroff
    poweroff
    
    :exit
    clear menu
    exit 0
    
  6. Copy an SSH public key to /srv/http/netsoc.pub

NFS

NFS allows the booted systems to update their apkovl archives.

  1. Install nfs-utils
  2. Put /srv/http/apkovl 10.69.0.0/16(rw,sync,no_subtree_check,no_root_squash,fsid=0) into /etc/exports (any machine on the LAN will have access as root)
  3. Enable nfs-server (systemctl enable nfs-server)

Firewall (nftables)

  1. Install nftables
  2. Replace /etc/nftables.conf with a symlink to boot/config/nftables.conf. Current live configuration:

    #!/usr/bin/nft -f
    flush ruleset
    
    define lan_net = 10.69.0.0/16
    define vpn_net = 10.79.0.0/24
    
    define wireguard = 51820
    define iperf3 = 5201
    define fou = 5555
    
    table inet filter {
      chain wan-tcp {
        tcp dport ssh accept
        tcp dport https accept
        tcp dport $iperf3 accept
      }
      chain wan-udp {
        udp dport $wireguard accept
        udp dport $fou accept
        udp dport $iperf3 accept
      }
    
      chain wan {
        # ICMP & IGMP
        ip6 nexthdr icmpv6 icmpv6 type {
          destination-unreachable,
          packet-too-big,
          time-exceeded,
          parameter-problem,
          mld-listener-query,
          mld-listener-report,
          mld-listener-reduction,
          nd-router-solicit,
          nd-router-advert,
          nd-neighbor-solicit,
          nd-neighbor-advert,
          ind-neighbor-solicit,
          ind-neighbor-advert,
          mld2-listener-report,
          echo-request
        } accept
        ip protocol icmp icmp type {
          destination-unreachable,
          router-solicitation,
          router-advertisement,
          time-exceeded,
          parameter-problem,
          echo-request
        } accept
        ip protocol igmp accept
    
        # separate chains for TCP / UDP
        ip protocol tcp tcp flags & (fin|syn|rst|ack) == syn ct state new jump wan-tcp
        ip protocol udp ct state new jump wan-udp
      }
    
      chain filter-port-forwards {
        # example
        #ip daddr 10.69.1.1 tcp dport { http, https } accept
      }
    
      chain input {
        type filter hook input priority 0; policy drop;
    
        # established/related connections
        ct state established,related accept
    
        # invalid connections
        ct state invalid drop
    
        # allow all from loopback / lan
        iif lo accept
        iifname { eth0, lan, vpn, tmpwan } accept
    
        iifname { wan, forward } jump wan
      }
      chain forward {
        type filter hook forward priority 0; policy drop;
    
        # see comment on same rule in output chain
        oifname wan tcp flags { syn, rst } tcp option maxseg size set 1000
    
        # lan can go anywhere
        iifname { eth0, lan, vpn, tmpwan } accept
        # allow specific tmpwan ips
        ip daddr 192.168.76.10-192.168.76.17 accept
    
        iifname wan oifname lan ct state related,established accept
        iifname wan oifname lan jump filter-port-forwards
      }
      chain output {
        type filter hook output priority 0; policy accept;
    
        # something is weird with downstream networking in maths, clamping the mss
        # greatly reduces loss and improves tcp bandwidth
        oifname wan tcp flags { syn, rst } tcp option maxseg size set 1000
      }
    }
    
    table nat {
      chain port-forward {
        # example
        #tcp dport { http, https } dnat 10.69.1.1
      }
      chain prerouting {
        type nat hook prerouting priority 0;
    
        iifname wan jump port-forward
      }
    
      chain lan-port-forwarding {
        # example
        #ip daddr 10.69.1.1 tcp dport { http, https } snat $firewall
      }
      chain postrouting {
        type nat hook postrouting priority 100;
    
        oifname wan counter masquerade
        oifname lan ip saddr $lan_net jump lan-port-forwarding
        oifname lan ip saddr $vpn_net snat 10.69.0.1
      }
    }
    
    # vim:set ts=2 sw=2 et:
    
  3. Enable nftables (systemctl enable nftables)

  4. Write net.ipv4.ip_forward=1 into /etc/sysctl.d/forwarding.conf

WireGuard

  1. Install wireguard-tools and wireguard-dkms (you'll also need the kernel headers, e.g. linux-headers for regular Arch, linux-raspberrypi4-headers for a Raspberry Pi 4)
  2. Generate private and public key (as root): wg genkey | sudo tee /etc/wireguard/privkey | wg pubkey > /etc/wireguard/pubkey
  3. Change private key permissions chmod 600 /etc/wireguard/privkey
  4. Create /etc/wireguard/vpn.conf:

    [Interface]
    PrivateKey = theprivatekeyhere
    Address = 10.79.0.1/24
    ListenPort = 51820
    
    [Peer]
    PublicKey = theirpublickeyhere
    AllowedIPs = 10.79.0.2/32
    

    Replace the private key with the contents of /etc/wireguard/privkey! For each user, create a [Peer] section with their public key and a new IP.

  5. Create a client configuration file:

    [Interface]
    PrivateKey = theprivatekeyhere
    Address = 10.79.0.2/24
    DNS = 10.69.0.1, netsoc.internal
    
    [Peer]
    PublicKey = serverpublickeyhere
    AllowedIPs = 10.79.0.0/24, 10.69.0.0/16, 192.168.69.0/24, 134.226.0.0/16
    Endpoint = shoe.netsoc.tcd.ie:51820
    

    A private key for the client can be generated with wg genkey as before.

  6. Enable and start the WireGuard service: systemctl enable --now wg-quick@vpn

APKOVL backup

  1. Import the Netsoc PGP secret key. To back up Alpine configurations stored on the boot server, they first must be encrypted. You can transfer the PGP key from a machine which already has it by running the following:

    gpg --export-secret-keys --armor DB2E28B13D53C8DD62FE560B408F6E592A12DF74 | ssh netsoc@my.boot.server -- gpg --import
    
  2. Mark the key as trusted. Run gpg --edit DB2E28B13D53C8DD62FE560B408F6E592A12DF74. Type trust, set the level to 5 ("I trust ultimately") and accept, before quitting gpg.

  3. Install the backup service by symlinking boot/scripts/backup-apkovl.service into /etc/systemd/system/backup-apkovl.service. Current live service:

    [Unit]
    Description=APKOVL backup
    
    [Service]
    Type=oneshot
    ExecStart=/var/lib/infrastructure/boot/scripts/backup_apkovl.sh
    User=netsoc
    Group=netsoc
    
  4. Install the backup timer by symlinking boot/scripts/backup-apkovl.timer into /etc/systemd/system/backup-apkovl.timer. Current live timer:

    [Unit]
    Description=Backup APKOVL's weekly
    
    [Timer]
    OnCalendar=Wed *-*-* 07:00
    Persistent=true
    
    [Install]
    WantedBy=timers.target
    
  5. Enable and start the timer (systemctl enable --now backup-apkovl.timer)

Pi-KVM

Pi-KVM is a neat software solution adding a sort of software BMC with a Raspberry Pi 4.

Disable auditing

Add audit=0 to /boot/cmdline.txt.

pikvm pacman repo

Pi-KVM provides pre-built packages for the Raspberry Pi via their own repo.

  1. Import the Pi-KVM PGP key (run pacman-key -r 912C773ABBD1B584 && pacman-key --lsign-key 912C773ABBD1B584)
  2. Add the following to /etc/pacman.conf:

    [pikvm]
    Server = https://pikvm.org/repos/rpi4-arm
    SigLevel = Required DatabaseOptional
    

watchdog

The Linux watchdog will attempt to reset the machine if the system locks up.

  1. Install watchdog
  2. Replace /etc/watchdog.conf with:

    min-memory       = 1280
    max-load-1       = 24
    max-load-5       = 18
    max-load-15      = 12
    watchdog-device  = /dev/watchdog
    watchdog-timeout = 15
    interval         = 1
    realtime         = yes
    priority         = 1
    
  3. Enable watchdog (systemctl enable watchdog)

kvmd

kvmd is the main Pi-KVM component.

  1. Add a USB drive (or additional SD card partition) for storing virtual media images. Format the partition as ext4 and add the following to /etc/fstab:

    /dev/sda1 /var/lib/kvmd/msd ext4 nodev,nosuid,noexec,ro,errors=remount-ro,data=journal,X-kvmd.otgmsd-root=/var/lib/kvmd/msd,X-kvmd.otgmsd-user=kvmd  0 0
    

    Be sure to replace /dev/sda1 with the actual device name!

  2. Install kvmd-platform-v2-rpi4 and kvmd-webterm

    Warning

    nginx may be replaced by nginx-mainline (a dependency of kvmd). If this is the case, /etc/nginx/nginx.conf will be backed up to /etc/nginx/nginx.conf.pacsave. Be sure to move this file back to /etc/nginx/nginx.conf once the install is complete.

  3. Disable kvmd's nginx on port 80 (in /etc/kvmd/nginx/nginx.conf)

  4. Enable kvmd, kvmd-nginx, kvmd-webterm and kvmd-otg.
  5. Add tcp dport https accept to the wan-tcp chain in /etc/nftables (and reload nftables)
  6. Add the following to /boot/config.txt:

    hdmi_force_hotplug=1
    gpu_mem=16
    enable_uart=1
    dtoverlay=disable-bt
    dtoverlay=dwc2,dr_mode=peripheral
    
  7. Check the USB port for the capture card. Once plugged in, kvmd uses a udev rule to create a symlink /dev/kvmd-video -> /dev/video0. This is only done if the /dev/video0 is connected to a hardcoded USB port, however. The script /usr/bin/kvmd-udev-hdmiusb-check will perform this check. Edit the script and replace the rpi4 port with the output of the following command: sudo udevadm info -q path -n /dev/video0 | sed 's|/| |g' | awk '{ print $11 }'

Upgrading Alpine

Upgrading Alpine Linux on nodes is actually quite easy:

On each node, edit /etc/apk/repositories and replace the branch in the mirror URL's (e.g. http://uk.alpinelinux.org/alpine/v3.12/main would become http://uk.alpinelinux.org/alpine/v3.13/main in an upgrade from 3.12 to 3.13). Don't forget to lbu commit!

Following this, edit /srv/http/boot.ipxe on the boot server and replace the values of the branch and version variables with the latest release. Once that's done, simply reboot the nodes.


Last update: 2021-02-26