Skip to content

Provisioning

Netsoc servers run Alpine Linux as their base OS. This is loaded over the network from the boot server and runs from RAM. Packages are installed from the internet. Configuration is downloaded over HTTP from the boot server and overlayed on the base system.

Boot server

These instructions assume a working Arch Linux installation (and should be run as root).

Make sure packages are up to date with pacman -Syu (reboot if kernel was upgraded). Once all of the sections below are completed, reboot.

dnsmasq

Set up dnsmasq, the DNS and DHCP server

  1. Install dnsmasq
  2. Replace the contents of /etc/dnsmasq.conf with:

    # Interface for DHCP and DNS
    interface=lan
    # Bind only to the LAN interface
    bind-interfaces
    
    # Port for DNS server
    port=53
    domain=netsoc.internal
    # Append full domain to hosts from /etc/hosts
    expand-hosts
    no-resolv
    # Upstream DNS servers
    server=1.1.1.1
    server=8.8.8.8
    
    dhcp-range=10.69.10.1,10.69.20.254,12h
    # Static leases
    dhcp-host=52:54:00:7a:55:f7,napalm,10.69.1.1
    
    dhcp-option=option:router,10.69.0.1
    dhcp-option=option:dns-server,10.69.0.1
    
    enable-tftp
    tftp-root=/srv/tftp
    dhcp-boot=ipxe.efi
    
    # When a client is using iPXE (detected by DHCP option 175), we want to give
    # them the iPXE script
    dhcp-match=set:ipxe,175
    dhcp-boot=tag:ipxe,http://shoe.netsoc.internal/boot.ipxe
    

    This configuration sets up

    • A forwarding DNS server
    • DHCP server (with static leases, add a new dhcp-host line for each new server that should get the same IP)
    • DNS resolution for clients by hostname (*.netsoc.internal)
    • TFTP server for loading iPXE over PXE (and then chain loading to the boot script over HTTP)
  3. Create the TFTP directory /srv/tftp

  4. Replace /etc/hosts with:

    10.69.0.1 shoe.netsoc.internal shoe
    192.168.69.10 nintendo.netsoc.internal
    
  5. Enable dnsmasq (systemctl enable dnsmasq)

Network interfaces

  1. Install netctl
  2. Remove any existing network configuration

  3. Paste the following into /etc/netctl/lan:

    Description='VLAN 69 Netsoc LAN'
    Interface=lan
    Connection=vlan
    BindsToInterfaces=enp1s0
    VLANID=69
    IP=static
    Address="10.69.0.1/16"
    

    This sets up the lan interface with a static IP address. Make sure to replace enp1s0 with the name of the ethernet interface!

  4. Enable the lan config (netctl enable lan)

  5. Write the following into /etc/netctl/wan:

    Description='VLAN 420 public TCD network'
    Interface=wan
    Connection=vlan
    BindsToInterfaces=enp1s0
    VLANID=420
    IP=static
    Address="134.226.83.xxx/24"
    Gateway="134.226.83.1"
    

    This sets up the lan interface with a static IP address. Make sure to replace enp1s0 with the name of the ethernet interface!

  6. Enable the wan config (netctl enable wan)

  7. Ensure systemd-resolved is stopped and disabled (systemctl disable --now systemd-resolved)
  8. Replace /etc/resolv.conf with:

    nameserver 127.0.0.1
    domain netsoc.internal
    

    Warning

    Make sure /etc/resolv.conf isn't a symlink to a volatile generated file (rm it first to be safe)

nginx

  1. Install nginx
  2. Replace /etc/nginx/nginx.conf with:

    user http;
    worker_processes 1;
    
    events {
      worker_connections 1024;
    }
    
    http {
      include mime.types;
      default_type application/octet-stream;
      sendfile on;
    
      server {
        listen 80 default_server;
        server_name _;
    
        location / {
          root /srv/http;
          index index.html;
          autoindex on;
        }
    
        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
          root /usr/share/nginx/html;
        }
      }
    }
    
  3. Enable nginx (systemctl enable nginx)

  4. Create the apk overlay directory /srv/http/apkovl

iPXE

iPXE is an advanced bootloader designed for use with network booting. This is used to boot Alpine over the network. The version used on Netsoc is the current revision of the submodule in boot/ipxe (built from source).

To update and build the latest iPXE EFI binary (all our servers boot with UEFI), you should probably do this on a fast machine:

  1. Clone this repo and then iPXE: git submodule update --init
  2. Update to the latest version:
    git -C boot/ipxe pull
    git commit -am "Update iPXE version"
    
  3. Build the latest EFI binary: make -C boot/ipxe/src -j$(nproc) bin-x86_64-efi/ipxe.efi
  4. Copy boot/ipxe/src/bin-x86_64-efi/ipxe.efi to the boot server (/srv/tftp/ipxe.efi)
  5. Create the iPXE boot script:

    #!ipxe
    # Based on https://boot.alpinelinux.org/boot.ipxe
    
    set mirror http://dl-cdn.alpinelinux.org/alpine
    set branch v3.12
    set version 3.12.1
    set flavor vanilla
    set arch x86_64
    
    set console tty0
    set cmdline modules=loop,squashfsBOOTIF=01-${net0/mac:hexhyp} ip=dhcp apkovl=http://shoe.netsoc.internal/apkovl/{MAC}.tar.gz ssh_key=http://shoe.netsoc.internal/netsoc.pub
    set default_cmdline default
    set title Netsoc network boot
    iseq ${manufacturer} QEMU && set flavor virt && set console ttyS0 ||
    
    :menu
    set space:hex 20:20
    set space ${space:string}
    menu ${title}
    item --gap Boot options
    item flavor ${space} Kernel flavor [ ${flavor} ]
    item console ${space} Set console [ ${console} ]
    item cmdline ${space} Linux cmdline [ ${default_cmdline} ]
    item --gap Booting
    item --default boot ${space} Boot with above settings
    item --gap Utilities
    item shell ${space} iPXE Shell
    item exit ${space} Exit iPXE
    item reboot ${space} Reboot system
    item poweroff ${space} Shut down system
    choose --timeout 5000 item
    goto ${item}
    
    :flavor
    menu ${title}
    item lts Linux lts
    item virt Linux virt
    choose flavor || goto shell
    goto menu
    
    :console
    menu ${title}
    item tty0 Console on tty0
    item ttyS0 Console on ttyS0
    item ttyS1 Console on ttyS1
    item ttyAMA0 Console on ttyAMA0
    item custom Enter custom console
    choose console || goto menu
    iseq ${console} custom && goto custom_console ||
    goto menu
    
    :custom_console
    clear console
    echo -n Enter console:${space} && read console
    goto menu
    
    :cmdline
    echo -n Enter extra cmdline options:${space} && read cmdline
    set default_cmdline modified
    goto menu
    
    :boot
    isset ${console} && set console console=${console} ||
    set img-url ${mirror}/${branch}/releases/${arch}/netboot-${version}
    set repo-url ${mirror}/${branch}/main
    set modloop-url ${img-url}/modloop-${flavor}
    imgfree
    kernel ${img-url}/vmlinuz-${flavor} initrd=/initramfs-${flavor} ${cmdline} alpine_repo=${repo-url} modloop=${modloop-url} ${console}
    initrd ${img-url}/initramfs-${flavor}
    boot
    goto exit
    
    :shell
    echo Type "exit" to return to menu.
    shell
    goto menu
    
    :reboot
    reboot
    
    :poweroff
    poweroff
    
    :exit
    clear menu
    exit 0
    
    6. Copy an SSH public key to /srv/http/netsoc.pub

NFS

NFS allows the booted systems to update their apkovl archives.

  1. Install nfs-utils
  2. Put /srv/http/apkovl 10.69.0.0/16(rw,sync,no_subtree_check,no_root_squash,fsid=0) into /etc/exports (any machine on the LAN will have access as root)
  3. Enable nfs-server (systemctl enable nfs-server)

Firewall (nftables)

  1. Install nftables
  2. Replace the contents of /etc/nftables.conf with:

    #!/usr/bin/nft -f
    flush ruleset
    
    define lan_net = 10.69.0.0/16
    define vpn_net = 10.42.0.0/24
    
    define wireguard = 51820
    
    table inet filter {
      chain wan-tcp {
        tcp dport ssh accept
      }
      chain wan-udp {
        udp dport $wireguard accept
      }
    
      chain wan {
        # ICMP & IGMP
        ip6 nexthdr icmpv6 icmpv6 type {
          destination-unreachable,
          packet-too-big,
          time-exceeded,
          parameter-problem,
          mld-listener-query,
          mld-listener-report,
          mld-listener-reduction,
          nd-router-solicit,
          nd-router-advert,
          nd-neighbor-solicit,
          nd-neighbor-advert,
          ind-neighbor-solicit,
          ind-neighbor-advert,
          mld2-listener-report
        } accept
        ip protocol icmp icmp type {
          destination-unreachable,
          router-solicitation,
          router-advertisement,
          time-exceeded,
          parameter-problem,
          echo-request
        } accept
        ip protocol igmp accept
    
        # separate chains for TCP / UDP
        ip protocol tcp tcp flags & (fin|syn|rst|ack) == syn ct state new jump wan-tcp
        ip protocol udp ct state new jump wan-udp
      }
    
      chain filter-port-forwards {
        # example
        #ip daddr 10.69.1.1 tcp dport { http, https } accept
      }
    
      chain input {
        type filter hook input priority 0; policy drop;
    
        # established/related connections
        ct state established,related accept
    
        # invalid connections
        ct state invalid drop
    
        # allow all from loopback / lan
        iif lo accept
        iifname { eth0, lan, vpn } accept
    
        iifname wan jump wan
      }
      chain forward {
        type filter hook forward priority 0; policy drop;
    
        # lan can go anywhere
        iifname { eth0, lan, vpn } accept
    
        iifname wan oifname lan ct state related,established accept
        iifname wan oifname lan jump filter-port-forwards
      }
      chain output {
        type filter hook output priority 0; policy accept;
      }
    }
    
    table nat {
      chain port-forward {
        # example
        #tcp dport { http, https } dnat 10.69.1.1
      }
      chain prerouting {
        type nat hook prerouting priority 0;
    
        iifname wan jump port-forward
      }
    
      chain lan-port-forwarding {
        # example
        #ip daddr 10.69.1.1 tcp dport { http, https } snat $firewall
      }
      chain postrouting {
        type nat hook postrouting priority 100;
    
        oifname wan counter masquerade
        oifname lan ip saddr $lan_net jump lan-port-forwarding
        oifname lan ip saddr $vpn_net snat 10.69.0.1
      }
    }
    
    # vim:set ts=2 sw=2 et:
    
    3. Enable nftables (systemctl enable nftables) 4. Write net.ipv4.ip_forward=1 into /etc/sysctl.d/forwarding.conf

WireGuard

  1. Install wireguard-tools and wireguard-dkms (you'll also need the kernel headers, e.g. linux-headers for regular Arch, linux-raspberrypi-headers for ARMv7 Raspberry Pis)
  2. Generate private and public key (as root): wg genkey | sudo tee /etc/wireguard/privkey | wg pubkey > /etc/wireguard/pubkey
  3. Change private key permissions chmod 600 /etc/wireguard/privkey
  4. Create /etc/wireguard/vpn.conf:

    [Interface]
    PrivateKey = theprivatekeyhere
    Address = 10.69.255.1/16
    ListenPort = 51820
    
    [Peer]
    PublicKey = theirpublickeyhere
    AllowedIPs = 10.69.255.2/32
    

    Replace the private key with the contents of /etc/wireguard/privkey! For each user, create a [Peer] section with their public key and a new IP.

  5. Create a client configuration file:

    [Interface]
    PrivateKey = theprivatekeyhere
    Address = 10.69.255.2/16
    DNS = 10.69.0.1, netsoc.internal
    
    [Peer]
    PublicKey = serverpublickeyhere
    AllowedIPs = 10.69.0.0/16, 192.168.69.0/24, 134.226.0.0/16
    Endpoint = shoe.netsoc.ie:51820
    

    A private key for the client can be generated with wg genkey as before.

  6. Enable and start the WireGuard service: systemctl enable --now wg-quick@vpn

Alpine Linux setup

Make sure the server to be provisioned is set to UEFI mode and boot over PXE (IPv4). To install:

  1. Boot over PXE, the correct flavor should be pre-selected (lts for bare metal, virt for a VM).
  2. Log in as root and run:
    1. apk add vlan (to pre-install VLAN support for ifupdown)
    2. mkdir /etc/udhcpc && echo 'NO_GATEWAY="lan"' > /etc/udhcpc/udhcpc.conf (to prevent using the LAN interface for internet access)
  3. Run setup-alpine and follow the prompts:

    1. Choose ie as the keyboard layout and then ie as the variant
    2. Enter the name of the server for the hostname, e.g. napalm
    3. When asked which network interface you'd like to initialize, say done.
    4. Say yes to do manual network config and paste the following (after the section configuring the loopback interface):

      Note

      Make sure to change the MAC address to match the LAN interface's MAC, set the hostname appropriately and the public IP address.

      auto lan
      iface lan inet dhcp
          pre-up [ -e /sys/class/net/eth0 ] && (ip addr flush dev eth0 && ip link set dev eth0 down) || true
          pre-up nameif $IFACE 52:54:00:12:34:57
      
      auto wan
      iface wan inet static
          vlan-raw-device lan
          vlan-id 420
          address 134.226.83.xxx
          netmask 255.255.255.0
          broadcast 134.226.83.255
          gateway 134.226.83.1
      
    5. Enter Europe/Dublin for the timezone

    6. Use none for the HTTP proxy
    7. chrony is fine for the NTP client
    8. The default mirror (dl-cdn.alpinelinux.org) is fine
    9. Use openssh for the SSH server
    10. Enter none for the disk
    11. none for where to store configs and apk cache directory
  4. Replace the contents of /etc/apk/repositories with:

    http://uk.alpinelinux.org/alpine/v3.12/main
    http://uk.alpinelinux.org/alpine/v3.12/community
    @edge http://uk.alpinelinux.org/alpine/edge/main
    @edge http://uk.alpinelinux.org/alpine/edge/community
    @testing http://uk.alpinelinux.org/alpine/edge/testing
    

    Note the mirror name and Alpine branch (v3.12 in this case). Run apk update after to update the package lists.

  5. Run apk add nfs-utils and rc-update add nfsclient to install NFS and start the client on boot. Run rc-service nfsclient start to start the client now

  6. Install and enable autofs (apk add autofs, rc-update add autofs) and configure the apkovl NFS share:

    1. Replace the contents of /etc/autofs/auto.master with the following:

      /mnt /etc/autofs/auto.shoe --timeout 5
      
    2. Paste the following into /etc/autofs/auto.shoe:

      lbu -rw shoe.netsoc.internal:/srv/http/apkovl
      

      Note

      autofs is used instead of just putting the mount into /etc/fstab to only mount the share when required and more gracefully handle disconnects

      Run rc-service autofs start to start autofs now.

  7. Set up LBU (Alpine Local Backup):

    When running in diskless mode, Alpine Linux overlays configuration from a tar stored on a remote server. In order to update configuration, the system must be configured. In this case the configs will be stored on the boot server via the NFS share set up above.

    Now that the share is set up and accessible, edit /etc/lbu/lbu.conf and:

    • Uncomment and set LBU_BACKUPDIR to /mnt/lbu
    • (Optional but recommended) Uncomment and set BACKUP_LIMIT to some value (e.g. 5) in order to keep a number of backups

    Tip

    If you make a mistake in any of the above steps, just reboot to start over! All changes up to this point exist only in memory!

  8. Save the configuration:

    1. Run lbu status (or lbu st for short) - a big list of files should appear; this is the list of modified files compared to the existing overlay archive
    2. Do lbu commit to save the changes (a file myserver.apkovl.tar.gz should now be present in /mnt/lbu)
    3. Re-run lbu status - nothing should be listed, this means lbu is reading the saved archive and detecting there are no changed files.
    4. Run ln -sf myserver.apkovl.tar.gz /mnt/lbu/52:54:00:12:34:57.tar.gz (ensuring to insert the correct hostname and MAC address). This will enable the Alpine Linux init script to locate the overlay archive without the hostname on boot.

Danger

Make sure to commit any filesystem changes with lbu commit in future!


Last update: 2020-11-18