Network

Architecture

Netsoc uses three subnets:

10.69.0.0/16: Netsoc private LAN
10.79.0.0/24: Netsoc private VPN
134.226.0.0/16: TCD main allocation, Netsoc is permitted only to use 134.226.83.3-254
192.168.69.0/24: Maths server room management network

shoe acts as a NAT router for all boxes on the private VLAN. Additionally, due to limitations placed on most IPs in 134.226.83.0/24, shoe forwards mail and DNS traffic.

IP address allocation

Management VLAN

192.168.69.1: Maths room boot server
192.168.69.10: nintendo (maths room switch)

Netsoc VLAN

Note

LAN IP addresses are handed out over DHCP (IP addresses fixed in dnsmasq.conf)

Physical machines

10.16.0.1: shoe (Maths room boot server)
10.16.1.1: napalm (Kubernetes node)
10.16.1.2: spoon (Kubernetes node)
10.16.1.3: cube (Kubernetes node)
10.16.1.4: gandalf (VM host)

BMCs

10.16.10.1: napalm-idrac
10.16.10.3: cube-idrac
10.16.10.4: gandalf-ilo

VMs

10.69.2.1: mail (mail server)
10.69.2.2: saruman (Kubernetes node)

TCD WAN

VIP in this case refers to a "floating" IP that is dynamically assigned to a Kubernetes node in order to forward traffic to a Kubernetes server. See the MetalLB deployment for details.

134.226.83.42: shoe (boot server and mail / DNS proxy)
134.226.83.27: DNS VIP
134.226.83.100: HTTP(S) load balancer VIP
134.226.83.101: Webspaces port forwarding VIP
134.226.83.102: SHH VIP
134.226.83.103: Gitea VIP

Note

Traffic on 134.226.83.101 is forwarded directly to Netsoc members' webspaces and is therefore not managed by us. Webspaces are entirely isolated from internal networks and use of the port forwarding service is subject to our ToS.

There are a few known externally imposed firewall rules on IPs in 134.226.83.0/24:

Most IPs are completely shut from both internal and external traffic
134.226.83.42 has no (known) limits
134.226.83.100-107 have all UDP and TCP ports open except for DNS and SMTP
134.226.83.27 has DNS ports open (and possibly a few others, such as SSH and HTTP)

Firewall rules

shoe

Since shoe acts as a router, it has a set of nftables rules to accommodate this. The current live configuration (synced through Git) is:

#!/usr/bin/nft -f
flush ruleset

define lan_net = 10.69.0.0/16
define vpn_net = 10.79.0.0/24

define firewall = 10.69.0.1
define firewall_public = 134.226.83.42

define mail_host = 10.69.2.1
define ns1 = 134.226.83.27

define wireguard = 51820
define iperf3 = 5201
define mail_ports = { smtp, submissions, submission, imap, imaps, pop3, pop3s, sieve }

table inet filter {
  chain wan-tcp {
    tcp dport ssh accept
    tcp dport $iperf3 accept
  }
  chain wan-udp {
    udp dport $wireguard accept
    udp dport $iperf3 accept
  }

  chain wan {
    # ICMP & IGMP
    ip6 nexthdr icmpv6 icmpv6 type {
      destination-unreachable,
      packet-too-big,
      time-exceeded,
      parameter-problem,
      mld-listener-query,
      mld-listener-report,
      mld-listener-reduction,
      nd-router-solicit,
      nd-router-advert,
      nd-neighbor-solicit,
      nd-neighbor-advert,
      ind-neighbor-solicit,
      ind-neighbor-advert,
      mld2-listener-report,
      echo-request
    } accept
    ip protocol icmp icmp type {
      destination-unreachable,
      router-solicitation,
      router-advertisement,
      time-exceeded,
      parameter-problem,
      echo-request
    } accept
    ip protocol igmp accept

    # separate chains for TCP / UDP
    ip protocol tcp tcp flags & (fin|syn|rst|ack) == syn ct state new jump wan-tcp
    ip protocol udp ct state new jump wan-udp
    ip protocol esp accept
  }

  chain filter-port-forwards {
    ip daddr $mail_host tcp dport $mail_ports accept

    ip daddr $ns1 udp dport domain accept
    ip daddr $ns1 tcp dport domain accept
  }

  chain input {
    type filter hook input priority 0; policy drop;

    # established/related connections
    ct state established,related accept

    # invalid connections
    ct state invalid drop

    # allow all from loopback / lan
    iif lo accept
    iifname { eth0, lan, vpn } accept

    iifname wan jump wan
  }
  chain forward {
    type filter hook forward priority 0; policy drop;

    # see comment on same rule in output chain
    oifname wan tcp flags { syn, rst } tcp option maxseg size set 1000

    # lan can go anywhere
    iifname { eth0, lan, vpn } accept

    iifname wan oifname { lan, wan } ct state related,established accept
    iifname wan oifname { lan, wan } jump filter-port-forwards
  }
  chain output {
    type filter hook output priority 0; policy accept;

    # something is weird with downstream networking in maths, clamping the mss
    # greatly reduces loss and improves tcp bandwidth
    oifname wan tcp flags { syn, rst } tcp option maxseg size set 1000
  }
}

table nat {
  chain port-forward {
    tcp dport $mail_ports dnat $mail_host

    # Hack for a "second nameserver"
    ip daddr $firewall_public udp dport domain dnat $ns1
    ip daddr $firewall_public tcp dport domain dnat $ns1
  }
  chain prerouting {
    type nat hook prerouting priority 0;

    iifname wan jump port-forward
    iifname lan ip daddr { $firewall, $firewall_public } jump port-forward
  }

  chain lan-port-forwarding {
    ip daddr $mail_host tcp dport $mail_ports snat $firewall_public
  }
  chain postrouting {
    type nat hook postrouting priority 100;

    oifname wan counter masquerade
    oifname lan ip saddr $lan_net jump lan-port-forwarding
    oifname lan ip saddr $vpn_net snat $firewall
  }
}

# vim:set ts=2 sw=2 et:

Kubernetes VIPs

Since traffic for Kubernetes public IPs can enter the cluster on any Kubernetes node, each hosts the same set of nftables rules. Here's the current live configuration:

#!/usr/bin/nft -f
flush ruleset

define lb_dns = 134.226.83.27
define lb_http = 134.226.83.100
define lb_ws_forward = 134.226.83.101
define lb_shh = 134.226.83.102
define lb_git = 134.226.83.103

table inet filter {
  chain wan-tcp {
    ip daddr $lb_dns tcp dport domain accept
    ip daddr $lb_http tcp dport { http, https, 8443 } accept
    ip daddr $lb_ws_forward tcp dport 49152-65535 accept
    ip daddr $lb_shh tcp dport ssh accept
    ip daddr $lb_git tcp dport { ssh, http, https } accept
  }
  chain wan-udp {
    ip daddr $lb_dns udp dport domain accept
  }

  chain wan {
    ip protocol icmp icmp type {
      destination-unreachable,
      router-solicitation,
      router-advertisement,
      time-exceeded,
      parameter-problem,
      echo-request
    } accept

    ip protocol tcp tcp flags & (fin|syn|rst|ack) == syn ct state new jump wan-tcp
    ip protocol udp ct state new jump wan-udp

    drop
  }

  chain input {
    type filter hook input priority 0; policy accept;

    ct state established,related accept
    ct state invalid drop

    iifname wan jump wan
  }
}

# vim:set ts=2 sw=2 et:

Maths server room

All our hardware in maths' server room is connected to nintendo (our switch, see the hardware docs).

VLANs

In order to isolate traffic from each subnet (particularly the TCD public network!), VLAN's are configured on nintendo.

VLAN 1 (management)

This VLAN exists just to configure nintendo. It's untagged on all ports (since it's the default VLAN) and the PVID on ports 1, 2, 47 and 48 (leftmost 2 and rightmost 2 ports respectively).
VLAN 69 (Netsoc private LAN)

Untagged on all ports (except for management and TCD uplink 3, 4, 5 and 6), tagged on management ports. Also the PVID for all ports except management and TCD uplink ports.
VLAN 420 (TCD public network)

Untagged and PVID on ports 3, 4, 5, and 6 (for uplink to maths' switches). Tagged only on all others to avoid accidentally connecting to the public network!

Last update: 2021-08-29