Skip to content

Sysadmin guides

Getting started

In order to get started as a sysadmin for Netsoc, you'll need:

  • A good level of Linux system administration knowledge (and ideally experience with Alpine Linux)
  • A strong knowledge of Kubernetes
  • An understanding of GitOps (specifically how Flux2) works
  • Ideally some experience in Go programming and REST APIs

To connect to Netsoc you'll need a few things (provided by an existing sysadmin!):

  • Your SSH public added to ~/.ssh/authorized_keys on shoe
  • A copy of the Netsoc SSH key (~/.ssh/id_rsa on shoe) to connect to machines
  • A copy of the Netsoc PGP key (can be exported from shoe via gpg --export-secret-keys --armor DB2E28B13D53C8DD62FE560B408F6E592A12DF74 and imported with gpg --import)
  • An account for our password manager, see here for details
  • A VPN config file, see here for details on how to create one
  • A kubeconfig to access the Kubernetes cluster. A copy of /etc/rancher/k3s/k3s.yaml from a Kubernetes node will work, but the server must be changed to https://shoe:6443

Once you have all of these pieces, you can connect to shoe by ssh If you connect to the VPN, you can access other machines, e.g. ssh root@cube. You must be on the VPN to access the Kubernetes cluster.

Accessing BMCs

Each of our main servers has a BMC that features some sort of remote management that includes power on / off and virtual KVM capabilities. All of the BMCs require VPN connectivity to access.


shoe hosts PiKVM for access to spoon. Once connected to the VPN, simply visit https://shoe. The credentials are root:hunter22.


gandalf features a HP iLO 4 with an "Advanced" license. Visit https://gandalf-ilo and click on "HTML5" under "Integrated Remote console" once logged in. The credentials are Administrator:hunter22.

napalm / cube

Both napalm and cube feature a Dell iDRAC 6 Enterprise. Once logged in, you'll need to use the Java Web Start console. Click the "Launch" button under the "Virtual Console Preview" section on the "System Summary" page. The credentials are root:hunter22 and you can connect via https://napalm-idrac and https://cube-idrac.


The iDRAC web UI is a bit old and finicky. Although it works in modern browsers (as of Q3 2021), you might need to reload the page a few times to see all UI elements... You might also need Java 8 to get the console to work.

Maintenance domain

Needs to be renewed with our registrar. See the DNS docs for details.

* TLS certificate

This certificate is issued to us by IT Services and must be manually renewed. Once obtained, the cert and key should be updated in both gitops/infrastructure/common/ and on the mailcow VM. See here for more details about updating the mailcow cert.

GitHub Actions tokens

There are currently to Personal Access Tokens (PATs) in use by many of our repos on GitHub. The CI_PAT is set in the Netsoc GitHub organisation's settings and needs the public_repo scope. This allows repos to push charts and documentation to their respective central repos. Additionally, a PAT with the repo:status scope is needed for Flux2 notifications (stored at infrastructure/monitoring/notifications/secrets/github-token.bin).

See here for details on creating GitHub Personal Access Tokens. These tokens must be renewed regularly by the primary sysadmin.

Last update: 2021-08-30