There are several ways to access the servers:
- eduroam and LAN (All servers in TU munich are accessible from within the TUM network)
- Via SSH jump host: recommended for ssh, required for students
- We have one Proxy jump host that contains all SSH keys that are added to the nixos configuration i.e. in modules/users.nix
- Reproducible example:
SSH_AUTH_SOCK= ssh -v -F /dev/null -i <path/to/privkey> -oProxyCommand="ssh tunnel@login.dos.cit.tum.de -i <path/to/privkey> -W %h:%p" <yourusername>@graham.dos.cit.tum.de - Alternatively, add the following to your
~/.ssh/config:
Host *.dos.cit.tum.de !login.dos.cit.tum.de ProxyJump tunnel@login.dos.cit.tum.de- Keys are uploaded via the machine astrid whenever nixos configuration is updated.
- You can generate an SSH config file for all TUM hosts with this script, providing your username as an argument
- VPN provided by RBG: recommended for admins
- this option only works for ls1 employes
- this vpn also gives access to the management network (i.e. for IPMI access)
- use the
dosprofile from here
All servers in TUM have public ipv6/ipv4 addresses and dns record following the format:
$hostname.dos.cit.tum.defor the machine itself.$hostname-mgmt.dos.cit.tum.defor the IPMI/BMC interface.
i.e. astrid has the addresses astrid.dos.cit.tum.de and astrid-mgmt.dos.cit.tum.de.
On servers where we import xrdp.nix, we have graphical access via xrdp. This is mainly useful for xilinx development.
User need to have xrdpAccess set to true in their account entry in ../modules/users.
After than run the following commands:
$ inv generate-password --user <USER>
Send the password in <USER>-password to the student
and store -password-hash in ./modules/users/xrdp-passwords.yml by doing:
$ sops ./modules/users/xrdp-passwords.yml
You may have to restart xrdp-sesman.service for the changes to apply.
We have dedicated management interfaces (IPMI/BMC) for our servers. See IPMI.md for documentation on how to access the web interface and how to use the CLI tools for power cycling, serial console access, etc.
- Expansion cards and PCI slots in general
- GPUs in particular
- Network graph (see also networking notes in "Expansion cards and slots")
Our epyc servers are shared devices on which many users usually work concurrently.
- single NUMA node (EPYC 7713P):
- single NUMA node (EPYC 9654P)
- dual NUMA node (EPYC 9654), GPU
- dual NUMA node (EPYC 7413, for many expansion cards)
- dual NUMA node (AMD EPYC 9334)
- dual NUMA node (AMD EPYC 9965)
Those servers (or individual devices) are sometimes used exclusively by a single user to conduct benchmarks.
- single socket Xeon Gold 5317
- dual socket Xeon Gold 6326, GPU
- dual socket Xeon Gold 6438Y+, CXL support
- dual socket Xeon Platinum 8562Y+, TDX support
- dual socket Xeon 6756E, TDX support
- dual socket Xeon 6980P, TDX not enabled
Those serve as a github action runner for Systemprogramming + cloud systems lab. Graham runs the nixbot CI service (doctor reverse-proxies it; eliza is an aarch64 remote builder). See nixbot.md for documentation on how to add repositories and checks.
- bill
- nardole
Ace is currently running Debian. To add users, see these instructions.
- ace (Morello)
- For debugging, see these instructions
Each of these machines is equipped with an Alveo U50 FPGA card. Those servers are manually managed by @atsushikoshiba. They run ubuntu - that means that accounts/ssh keys added to this repos won't appear on those machines. Those machines also are not backed up.
- RBG VMs:
- dosvm5.cit.tum.de (ubuntu VM itself), doctor.dos.cit.tum.de (nixos container in VM) doctor.nix: borg backup target, monitoring
- grafana - login with your RBG/CIT account (same as for login.dos.cit.tum.de, not the TUM-ID). Everyone in the DOS org (rbgOrgZug=DOS) can log in; dosvm5 admins get grafana admin
- prometheus/alertmanager are behind authelia: RBG/CIT password or passkey (register one under auth.dos.cit.tum.de → settings; the confirmation code is mailed to your CIT address)
- SSHing into the nixos container that runs all services: use the login.dos.cit.tum.de jumphost as usual, but ssh on doctor.dos.cit.tum.de uses port 2222!
- prometheus - see monitoring.md for adding machines
- alertmanager
- buildbot
- doctorold (vmbhatotia43.in.tum.de, 131.159.102.4): previous monitoring VM, only forwards :80/:443 to doctor.r until the CNAMEs point to dosvm5
- login.dos.cit.tum.de README: ssh jumphost
- ls1-coffee.dse.cit.tum.de: coffee backend
- web.dse.in.tum.de: public website
- dosvm1.cit.tum.de: pxeboot
- dosvm5.cit.tum.de (ubuntu VM itself), doctor.dos.cit.tum.de (nixos container in VM) doctor.nix: borg backup target, monitoring
We have a shared nfs-based /home mounted. The nfs for /home is based on a NVME
disk on mickey and is limited to 1.5TB.
Please do not store large amounts of data such as VM images here. VM images of
running VMs will also interfere with the Backupsoftware.
Instead if you need fast local disk access use /scratch/$YOURUSER
- however unlike
/homeand/sharethis directory are not included in the backup. If you want to share larger datasets between machines use/share, which is based on two hard disk (15TB capacity).
Both nfs export stored on mickey are also replicated to dan every 15
minutes using zfs replication based on
syncoid.
In case there are hardware problems with mickey, dan can take over serving
the nfs.
Our nfs servers allows connections from the 2a09:80c0:102::/64 network.
Add the following line to /etc/hosts:
2a09:80c0:102::f000:0 nfs
And the following lines to /etc/fstab to mount a shared /home and /share
nfs:/export/home /home nfs4 nofail,timeo=14 0 2
nfs:/export/share /share nfs4 nofail,timeo=14 0 2
On NFS server (mickey), run
$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
nfs-data 14.5T 4.23T 10.3T - - 0% 29% 1.00x ONLINE -
nfs-home 3.48T 3.38T 110G - - 49% 96% 1.00x ONLINE -
zroot 1.45T 152G 1.30T - - 12% 10% 1.00x ONLINE -
To get the disk usage per user, run
sudo nix run nixpkgs#ncdu -- /export/home
ZFS is used on all machines whenever possible. We enable automatic snapshots of
the filesystem every 15 minutes. The snapshot can be accessed by entering the
.zfs directory of a zfs dataset mountpoint.
- for NFS mounted directories, snapshots are on the NFS master node (mickey,
/export/home/.zfsor/export/share/.zfs) - for local zfs datasets (
zfs list) snapshots are at/.zfs,/home/.zfs, ... - note that
.zfsis not seen byls
Furthermore /share and /home are backed up daily to get RBG storage using
borgbackup. See also nixos borg wiki.
[root@mickey:~]# eval $(ssh-agent)
[root@mickey:~]# ssh-add /run/secrets/tum-borgbackup-home-ssh
[root@mickey:~]# borg-job-nfs-home listZFS taking snapshots means that deleting files does not immediately free up space. To reclaim space immediately, you have to delete the snapshots.
To check which snapshots takes up the most space, (on the NFS server)
$ zfs list -t snapshot | sort -k2 -rh | head
nfs-home/home@syncoid_mickey_2025-07-29:10:45:02-GMT00:00 72.7G - 1.85T -
nfs-home/home@syncoid_mickey_2025-07-15:08:45:02-GMT00:00 18.5G - 1.54T -
nfs-home/home@syncoid_mickey_2025-07-03:13:15:01-GMT00:00 17.0G - 1.45T -
nfs-home/home@syncoid_mickey_2025-07-07:11:00:01-GMT00:00 16.2G - 1.47T -
nfs-home/home@syncoid_mickey_2025-07-04:12:15:01-GMT00:00 16.2G - 1.47T -
nfs-home/home@syncoid_mickey_2025-07-01:12:00:01-GMT00:00 16.2G - 1.44T -
nfs-home/home@syncoid_mickey_2025-07-08:11:00:01-GMT00:00 16.1G - 1.51T -
nfs-home/home@syncoid_mickey_2025-07-08:08:45:01-GMT00:00 16.1G - 1.51T -
nfs-home/home@syncoid_mickey_2025-07-03:11:00:01-GMT00:00 16.1G - 1.45T -
nfs-home/home@syncoid_mickey_2025-07-01:11:45:01-GMT00:00 16.1G - 1.44T -
The snapshot files might be shared between multiple snapshots. To check actual
size that we can reclaim by deleting a snapshot, run the following command
(-nv means "dry run"):
$ zfs destroy -nv nfs-home/home@syncoid_mickey_2025-07-29:10:45:02-GMT00:00
would destroy nfs-home/home@syncoid_mickey_2025-07-29:10:45:02-GMT00:00
would reclaim 72.7G
If this is ok, then run the command without -nv to actually delete the snapshot:
$ sudo zfs destroy nfs-home/home@syncoid_mickey_2025-07-29:09:00:02-GMT00:00
zfs destory nfs-home/home (without snapshot name) will delete the entire file
system, so be careful!
IMPORTANT: Do not delete the latest snapshots, as which are being synced with dan.
It would be safer to delete old ones. You can check the synchronization
status with the following command:
[root@mickey:~]# systemctl status syncoid-nfs-home-home
Our chair currently has three networks:
il01: for devices in the officeruby: firewall port 22 open (only forloginjumphost)ace: firewall port 22 open from anywherejoy: firewall port 22 open from anywhere
il01_16: for the servers- open to
il01(and VPN) - usually 10Gbit/s SFP+ connectors for fiber
- ipv4: 131.159.102.0/24
- ipv6: 2a09:80c0:102::/64
- open to
il01_15for management- open only to il01 VPN provided by RBG
- usually 1Gbit/s RJ-45
- ipv4: 172.24.90.0/24
il01_14for internal connections- closed, internal network, private ips
- ipv4: 172.24.89.0/24
- LRZs eduvpn:
- open to Münchner Wissenschaftsnetz
- ipv4: 10.0.0.0/8
- Via eduvpn client lrz eduvpn guide
- Or via OpenVPN client (certificate expires every few months) tum.eduvpn.lrz.de
- TUM-ITO dosvpn:
- may access:
- 131.159.0.0/16
- 192.187.0.0/16
- 10.0.0.0/8
- 172.24.0.0/17
- ipv4: 172.24.238.0(/24 maybe?)
- may access:
- LRZ: 192.187.0.0/16
- TUM-ITO/RBG: 131.159.0.0/16
- TUM-ITO/RBG private: 172.24.0.0/16
- TUM-ITO/RBG vlans: 172.24.0.0/17
- L3 Switch "Adric"
adric-mgmt.dos.cit.tum.de- see adric.md
- FibreStore N8550-32C switch with Broadcom BCM56870 Trident III chip
- 32x 100G QSFP
- 2x 10G SFP+
- 8 core Intel Xeon D-1518 @ 2.20GHz
- subnet: 10.0.0.0/24
- management via ssh with username admin
- Retired: L3 Switch "Craig"
craig-mgmt.dos.cit.tum.de(sops encrypted (config)[./craig.sops])- 6x 100Gbit/s QSFP
- many 10Gbit/s SFP+
- ip: 172.24.90.18
- vlan example config (layer2->static vlan config)
- vlan id: 1; untagged ports: Fx0/1-48,Cx0/1-2,Cx0/4; forbidden ports: Cx0/3,Cx0/5;
- vlan id: 2; vlan name: vlan2; untagged ports: Cx0/3,Cx0/5; forbidden ports: Fx0/1-48,Cx0/1-2,Cx0/4;
To add a new machine send the MAC address of your host interface and your IPMI/management interface to ls1.admin@in.tum.de.
If the RGB group asks which networks to connect your machine to, tell them il01_16 for the machine and il01_15 for IPMI/BMC.
A graph of how the servers are connected right now can be found here.
Physical position of each server in our racks. For per-host expansion cards see expansion_cards.md.
Last visually verified: 2026-07-30
┌───────────────┐ ┌────┐ ┌────┐ ┌────┐ ┌───────────────┐ ┌───────────────┐
│ │ │ │ │ │ │ │ │ │ │ vislor │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ rose (F) │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ amy (F) │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ clara (F) │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ momiji (F) │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ jackson │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ christina (S) │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ adelaide │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ wilfred │
├───────────────┤ │ │ │ │ │ │ │ │ ├───────────────┤
│ steve (G) │ │ │ │ │ │ │ │ │ │ river │
├───────────────┤ │ │ │ │ │ │ │ │ ├───────────────┤
│ eliza │ │ │ │ │ │ │ │ │ │ jack (G) │
├───────────────┤ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ dan │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ astrid │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ mickey │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ ryan │
│ │ │ │ │ │ │ │ │ │ ├───────────────┤
│ │ │ │ │ │ │ │ │ │ │ graham │
│ │ │ │ │ │ │ │ ├───────────────┤ ├───────────────┤
│ │ │ │ │ │ │ │ │ martha │ │ yasmin │
│ │ │ │ │ │ │ │ ├───────────────┤ ├───────────────┤
│ │ │ │ │ │ │ │ │ sakura (F) │ │ hinoki (F) │
│ │ │ │ │ │ │ │ ├───────────────┤ ├───────────────┤
│ │ │ │ │ │ │ │ │ irene │ │ ian │
│ │ │ │ │ │ │ │ ├───────────────┤ ├───────────────┤
│ │ │ │ │ │ │ │ │ xavier │ │ jamie (G) │
└───────────────┘ └────┘ └────┘ └────┘ └───────────────┘ └───────────────┘
leftmost rightmost
(left row from entry)