- 1 Specification
- 2 Description
- 3 Current role
- 4 Planned roles
- 5 Name
- 6 Storage
- 7 Networking
- 8 Virtualization
- Dell Poweredge 2950
- Dual quad-core 2.0GHz L5335 Low voltage Xeon processors with 8MB L2 Cache
- 16GB DDR2F RAM
- 2×146GB + 4×300GB 15,000 RPM SAS hard disks
- Dual gigabit ethernet
- Dual, redundant, hot-swappable power supplies
snark is a much-better-than-spoon-but-not-quite-cube 2U monster that was bought on adverts.ie in April 2012.
- Primary services server
- Netsoc webserver
- User webserver
- Primary LDAP server
- Authoritative DNS server
- Secondary mail server
- IRC server
- IRC services
Snark runs most of our essential services, partitioned into VMs under the Xen hypervisor. It is running Debian wheezy with kernel version 3.2.0-2-amd64 and xen version 4.1.3-rc2-pre. Currently only admins have access to both the dom0 (snarkroot) and the domUs (the various VMs that are running on it - see below for a list). It is connected to the College network via it's eth0 interface, which is running as a bridge for Xen to allow the domUs communicate. The configuration of the interface is given below:
auto xenbr0 iface xenbr0 inet static bridge_ports eth0 address 18.104.22.168 gateway 22.214.171.124 broadcast 126.96.36.199 netmask 255.255.255.0 dns-nameservers 188.8.131.52 dns-search netsoc.tcd.ie
There are two domUs running on snark at the moment: Freedom and snark. Freedom is a freeBSD 9.0 HVM whilst snark is a paravirtualised domU running Debian Wheezy. snark's IP is 184.108.40.206. both snark and freedom are running SSH servers, but neither are interfaced with the LDAP system and so only local accounts will work (and only root accounts exist at this time).
Backup login server
We'll also include a small login VM called 'snark', so that people have another place to log into in the case of downtime. This will have roughly Spoon-level resources allocated to it, and a modestly-sized /home. We won't be installing any dev software (fancy compilers, obscure libs, etc.) here, to discourage people from ruining it with coursework.
An important service which will run on snark is backups. We now have the right card and cable to drive the PowerVault disk array, and we have enough disks now. We'll back up everything on Spoon, Cube and snark to it, including all member data. Because the machine is hosted in TCHPC, this provides us with redundancy and means that if some day either the Maths department or TCHPC burns down, member data will be safe.
snark is a weapon in the original Half Life game by "Valve":http://www.valvesoftware.com/. It is from the border world, Xen, which is also the name of the virtualisation software on snark. It was also once known as Cypher.
The storage on snark is set up as a RAID1 of the 2x146GB drives whilst the 4x300GB are in a RAID5, both controlled by the onboard PERC 6/i card. LVM is set up on top of the RAID5 completely in a physical volume and volume group both called onboard-raid5 (/dev/sdb1). The 146GB RAID1 is split to a 200MB ext2 /boot partition, a 50GB ext4 /, a 24GB swap, and a 72GB LVM with its physical volume called onboard-raid1. The onboard-raid1 volume contains / for the Dom0, and space for miscellaneous storage.The onboard-raid5 volume contains the logical volumes for the various VMs the machine runs.
Snark goes hand-in-hand with the Snark disk array, which is connected via an infiniband cable to the PERC 6/E card. It contains four 1TB SATA disks, which are pooled as a RAID5, providing 3TB of storage. The disk array's storage appears in a volume group called diskarray-raid5.
Snark uses transparent (bridged, or level 2) firewalling for the VMs.
Networking in Xen
Xen has two ways of setting up networking for the VMs: routed, or bridged. In snark, we have used the bridged networking option. By default this means that all VMs and the Dom0 bind to a bridge called xenbr0. Inside each of the VMs, the interfaces are always called the usual eth0. In the Dom0, each of the interfaces that connect the bridge to the VMs is called vifX.X. For example, vif0.0 is the interface that connects the Dom0 to the bridge, but the Dom0 doesn't see it as vif0.0, but as eth0. This is because Xen masks the real name for the more conventional names. vif1.0 and vif1.1 may be two interfaces given to a VM.
Networking in Snark
However we have changed how this works, and all VMs should not use xenbr0, but instead a bridge called xenbrdmz. xenbrdmz is passed to the firewall VM snark-fw along with xenbr0. This VM does the transparent firewalling between the two bridges, which it sees as two normal ethernet interfaces (eth0 for xenbr0 and eth1 for xenbrdmz)
Shorewall is used to make the firewall rules. Shorewall has had support for bridge firewalls for a while, but has recently gotten good at it. There are a number of documents on the Shorewall website about bridge firewalls, as well as using Xen with Shorewall (Mind: some of the stuffrelating to Xen is out of date, and the 3.0 documentation isn't necessarily compatible with Shorewall 4.5). There are two firewalls needed: one on the Dom0, snarkroot which does its own firewalling, as well as passing everything through the bridge to be filtered by snark-fw.
TODO: Talk about shorewall rules, policies, and the bport zones.
Xen is used as the hypervisor. The processors in snark support VT-x. Unlike on Cube, each VM is able to have its own kernel (as opposed to sharing a single kernel with the hardware node).
Adding new vms
It's easy to make a new VM, allocate its disk space, and install debian on it. First, you need to make a partition definition file in
/etc/xen-tools/partitions.d. These files are used to tell Xen how to lay out the partitions on the machine you're creating. For example,
/etc/xen-tools/partitions.d/snark is as follows:
[root] size=2G type=ext3 mountpoint=/ options=sync,errors=remount-ro [swap] size=2G type=swap [home] size=200G type=ext3 mountpoint=/home options=nodev,nosuid [opt] size=2G type=ext3 mountpoint=/opt options=nodev [tmp] size=1G type=ext3 mountpoint=/tmp options=nodev,nosuid [usr] size=4G type=ext3 mountpoint=/usr options=nodev [var] size=4G type=ext3 mountpoint=/var options=nodev,nosuid [var-tmp] size=1G type=ext3 mountpoint=/var/tmp options=nodev,nosuid
To create this machine, you would run
xen-create-image --hostname NAMEHERE --partitions=/etc/xen-tools/partitions.d/snark --mem=4Gb --swap=1Gb --ip=134.226.83.foo --gateway=220.127.116.11 --netmask=255.255.0.0 --dist=wheezy
xm create /etc/xen/NAMEHERE.cfg, then
xm console NAMEHERE to get a shell, but an ssh server should come up by default.
REMEMBER TO UPDATE ips
In the spirit of Cube, the Dom0 is called snarkroot. Nothing will be set up on this machine except Xen, shorewall, and things which require access to hardware, like RAID and LVM and so on. It runs Debian stable
Exactly what it says on the tin.
Running squeeze, because wheezy has no hybserv package.
No longer using hybserv, will upgrade.
This machine is relatively small, and runs an authoritative DNS server based on bind. For security, bind is chrooted, and a minimal installation of debian is used.
This machine will host whatever backup software we decide to run (fwbackups, bacula, amanda...), and coordinate backups of all machines. It will only be accessible by admins (for now), and will run a stable release. If the disk array is to be used for backups only, we'll make this machine the only one with access to it.
A few somewhat sillier VMs also exist on snark. They're not powered on all the time though.
I can't believe I'm writing this, but this is an instance of Windows Server 2012, set up by DUCSS admin k3ypad.
A FreeBSD VM, for people to play with.
Other proposed VMs are;
- Morpheus - Matrix clone with netris.real and amazing (bad?) tcsh prompts. OpenIndiana, most likely, as Solaris itself is verging on really-quite-non-free.
- Shipwreck - A proposed VM for Titanic that ran legacy operating systems inside emulators (preferably ones that don't eat CPUs)