Snark has IPMI (Intelligent Platform Management Interface; a standard for baseboard management control) v 1.21 on it.
It is essentially a miniature operating system that allows us to fiddle with the hardware without being physically at the server. This is turned on automatically when power is connected to the server, and uses the same network interface (eth0; the only interface connected to snark at time of writing) as the server.
It's IPv4 address is 220.127.116.11, with FQDN of snark-ipmi.netsoc.tcd.ie.
We can do basic manipulations to the hardware, such as powering up or down the server, getting sensor data (even though munin does some of this anyway), and getting a serial connection over the network (called serial on LAN (SOL)).
How to use the IPMI: There is a password in the vault for snark's ipmi.
Use ipmitool -H snark-ipmi.netsoc.tcd.ie -U root -I lanplus shell
(-H ~ host, -U ~ user, -I ~ interface, we use lanplus since it has SOL capability; shell is just the command we exeucte at the end to give us an iterative prompt)
We enter the password, and get a shell.
To check if snark is on or off: power status
...gives Chassis Power is on
To turn off: power off
To turn on: power on
Serial on Lan: This is a bit trickier than usual since we have Xen. Firstly, we need to enable the serial console in the BIOS, we redirect output to COM2 (COM1 is the physical serial port on the back of the machine). We use a rate of 57600 bps.
Next we must set up grub to give us output for the bootloader. In /etc/default/grub we added:
GRUB_TERMINAL="serial console" GRUB_SERIAL_COMMAND="serial --unit=1 --speed=57600" GRUB_CMDLINE_XEN="loglvl=all guest_loglvl=all com1=57600,8n1,0x2f8,3 console=com1,vga" GRUB_CMDLINE_LINUX="console=hvc0 earlyprintk=xen"
Taking these in turn:
1) enable serial console output from GRUB
2) Specify certain settings to GRUB on how to output to the serial console
3) To the XEN kernel, pass it the information about the serial port (must be called com1 or else xen breaks for some weird reason), 57600bps, 8 databits, no parity bit, 1 stopbit, the IOport for ttyS1 (obtained by looking at dmesg | grep ttyS), and the IRQ for ttyS1
4) To the linux kernel we set the console to hvc0 (the xen console, which is now acting as the serial console)
and do update-grub.
Finally to allow us to login to the machine, we need to set up an entry for ttyS1 in /etc/inittab
s1:2345:respawn:/sbin/agetty -L 57600
To allow root logins we need to have the entry ttyS1 in /etc/securetty (but that is there by default in Debian)