DRBD
From Linux-VServer
Contents |
A Vserver guest mirrored by DRBD with heartbeat failover
The idea is simple: Install a vserver on a DRBD resource and let it failover to the other node of a cluster.
This configuration uses vserver, DRBD and heartbeat. This page will not describe, how these packages are installed and configured. Only the important topics are mentioned.
Steps of installation:
- compile a vserver kernel
- compile the DRBD module against this kernel
- install the kernel, the module and reboot
- install the user space utilities of DRBD, vserver and heartbeat
- install the same packages at the second node
- configure and activate a DRBD resource
- make the DRBD resource primary on one node
- create a journalled filesystem (ext3, xfs, reiser) on top of the DRBD device
- mount the DRBD device as /var/lib/vservers/<vserver-name>
- install a vserver
- finish configuration
- sync the configuration to the other node
In detail
Install the necessary packages
On Debian Sarge we can use ready packages. Your /etc/apt/sources.list should have an entry pointing to unstable.
apt-get update
Maybe some additional packages are necessary:
apt-get install libdb3-dev libncurses-dev docbook-utils apt-get install fakeroot dpatch
Get the kernel source and unpack it:
apt-get install linux-source-2.6.12 cd /usr/src tar xjf /usr/src/linux-source-2.6.12.tar.bz2
Maybe you want to have your old .config
cd linux-source-2.6.12 cp /boot/config-2.6.12-1-386 .config
Now it's time to install patches:
apt-get install util-vserver -t unstable apt-get install kernel-patch-vserver -t unstable
Compile the Kernel:
make-kpkg --rootcmd fakeroot --revision custom01 --added-patches vserver--append-to-version +vserver --initrd binary-arch
Now you should find a .deb ready to install:
cd .. dpkg -i kernel-image-2.6.12+vserver_custom01_i386.deb
The configuration of GRUB should be updated now to boot the new kernel - try a reboot.
Install the DRBD packages:
apt-get install drbd0.7-module-source -t unstable apt-get install drbd0.7-utils -t unstable
Unpack the module source:
cd /usr/src/ tar xzf drbd0.7.tar.gz
Compile the DRBD module:
cd linux-source-2.6.12 make-kpkg --append-to-version +vserver modules_image
Install the module:
cd .. dpkg -i /usr/src/drbd0.7-module-2.6.12+vserver_0.7.11-1+custom01_i386.deb update-modules modprobe drbd
Prepare the disk space
You will need some block device like /dev/hdaX, or a better choice is to use LVM.
lvcreate -L3000 -nlvvs03 vg0
A recommended method is, to have DRBD metadata on an extra block device for each DRBD device.
Prepare space for it:
lvcreate -L132 -nlvvs03md vg0
Configure DRBD
Create an entry in /etc/drbd.conf using your favorite editor:
resource vs03 {
protocol C;
incon-degr-cmd "echo '!DRBD! pri on incon-degr' | wall ; sleep 60 ; halt -f";
startup {
wfc-timeout 100;
degr-wfc-timeout 120; # 2 minutes.
}
disk {
on-io-error pass_on;
}
net {
# sndbuf-size 512k;
# timeout 60; # 6 seconds (unit = 0.1 seconds)
# connect-int 10; # 10 seconds (unit = 1 second)
# ping-int 10; # 10 seconds (unit = 1 second)
# max-buffers 2048;
# max-epoch-size 2048;
# ko-count 4;
# on-disconnect reconnect;
}
syncer {
rate 10M;
group 1;
al-extents 257;
}
on cel600 {
device /dev/drbd1;
disk /dev/vg0/lvvs03;
address 192.168.0.22:7789;
meta-disk /dev/vg0/lvvs03md[0];
}
on xp2400 {
device /dev/drbd1;
disk /dev/vg0/lvvs03;
address 192.168.0.21:7789;
meta-disk /dev/vg0/lvvs03md[0];
}
}
Start DRBD
/etc/init.d/drbd start
The DRBD resource is in an undefined state. It needs to be forced to be 'primary' on the current node.
drbdsetup /dev/drbd1 primary --do-what-I-say
Obviously a filesystem is needed on top of DRBD:
mke2fs -j /dev/drbd1
Also we need a mount point somewhere:
mkdir /mnt/vs03
As heartbeat is intended to manage the vserver, using the heartbeat script for mounting is a first test:
/etc/ha.d/resource.d/Filesystem /dev/drbd1 /mnt/vs03 ext3 start
Install the vserver guest
There are many methods how to install a vserver. Here 'debootstrap' is used. The important option is '--rootdir' to force the installation to the DRBD device.
vserver -v vs03 build --force --rootdir /mnt/vs03 \ -m debootstrap --hostname vs03.nack.at \ --netdev eth0 --interface 192.168.0.3/24 --context 3 -- \ -d sarge -m http://ftp.at.debian.org/debian/
We could start the vserver now. But first we try to make the configuration more pretty.
cd /mnt/vs03/vs03/etc/rc0.d rm K20makedev K25hwclock.sh S30urandom S31umountnfs.sh S35networking S36ifupdown S40umountfs S90halt K89klogd cd ../rc6.d rm K20makedev K25hwclock.sh S30urandom S31umountnfs.sh S35networking S36ifupdown S40umountfs S90reboot K89klogd cd ../rcS.d rm S05keymap.sh S48console-screen.sh S50hwclock.sh S40networking S45mountnfs.sh S10checkroot.sh S02mountvirtfs rm S30procps.sh S35mountall.sh S36mountvirtfs S39ifupdown S30checkfs.sh S18ifupdown-clean S18hwclockfirst.sh cd ../rc2.d rm S20makedev S11klogd
If we want to have virtual IPs in the host context managed by heartbeat, there can be conflicts (and funny effects), if we do not 'name' the alias of the vserver IP.
echo vs03 > /etc/vservers/vs03/interfaces/0/name
sshd should only listen to the IPs of the host context:
xp2400:/# cat /etc/ssh/sshd_config | grep ListenAddress ListenAddress 192.168.0.21:22 ListenAddress 192.168.0.20:22
We want to limit the vserver to 1 GB AS/VM (virtual address space) and 128 MB RSS (real memory). Note, that the units are pages.
xp2400:/# mkdir /etc/vservers/vs03/rlimits xp2400:/usr/src# echo 250000 > /etc/vservers/vs03/rlimits/as xp2400:/# echo 32000 > /etc/vservers/vs03/rlimits/rss
We want control CPU consumption:
xp2400:/# echo sched_prio > /etc/vservers/vs03/flags xp2400:/# echo 6 > /etc/vservers/vs03/schedule xp2400:/# echo 30 >> /etc/vservers/vs03/schedule xp2400:/# echo 1000 >> /etc/vservers/vs03/schedule xp2400:/# echo 200 >> /etc/vservers/vs03/schedule xp2400:/# echo 3000 >> /etc/vservers/vs03/schedule xp2400:/# echo dummy >> /etc/vservers/vs03/schedule xp2400:/# cat /etc/vservers/vs03/schedule 6 30 1000 200 3000 dummy
After these checks it's time to start the vserver:
xp2400:/# vserver vs03 start
Check if the interface is nicely named and shown with ifconfig:
xp2400:/# ip addr show
[...]
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:30:bd:6b:d6:8f brd ff:ff:ff:ff:ff:ff
inet 192.168.0.21/24 brd 192.168.0.255 scope global eth0
inet 192.168.0.3/24 brd 192.168.0.255 scope global secondary eth0:vs03
inet6 fe80::230:bdff:fe6b:d68f/64 scope link
valid_lft forever preferred_lft forever
xp2400:/# ifconfig
eth0 Link encap:Ethernet [...]
eth1 [...]
eth0:vs03 Link encap:Ethernet HWaddr 00:30:BD:6B:D6:8F
inet addr:192.168.0.3 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:177 Base address:0xd000
Enter the vserver:
xp2400:/# vserver vs03 enter
In the vserver we check the base configuration and install some packages.
vs03:/# base-config vs03:/# apt-get update vs03:/# apt-get install locales vs03:/# apt-get install less vs03:/# nano /etc/resolv.conf vs03:/# ping www.linux-vserver.org
We want to install at least one service, sshd.
vs03:/# apt-get install ssh vs03:/# nano /etc/ssh/sshd_config vs03:/# cat /etc/ssh/sshd_config | grep ListenAddress ListenAddress 192.168.0.3:22 vs03:/# /etc/init.d/ssh restart Restarting OpenBSD Secure Shell server: sshd. vs03:/# passwd Enter new UNIX password: Retype new UNIX password: passwd: password updated successfully vs03:/# exit logout
Test the ssh login:
xp2400:/# ssh root@192.168.0.3 The authenticity of host '192.168.0.3 (192.168.0.3)' can't be established. RSA key fingerprint is e2:19:d1:8d:7f:ac:0a:ce:12:f5:06:b7:ae:41:96:59. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.0.3' (RSA) to the list of known hosts. Password: <secret> vs03:~#
As another service Apache2 should be used:
vs03:/# apt-get install apache2 [...] Starting web server: Apache2(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs invoke-rc.d: initscript apache2, action "start" failed. Setting up apache2 (2.0.54-4) ...
To avoid the above error:
vs03:/# nano /etc/apache2/ports.conf vs03:/# cat /etc/apache2/ports.conf Listen 192.168.0.3:80 vs03:/# /etc/init.d/apache2 start Starting web server: Apache2(98)Address already in use: make_sock: could not bind to address 192.168.0.3:80 no listening sockets available, shutting down Unable to open logs
Do we have an Apache listening on the vserver IP in the host context?
xp2400:/# cat /etc/apache2/ports.conf Listen 80 xp2400:/# nano /etc/apache2/ports.conf xp2400:/# cat /etc/apache2/ports.conf Listen 192.168.0.21:80 Listen 127.0.0.1:80 xp2400:/# /etc/init.d/apache2 restart Forcing reload of web server: Apache2. vs03:/# /etc/init.d/apache2 start Starting web server: Apache2. xp2400:/# lynx http://192.168.0.3/ Test Page for Apache Installation If you can see this, it means that the installation of the Apache web server software on this system was successful. You may now add content to this directory and replace this page. [...]
The configuration on one node is ready.
To be continued ...
