HIGH AVAILABILITY SYSTEM – A FAILOVER MECHANISM

What we want to achieve:

The seamless transfer of workload among redundant components in the cluster ensures a highly available system and less or no effort in disaster recovery.

Our scenario contains two machines, machine1 with a static IP address 192.168.3.47 and machine2 with a static IP address 192.168.3.41 Let us now assume that machine1 and machine2 share a virtual IP address 192.168.3.31 This configuration is done for the LAN side of the network. The following figure illustrates the situation:

The protocol used:

We use the Common Address Redundancy Protocol (CARP) to achieve the sharing of IP Addresses. Service-requests that come from the internet are received by both the machines, but processed only by the one with a higher priority (which is called the Master).

If the master happens to fail for any reason, then the slave starts processing the requests until the master comes back up. The shared virtual IP address will be the default gateway for machines outside this network.

About UCARP:

CARP was introduced by the Berkeley Software Distribution (BSD) that allowed (and guaranteed) multiple machines to share the same address. However, the Linux world brought UCARP into existence. UCARP guarantees functionality for two machines.

UCARP allows a pair of hosts to share common IP addresses in order to provide automatic failover from one machine to another using that common IP address.

But before diving into the details, let’s review some basic concepts of Linux and Networking.

IP Addresses and Addressing schemes – What and why:

IP (Internet Protocol) is a name given to a machine for the purposes of identification and addressing. Each machine has a unique IP address on the internet.

We have used Class C addressing. Furthermore, addresses are classified as

Static – manually assigned addresses (by a admin)
Dynamic – Assigned by the server, subject to change. This uses DHCP. (Dynamic Host Configuration Protocol)
Loopback – send traffic back to the source without processing it.

In specific, for a network interface to allow UCARP to be mounted on it, it should be defined as a static address.

Network Interface Card (NIC):

Network interface card is a component had connects our local-host to the network.

Manages communication in the lower layers of the protocol stack (physical/data-link layers)

Daemons in UNIX:

Process that always run in the background, i.e., from system boot to shut down. Not controlled by the user.

Organization of files in Unix:

Daemons in à /etc/init/
data files in à /etc/
test scripts in à /usr/bin/
log in à /var/log/

The Network Interfaces file: (/etc/network/interfaces)

Auto – bring up interface on boot
iface – block describes properties defined for the interface.
- iface <interface name> <protocol> <addressing>
- <protocol> can be inet (for IPv4) or inet6 (for IPv6)
- <addressing> can be either static, dhcp or loopback.
- Other properties in the iface block are address, gateway, netmask and dns-nameservers.

Upstart:

We use “Upstart” to make our UCARP process a daemon. Every Upsart script has a .conf extension and the following structure:

Start on [condition] stop on [condition]

script

end script

The “condition” we need is “net-device-up IFACE= <interface name>” when the interface is available.

What is it and how does it work?

Traditionally before Upstart was created, there was serial booting and no hot-plugging. Upstart caters to asynchronocitiy. It allows multiple jobs to register an interest to a particular event occurring in the system. Upstart then concurrently starts all the registered jobs and stops them on a specified condition. By doing so, Upstart efficiently brings up a daemon’s dependant processes too.

‘ifconfig’ and its parameters:

The command ‘ifconfig’ gives the status of all network interfaces. Individual interface names can also be passed as parameters to get their configuration information. This info is then used for debugging purposes

To activate the interface we use an UP-flag on that interface. The UP-flag is set once the interface is assigned an address.

To shut down an interface we use DOWN-flag. Alternatively we can use ifdown and ifup commands.

Setting up UCARP:

To get started with UCARP, one has to follow the steps mentioned below:

Installing UCARP:

Open a new terminal.
Execute: sudo apt-get install ucarp

Adding a new static interface:

Execute: cat /etc/network/interfaces

The result should look similar to:

auto lo

iface lo inet loopback.

Next edit the /etc/network/interfaces file to define your own interface:

Execute: sudo gedit /etc/network/interfaces
Enter the following into the file and save the changes.

auto lo

iface lo inet loopback

auto eth0

iface eth0 inet static

address 192.168.3.47

netmask 255.255.255.0

gateway 192.168.3.1

dns-nameservers 192.168.3.5

Now to apply the changes, execute the command sudo service networking restart.
Then check if eth0 is running by executing an ifconfig.

For Master:

Next define a UCARP interface to run on the static interface defined earlier

Execute: sudo gedit /etc/network/interfaces.
Type the following into the file after what you had typed in earlier.

auto lo

iface lo inet loopback

auto eth0

iface eth0 inet static

address 192.168.3.47

netmask 255.255.255.0

gateway 192.168.3.1

dns-nameservers 192.168.3.5

ucarp-vid 1

ucarp-passwd tequila123

ucarp-vip 192.168.3.31

ucarp-advbase 1

ucarp-advskew 10

ucarp-master yes

iface eth0:ucarp inet static

address 192.168.3.31

netmask 255.255.255.0

Perform a reboot – Execute: sudo reboot
Check the interfaces – Execute: ifconfig

The output should be similar to the following:

eth0 Link encap:Ethernet HWaddr c8:60:00:3c:e4:8b

inet addr:192.168.3.47 Bcast:192.168.3.255 Mask:255.255.255.0

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Interrupt:47 Base address:0xe000

eth0:ucarp Link encap:Ethernet HWaddr c8:60:00:3c:e4:8b

inet addr:192.168.3.31 Bcast:192.168.3.255 Mask:255.255.255.0

UP BROADCAST MULTICAST MTU:1500 Metric:1

Interrupt:47 Base address:0xe000

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:4386 errors:0 dropped:0 overruns:0 frame:0

TX packets:4386 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:590477 (590.4 KB) TX bytes:590477 (590.4 KB)

For Slave:

Repeat the previous steps for another machine which will behave as the slave.
Parts of code which differ are shown in bold.

Execute: sudo gedit /etc/network/interfaces.
Enter the following into the file:

auto lo

iface lo inet loopback

auto eth0

iface eth0 inet static

address 192.168.3.41

netmask 255.255.255.0

gateway 192.168.3.1

dns-nameservers 192.168.3.5

ucarp-vid 1

ucarp-passwd tequila123

ucarp-vip 192.168.3.31

ucarp-advbase 1

ucarp-advskew 50

ucarp-master no

iface eth0:ucarp inet static

address 192.168.3.31

netmask 255.255.255.0

Execute: ifconfig on the slave machine to confirm that UCARP is not running on it.
Next on the master machine:
- Execute: sudo ifdown eth0
- This brings down the eth0 interface on the master.
- Now Execute: ifconfig on the slave. It should now be showing UCARP running on it
- Congrats! You’ve now successfully set up UCARP.

How UCARP Works:

Consider a pair of systems consisting of a Master and a Slave running UCARP.
Both the UCARP daemons constantly exchange “keep alive” messages. This informs the other machine of its presence.
We differentiate between the master and the slave machines by the frequency of these messages.
- The master sends the “keep alive” messages more frequently than the slave.
- The frequency of the “keep alive” messages are determined by their advertisement values – adv-base and adv-skew.
- The machine that has the lower skew is the Master.
- The traffic that comes to the shared interface is multicast to both the master and the slave, but only the master processes the requests and the slave ignores them.

The ‘ucarp’ Command:

The ucarp command is can be executed with the following parameters:

ucarp [-i, –interface=INTERFACE] [-s, –srcip=IPADDRESS] [-v, –vhid=VHID] [-p, –pass=PASSWORD] [-o, –passfile=PASSFILE] [-P, –preempt] [-n, –neutral] [-a, –addr=IPADDR] [-h, –help][-b, –advbase=SECS] [-k, –advskew=SKEW] [-u, –upscript=SCRIPT] [-d, –downscript=SCRIPT] [-r, –deadratio=RATIO] [-z, –shutdown] [-B, –daemonize] [-f, –facility=FACILITY] [-x, –xparam PARAM] [-S, –ignoreifstate] [-M, –nomcast]

ex for master: /usr/sbin/ucarp -i eth0 -s 192.168.3.47 -v 1 -p tequila123 -a 192.168.3.31 -b 1 -k 10 -B -S -u /usr/bin/ucarpup.sh -d /usr/bin/ucarpdown.sh -P

ex for slave : /usr/sbin/ucarp -i eth0 -s 192.168.3.41 -v 1 -p tequila123 -a 192.168.3.31 -b 1 -k 10 -B -S -u /usr/bin/ucarpup.sh -d /usr/bin/ucarpdown.sh

Some points to note:

The Up-script is executed when the interface moves from slave to master
The down-script is executed when the interface moves from master to slave
The ‘-P’ ensures that the UCARP interface assumes master status.

Our Implementation of UCARP:

We first create a UCARPeth0.conf file and place it in the /etc/init

This is going to be our ucarp daemon.

Start on net-device-up IFACE = <iface name>

stop on shutdown

script

ucarp command for master/slave specific configuration.

end script

Then we create the following files:
- A ucarpeth0.conf in /etc to store session data that UCAReth0.conf reads
- A UCARP upscript and downscript in /usr/bin
- Test scripts in /usr/bin to switch over between master and slave
- A ucarp.log file in /var/log

The USR-2 Signal:
- Usr2 is a user defined signal whose lifetime lasts for 3 full seconds. When used as

kill -usr2 <pid of the process> ,

This defaults to a kill on that PID(Process ID).
Our ucarpup.sh and ucarpdown.sh:
- The ucarpdown.sh brings down the UCARP interface.
- The ucarpup.sh performs the following actions:
- if masterupflag file exists
  - It brings up the UCARP interface.
  - else
    - sleep for a while and repeat after 90 seconds.

The toslave command:
- This first removes the masterupflag file and then logs the events in ucarpeth0.log in /var/log
- Then it kills the UCARP (on master) by sending a USR2 signal.

The tomaster command :
- This creates the masterupflag file. This grants the machine permission to become master.

NOTE : tomaster and toslave scripts enable full control over the interfaces. They elegantly overcome ucarp’s drawback that being no control.

MASTER: SLAVE:

P.S. – The above contribution is by Dr.Ravi.S.Iyer. For queries, leave a comment on the blog.

Download the pdf: http://goo.gl/LEiAe