HP Serviceguard
HP Serviceguard was the very first solution for high availability clustering software. Serviceguard uses resource virtualization to ensure that critical applications, databases, or services are continually up and running. Users typically don’t care which server they run their application on—they just want to make sure that application is running. Serviceguard hides the details of which server an application is actually running on at any given time, which means it can move applications between servers on the fly.
What is a Serviceguard cluster?
Any host system running in a Serviceguard cluster is called an active node. Under normal conditions, a fully operating Serviceguard cluster monitors the health of the cluster’s components on all its active nodes.Most Serviceguard packages are failover packages. When you configure a failover package, you specify which active node will be the primary node where the package will start, and one or more other nodes, called adoptive nodes, that can also run the package.
Creating a Serviceguard cluster
Before you configure the cluster, you will need to create a volume group on the shared disks, so that Service Guard can create a “Cluster Lock Disk”.
On testnode1
# pvcreate /dev/rdsk/c0t4d0
# pvcreate /dev/rdsk/c1t4d0
# mkdir /dev/vg01
# mknod /dev/vg01/group c 64 0×010000
# vgcreate vg01 /dev/dsk/c0t4d0 /dev/dsk/c1t4d0
# mkdir /etc/cmcluster/maps
# vgexport –p –s –m vg01 /etc/cmcluster/maps/vg01
On testnode2
# mkdir /dev/vg01
# mknod /dev/vg01/group c 64 0×010000
# mkdir /etc/cmcluster/maps
# rcp testnode1:/etc/cmcluster/maps/vg01 /etc/cmcluster/maps/vg01
# vgimport –s –m /etc/cmcluster/maps/vg01 vg01
# vgchange –a y vg01
# vgchange –a n vg01
You can now go ahead and create the cluster.
# cd /etc/cmcluster
# cmquerycl –v –C cmclconfig.ascii –n testnode1 -n testnode2
After creating this file, you will need to edit it to suit your cluster. You will need to
alter the following lines:
# vi cmclconfig.ascii
CLUSTER_NAME simeon1
NODE_NAME testnode1
NETWORK_INTERFACE lan0
HEARTBEAT_IP 192.168.100.101
NETWORK_INTERFACE lan1
HEARTBEAT_IP 164.39.8.92
NETWORK_INTERFACE lan2
NODE_NAME testnode2
NETWORK_INTERFACE lan0
HEARTBEAT_IP 192.168.100.102
NETWORK_INTERFACE lan1
HEARTBEAT_IP 164.39.8.93
NETWORK_INTERFACE lan2
HEARTBEAT_INTERVAL 2000000
NODE_TIMEOUT 5000000
MAX_CONFIGURED_PACKAGES 10
To check that there are no mistakes in the file, use the check command.
# cmcheckconf –C /etc/cmcluster/cmclconfig.ascii
To apply the configuration file, usr the apply command.
# cmapplyconf –C /etc/cmcluster/cmclconfig.ascii
You are now ready to startup the cluster.
On testnode1
# vgchange –a n vg01
# cmruncl –v
# cmviewcl –v
# vgchange –c y vg01
On Both nodes
# netstat -in
# vi /etc/rc.config.d/cmcluster
AUTOSTART_CMCLD=1
On testnode1
# cmhaltcl –v
# vgchange –a n vg01
Creating a basic package
Serviceguard packages group together applications and the services and resources they depend on. The typical Serviceguard package is a failover package that starts on one node but can be moved (“failed over”) to another if necessary.
Creating or modifying a package requires the following broad steps:
- Generate the package ASCII file
- Edit the package ASCII file
- Generate the package control script
- Edit the package control script
- Distribute the control script to the cluster nodes
- Apply the package ASCII file
Once the cluster is up and running, we can create a package to run in it. This package will run an “xclock” on a client PC, and mount a file system, using the IP Address 164.39.8.101 The first step is to make a logical volume.
On node1
# vgchange –a e vg01
# lvcreate –L 20 vg01
# lvextend –m 1 /dev/vg01/lvol1
# newfs –F vxfs /dev/vg01/rlvol1
# mkdir /pkg_1
# vi /etc/fstab
#/dev/vg01/lvol1 /pkg_1 vxfs delaylog 0 2 # MC/SG
# vgchange –a n vg01
# vgexport –p –s –m vg01 /etc/cmcluster/maps/vg01
On Node 2
# vgexport vg01
# mkdir /dev/vg01
# mknod /dev/vg01/group c 64 0×010000
# vgimport –s –m /etc/cmlcuster/maps/vg01 vg01
# mkdir /pkg_1
# vi /etc/fstab
#/dev/vg01/lvol1 /pkg_1 vxfs delaylog 0 2 # MC/SG
Now we can make the package.
On Node1
# cd /etc/cmcluster
# mkdir pkg_1
# cd pkg_1
# cmmakepkg –p pkg_1.conf
The file pkg_1.conf will need to be edited to suit.
# vi pkg_1.conf
PACKAGE_NAME pkg_1
NODE_NAME testnode1
NODE_NAME testnode2
RUN_SCRIPT /etc/cmcluster/pkg_1/pkg_1.cntl
RUN_SCRIPT_TIMEOUT NO_TIMEOUT
HALT_SCRIPT /etc/cmcluster/pkg_1/pkg_1.cntl
HALT_SCRIPT_TIMEOUT NO_TIMEOUT
SERVICE_NAME pkg_1_service
SUBNET 164.39.8.0
Add the IP address that will be used for the package in the hosts file.
# vi /etc/hosts
164.39.8.101 pkg_1 xclock
# cmmakepkg –s pkg_1.cntl
The file pkg_1.cntl will need to be edited to suit
# vi pkg_1.cntl
VG[0]=”vg01″
LV[0]=”/dev/vg01/lvol1″; FS[0]=”/pkg_1″; FS_MOUNT_OPT[0]=””
IP[0]=”164.39.8.101″
SUBNET[0]=”164.39.8.0″
SERVICE_NAME[0]=”pkg_1_service”
SERVICE_CMD[0/etc/cmcluster/pkg_1/service_pkg_1″
#SERVICE_RESTART[0]=””
function customer_defined_run_cmds
{
# ADD customer defined run commands.
if [ $(hostname) = “testnode1” ]
then COLOUR=”blue”
else COLOUR=”red”
fi
/usr/bin/X11/xclock -bg “$COLOUR” -update 1 –display \
164.39.11.232:0.0 &
test_return 51
}
function customer_defined_halt_cmds
{
# ADD customer defined halt commands.
if
ps -ef |grep -v grep |grep -q “xclock”
then
kill -9 $(ps -ef |grep -v grep |grep “xclock”|cut -c10-14)
fi
test_return 52
}
# vi pkg_1_service
#!/usr/bin/sh
#This is a script to monitor that xclock is running
while true
do
if
ps -ef |grep -v grep |grep -q “xclock”
then
sleep 5
else
echo “Package xclock failed at $(date) from node \
$(hostname) > /etc/cmcluster/pkg_1/pkg_1.cntl.log”
exit 99
fi
done
# chmod u+x pkg_1_servce
# cd /etc/cmcluster
# rcp –r testnode1:/etc/cmcluster/pkg_1
On node 1
# cmcheckconf -P pkg_1.conf
# cmapplyconf –P pkg_1.conf
# cmmodpkg –e –n testnode1 -n testnode2 pkg_1
# cmviewcl –v –p pkg_1
# cmhaltpkg pkg_1
# cmviewcl –v|more
# tail /etc/cmcluster/pkg_1/pkg_1.log
# tail /var/adm/syslog/syslog.log
Starting and stopping a Serviceguard Package
Normally, when a cluster starts up, the packages configured as part of the cluster will start up on their configured nodes. You may need to start a package manually after it has been halted manually. You can do this either in Serviceguard Manager or on the Serviceguard command line.
If you want to stop a package, use the following commands
# cmviewcl –v –p <package name>
# cmahalt <package name>
# cmviewcl –v –p <package name>
You halt a Serviceguard package when you wish to bring the package out of use but wish the node to continue in operation. You can halt a package using Serviceguard Manager or on the Serviceguard command line. Halting a package has a different effect than halting the node. When you halt the node, its failover packages may switch to adoptive nodes.
If you want to start a package, then use the following commands
# cmviewcl –v –p <package name>
# cmmodpkg –e <package name>
OR
# cmrunpkg –n <host name> <package name>
# cmmodpkg –e <package name>
Deleting a Serviceguard Package
To delete a package, use the following commands
# cmdeleteconf -p <package name>
Stopping and starting the Serviceguard Cluster
The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running.
To stop the cluster, use the following commands
The cmhaltcl command can be used to halt the entire cluster. This command causes all nodes in a configured cluster to halt their Serviceguard daemons. You can use the -f option to force the cluster to halt even when packages are running.
To stop the cluster, use the following commands
# cmiewcl –v
# cmhaltcl
OR
# cmhaltcl –f
To start the cluster: –
# cmruncl –v
# cmviewcl –v
Automatically Restarting the Serviceguard Cluster
You can configure your cluster to automatically restart after an event, such as a long-term power failure, which brought down all nodes in the cluster. This is done by setting AUTOSTART_CMCLD to 1 in the
/etc/rc.config.d/cmcluster file.
Stopping and starting a node in the Serviceguard Cluster
Starting a node will not cause any active packages to be moved to the new node. However, if a package is DOWN, has its switching enabled, and is able to run on the new node, that package will automatically run there.
To stop a node in the cluster
# cmhaltnode –n <node name>
# cmviewcl –v
# vi /etc/rc.config.d/cmcluster
AUTOSTART_CMCLD=0
To start a node in the cluster.
# cmrunnode –n <name name>
# cmviewcl –v
# vi /etc/rc.config.d/cmcluster
AUTOSTART_CMCLD=1
Deleting a cluster
When there are no packages left in the cluster, and you want to delete the cluster, use the following commands.
# vgchange –c n vg01
# cmhaltcl –f
# cmdeleteconf –c <cluster_name>
# vgchange –a y vg01
# lvremove /dev/vg01/lvol1
# vgremove vg01
# rm –r /dev/vg01