Kdump over Ethernet or HFI for Linux diskless nodes

There is a newer version of this page. You can find it here.

This document illustrates how to configure and use kdump on the RHEL6 and SLES11 diskless nodes for xCAT.

[TOC]
==Overview==
kdump is an advanced crash dumping mechanism. When enabled, the system is booted from the context of another kernel. This second kernel reserves a small amount of memory, and its only purpose is to capture the core dump image in case the system crashes. Since being able to analyze the core dump helps significantly to determine the exact cause of the system failure.
==xCAT Interface==
The pkglist, exclude and postinstall files' location can be got by command 'lsdef -t osimage '
===The pkglist file===
For RHEL6, there are two rpm packages for kdump:
*'''kexec-tools'''

  • '''crash'''
    For SLES11, there are 3 rpm packages for kdump:
    '''kdump'''
    '''kexec-tools'''
    '''makedumpfile'''
    For SLES10, there are 4 rpm packages for kdump:
    '''kernel-kdump'''
    '''kexec-tools'''
    '''kdump'''
    *'''makedumpfile'''

They should be put into the pkglist file'.

Before setting up kdump, the user should add them to the specified pkglist file.

===The exclude file===
The base diskless image excludes the '''/boot''' directory, but it is required for kdump. Update the exlist file and remove the entry for '''/boot'''. Then run ''packimage'' or ''liteimg'' command.

===The postinstall file===
The kdump will create a new initrd which used in the dumping stage. The /tmp or /var/tmp directory will be used as the temp direcotory. These 2 directory only own 10M space by default, had to enlarge it to 200M.

For RHELS6 or SLES10, modify the postinstall file like this:
tmpfs /var/tmp tmpfs defaults,size=200m 0 2

For SLES11, modify the postinstall file like this:
tmpfs /tmp tmpfs defaults,size=200m 0 2

===The ''dump'' attribute===
In order to support kdump, the ''dump'' attribute was added into '''linuximage''' table, which is used to define the remote path where the crash information should be dumped to. Use the chdef command to change the image's dump attribute using the URI format.

chdef -t osimage dump=nfs://<nfs_server_ip>/<kdump_path></kdump_path></nfs_server_ip>

The '''<nfs_server_ip>''' can be excluded if the destination NFS server is the service or management node.</nfs_server_ip>

chdef -t osimage dump=nfs:///<kdump_path></kdump_path>

===The ''crashkernelsize'' attribute===
For system x machine, on sles10 set the crashkernelsize attribute like this:
chdef -t osimage crashkernelsize=<size>M@16M</size>

On sels11 and rhels6 set the crashkernelsize attribute like this:
chdef -t osimage crashkernelsize=<size>M</size>

the size recommended value is 256. For more information about the size can refer [https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/ch-kdump.html#s2-kdump-configuration-cli kdump for rhels5 on system x], [http://www.novell.com/support/kb/doc.php?id=3374462 kdump for sles10 on system x], [https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s2-kdump-configuration-cli.html kdump on rhels6 on system x], and [http://www.novell.com/support/kb/doc.php?id=3374462 kdump for sles11 on system x]

For system p machine, set the crashkernelsize attribute like this:
chdef -t osimage crashkernelsize=<size>@32M
the size recommended value is 256, more information can refer the kdump document for the system x.</size>

When your node start, and meet the kdump start error like this
Your running kernel is using more than 70% of the amount of space you reserved for kdump, you should consider increasing your crashkernel
You should modify this attribute by chdef command:
chdef -t osimage crashkernelsize=512M@32M
If 512M@32M is not large enough, you should change the crashkernelsize larger like 1024M untill the error message disappear.

===The enablekdump postscript===
This postscript ''enablekdump'' is newly introduced, which is used to start the kdump service when the node is booting up.
chdef -t node <node range=""> -p postscripts=enablekdump</node>

===The litefile table===
This is for '''statelite''' only. When the ''kdump'' service is starting, one special initial ramdisk for kdump will be generated if not exist. The initial ramdisk will be put into the "/boot/" directory.

As a result, for RHEL6 statelite node, '''/boot/''' should be added into the ''litefile'' table. The config file (''/etc/kdump.conf'') will be updated in the ''enablekdump'' postscript, so we need to add '''/etc/kdump.conf''' to the ''litefile'' table.

#image,file,options,comments,disable
"ALL","/boot/",,,
"ALL","/etc/kdump.conf",,,

For SLES11 statelite node,

''/lib/mkinitrd/'' will be updated in the ''enablekdump'' postscript(the workaround for kdump on sles diskless install node), so we need to add '''/lib/mkinitrd/''' and '''/boot/''' to the ''litefile'' table.

#image,file,options,comments,disable
"ALL","/boot/",,,
"ALL","/lib/mkinitrd/",,,

===Notes===
Currently, only ''NFS'' is supported for the setup of kdump. ''scp'' is planed to be supported in the feature.

If the ''dump'' attribute is not set, the kdump service will not be enabled.

Please make sure the NFS remote path('''nfs://<nfs_server_ip>/<kdump_path>''') is exported and it is read-writeable to the node where kdump service is enabled.</kdump_path></nfs_server_ip>

===How to trigger kernel panic on Linux===

Normally, kernel panic() will trigger booting into capature kernel. Once the kernel panic is triggered, the node will reboot into the capture kernel, and a kernel dump (''vmcore'')will be automatically saved to the directory on the specified NFS server (''<nfs_server_ip>'').</nfs_server_ip>

For RHESL6 the directory is '''<kdump_path>/var/crash/<node_ip>-</node_ip></kdump_path>

For SLES11 the directory is '''<kdump_path>/<node hostname="">/<date>'''</date></node></kdump_path>

For SLES10 the directory is '''<kdump_path>/<node hostname="">'''</node></kdump_path>

For RHELS6 testing purposes, you can simluate the tirgger through /proc interface:
echo c > /proc/sysrq-trigger
For SLES11.1 testing, you can use the following command:
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

This will force the Linux kernel to crash, and the address-YYYY-MM-DD-HH:MM:SS/vmcore file will be copied to the location you have selected on the specified NFS server directory.

===Dump Analysis===
Once the system has returned from recovering the crash, you may wish to analyse the kernel dump file using the crash tool.

  1. Locate the recent vmcore dump file.

  2. Locate the kernel file for the crash server(the kernel is under /tftpboot/xcat/netboot/<os name="">/<arch>/<profile>/kernel on management node).</profile></arch></os>

  3. One you have located a vmcore dump file and kernel file, call crash
    crash <vmcore dump="" file=""> <kernel file=""></kernel></vmcore>

If crash cannot find any files under /usr/lib/debug? Make sure you have the kernel-debuginfo package installed.

For more information about the dump analysis you can refer the [http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/s1-kdump-crash.html RHEL document] or [http://www.novell.com/support/kb/doc.php?id=3374462 SLES document]

MongoDB Logo MongoDB