How to configure and analize Kdump in RHEL 6

Verify the kexec-tools package is installed:
# rpm -q kexec-tools
kexec-tools-2.0.0-245.el6.x86_64
If it is not installed, proceed to install it via yum:
# yum install kexec-tools

Specifying the Kdump Location

Kdump can be configured to dump directly to a device by using below mount points  in /etc/kdump.conf.
# vi /etc/kdump.conf


#raw /dev/sda5
#ext4 /dev/sda3
#ext4 LABEL=/boot
#ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
#net my.server.com:/export/tmp
#net user@my.server.com
#path /var/crash
#core_collector makedumpfile -c --message-level 1 -d 31
#core_collector cp --sparse=always
#link_delay 60
#kdump_post /var/crash/scripts/kdump-post.sh
#extra_bins /usr/bin/lftp
#disk_timeout 30
#extra_modules gfs2
#options modulename options
#default shell

Here we have configured with local partition  that is /dev/sdc1.

# df -hT

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root

              ext4     13G  9.0G  2.8G  77% /

tmpfs        tmpfs    182M   23M  160M  13% /dev/shm

/dev/sda1     ext4    485M   37M  423M   8% /boot

/dev/sdc1     ext4    5.0G  138M  4.6G   3% /kdump

Below settings will help you to configure kdump in local mounted partition as this has been done above in /kdump mount point.

# vi /etc/kdump.conf

#raw /dev/sda5
ext4 /dev/sdc1
ext4 LABEL=/kdump
ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
#net my.server.com:/export/tmp
#net user@my.server.com
#path /var/crash
#core_collector makedumpfile -c --message-level 1 -d 31
#core_collector cp --sparse=always
#link_delay 60
#kdump_post /var/crash/scripts/kdump-post.sh
#extra_bins /usr/bin/lftp
#disk_timeout 30
#extra_modules gfs2
#options modulename options
#default shell

After the changes done then save and exit and run below two commands.

# e2lable /dev/sdc1 /kdump
# service kdump restart

So now kdump location has been configured  

Testing the Configuration

Since there is no kernel panic or no issue we need to force the Linux kernel to crash, and the YYYY-MM-DD-HH:MM/vmcore file will be copied to the location you have selected in the configuration (that is, to /var/crash by default but here we have customize the path from kdump.conf).

# echo 1 > /proc/sys/kernel/sysrq
# echo c > /proc/sysrq-trigger

You need to wait for few minutes  as the system will reboot and then dump file will be created
After Rebooted then you can check the  YYYY-MM-DD-HH:MM/vmcore will be there.

Kdump Analyze

To analyze the vmcore dump file, you must have the crash and (kernel-debuginfo,kernel-debuginfo-common) packages installed. To install the crash package in your system, type the following at a shell prompt as root:

#yum install crash

Note that in order to use this command, you need to have access to the repository with debugging packages.
kernel-debuginfo,kernel-debuginfo-common packages only available in Redhat Repo.

Running the crash Utility

#crash /usr/lib/debug/lib/modules/`uname -r`/vmlinux     /kdump/YYYY-MM-DD-HH:MM/vmcore
From crash prompt you can type below command for more analysis.
Displaying the Message Buffer
crash> log
Displaying the kernel stack trace.
crash> bt
Displaying status of processes in the system.
crash> ps
Displaying virtual memory information of the current context
crash> vm
Displaying information about open files of the current context
crash> files
Exiting the crash utility
crash> exit