Kernel Modules and Configuration

The standard kernel supports SMP and 4GB of physical memory with a limit of 3GB per processor. The PAE kernel supports up to 16GB of RAM. The RHEL5 x86-64 kernel supports up to 256GB of Ram and 64 CPU's. The Xen kernel supports 64GB of physical RAM with a limit of 16GB per domain so you may allocate 16GB to Dom0 and 16GB to three additional DomU for example.

The kernel is stored in /boot/vmlinuz-....

The kernel modules are stored in /lib/modules/$(uname -r)/ and are loaded dynamically at will with support for 3rd party add-ons

Module Utilities:

• modprobe loads and unloads modules
• modinfo displays module info
• lsmod lists the currently loaded modules

Use modprobe <module> to install a module or modprobe -r <module> to remove a module

Use /etc/modprobe.conf to configure settings for common modules

In order to boot the system the kernel must load modules such as ext3, jbd, raid1, scsi_mod. Therefore, GRUB provides the modules in the form of an initrd image which is specified in the grub.conf file.

To create an initrd file examples ( use uname -r ):

• mkinitrd /boot/initrd-2.6.18-92.1.10.el5.img 2.6.18-92.1.10.el5
• mkinitrd /boot/initrd-$(uname -r).img $(uname -r)

To include modules example (scsi_mod):

• mkinitrd --with=scsi_mod /boot/initrd-2.6.18-92.1.10.el5.img 2.6.18-92.1.10.el5
• mkinitrd --with=scsi_mod /boot/initrd-$(uname -r).img $(uname -r)

To add qla2300 module and dependencies to initrd use:

echo "alias scsi_hostadapter qla2300" >> /etc/modprobe.conf
mkinitrd --with=scsi_mod /boot/initrd-$(uname -r).img $(uname -r)

Configuring the kernel with /proc can be useful for testing kernel configuration changes that will not persist across a reboot. The /proc filesystem can also be very useful for gathering system information i.e. cat /proc/meminfo or cat /proc/cpuinfo

To see info about running processes use: strings /proc/<PID>/* # replace <PID> with actual process id

Boot time options: cat /proc/cmdline

Dynamic modules: cat /proc/modules

Mounts: cat /proc/mounts

Network activity: strings /proc/net/*

Partitions: cat /proc/partitions

Turn IP Forwarding on: echo on > /proc/sys/net/ipv4/ip_forward

Free up cached memory: echo 1 > /proc/sys/vm/drop_caches

Adjust swap aggressiveness (1 - 100): echo 60 > /proc/sys/vm/swappiness

Making kernel configuration persistent across reboots

• sysctl -a lists all kernel parameters and values
• sysctl -w net.ipv4.tcp_syncookies=1 turns on syncookies and sysctl -p makes change permanent

Understanding Linux System Initialization

The BIOS is the initialization stage of the boot process that performs POST operations and determines what peripherals are available and which device to boot from. The BIOS obtains hardware configuration information from the CMOS which is where boot device order and other BIOS settings are stored. After the POST the selected boot device or the first boot device configured in CMOS will be booted from using the first sector of the boot media (Hard Disk, CDROM, network-adapter, floppy, usb device or removeable media).

In the case of the default RedHat installation using the GRUB bootloader, during Stage 1 of the boot process the BIOS passes control to the IPL installed in the MBR or boot sector. The second stage of the boot process involves loading the initrd image and kernel from the /boot partition based upon the /boot/grub/grub.conf configuration.

Example grub.conf:

#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux AS (2.6.9-67.0.20.EL)
       root (hd0,0)
       kernel /vmlinuz-2.6.9-67.0.20.EL ro root=/dev/sda2 rhgb quiet clock=pit noapic
       initrd /initrd-2.6.9-67.0.20.EL.img
title Red Hat Enterprise Linux AS (2.6.9-67.0.4.EL)
       root (hd0,0)
       kernel /vmlinuz-2.6.9-67.0.4.EL ro root=/dev/sda2 rhgb quiet clock=pit noapic
       initrd /initrd-2.6.9-67.0.4.EL.img
title Red Hat Enterprise Linux AS (2.6.9-42.0.10.EL)
       root (hd0,0)
       kernel /vmlinuz-2.6.9-42.0.10.EL ro root=/dev/sda2 rhgb quiet clock=pit noapic
       initrd /initrd-2.6.9-42.0.10.EL.img
title Red Hat Enterprise Linux AS (2.6.9-42.0.3.EL)
       root (hd0,0)
       kernel /vmlinuz-2.6.9-42.0.3.EL ro root=/dev/sda2 rhgb quiet clock=pit noapic
       initrd /initrd-2.6.9-42.0.3.EL.img
title Red Hat Enterprise Linux AS (2.6.9-34.EL)
       root (hd0,0)
       kernel /vmlinuz-2.6.9-34.EL ro root=LABEL=/ rhgb quiet clock=pit
       initrd /initrd-2.6.9-34.EL.img

To boot a Windows partition you could add an entry as follows provided you have a windows installation on Disk 0, Partition 1:

title Windows
       rootnoverify (hd0, 1)
       chainloader +1

To include a bootloader password:

 generate a password with - grub-md5-crypt
 add password --md5 <results of output from command above> to the top of grub.conf under hiddenmenu

To repair a corrupted bootloader in MBR on /dev/sda reinstall with:

 /sbin/grub-install /dev/sda

If grub-install fails try the following:

 grub
 root (hd0, 0)
 setup (hd0)
 quit

The kernel initialization now performs device detection, driver initialization, mounts the root filesystem and loads init which reads inittab and runs rc.sysinit. All the output from the kernel initialization can be viewed using /var/log/dmesg. If special drivers are required for boot and have been compiled as modules they must be included in the initrd image so that the kernel can mount the RAM disk to make the required modules available. For example, boot from SAN requires hba drivers to access the /boot filesytem on the SAN. Once the init process is loaded kernel passes control to it and the OS is now accessible.

To define the default runlevel for the system to boot edit the /etc/inittab file as follows:

change the line id:5:initdefault: substituting the desired run-level
2 multiuser no NFS
3 multiuser text, most common for servers
4 undefined
5 graphical login

To shutdown system: init 0

To reboot system: init 6

Single User Mode: s, S, or single

Emergency: bypasses rc.sysinit and prompts for root pw

Show current and recent run-levels: /sbin/runlevel

/etc/rc.d/rc.sysinit performs the following tasks - see the script for detailed info:

Activates udev (devices created dynamically) and selinux (security)
reads /etc/sysctl.conf for kernel parameters
sets the clock
loads keymaps
enables swap partitions
sets hostname
checks and remounts root filesystem rw
activates any raid or lvm devices that are present
enables disk storage quotas
checks and mounts any other local filesystems
cleans up locks and PID files

/etc/rc.d/rc defines services that should start based upon the default runlevel defined in /etc/inittab. If runlevel 3 is the default then rc will use /etc/rc.d/rc3.d/ directory as the basis for starting services. The service startup scripts are located in /etc/rc.d/init.d and each respective runlevel directory contains symbolic links to the files in the /etc/init.d directory beginning with S (Start) or K (Kill-Stop) and a number which identifies the Start/Kill order.

Required entries to create an init script:

First copy an existing init script in /etc/init.d as a baseline and edit the following
# chkconfig: - 85 15  # Runlevel (- for default)  StartNumber KillNumber
Manage a lock file with (example):
pidfile=${PIDFILE-/var/run/proc.pid}
lockfile=${LOCKFILE-/var/lock/subsys/proc}
RETVAL=0
stop() {
       echo -n $"Stopping $prog: "
       killproc $proc
       RETVAL=$?
       echo
       [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}
and verify the file contains:
# Source function library.
. /etc/rc.d/init.d/functions

/etc/rc.d/rc.local can be used for customizations that do not require a start and stop command but this is not recommended.

The xinetd service acts as a traffic cop for less used services. Xinetd provides host based access, logging, time based access, and address redirection. All port definitions are located in /etc/services and the default config file is in /etc/xinetd.conf which can be overridden by service specific files in the /etc/xinetd.d/ directory. See man 5 xinetd.conf for more detail.

Utilities:
chkconfig is a command line utility for managing services and acts similar to service for xinetd services - most commonly used
system-config-services is a  graphical utility for managing services
ntsysv is an ncurses based utility
service command line to to start,stop,restart,reload,status init script services located in /etc/init.d

To change runlevel from the grub bootloader:

1. Reboot the server and hit the space bar when the GRUB screen appears
2. Use arrow key to highlight the desired boot kernel and press the e key to edit the grub.conf
3. Use arrow key to highlight the kernel line and press the e key to edit the line
4. Hit the space bar and enter the desired runlevel 2,3,5 or s and hit Enter
5. Press the b key to boot using the new boot option

Yum and RPM Tips

To determine if a package is installed or use:

• yum list <package name> # shows installed and available packages
• rpm -qa |grep <package name> # shows installed packages only

To update a package:

• yum -y update <package name>
• rpm -Uvh <rpm file name>

If you are using rpm to install an updated kernel, always use the -ivh option; NEVER use -U or -F options with kernel installations

To update all installed packages:

• yum -y update

To verify availability of yum repository:

• yum list available

To find a package by partial name:

• yum list '*partial-name*'
• yum list installed '*partial-name*' # only list installed packages
• rpm -qa |grep partial-name

To determine what files belong to a package with rpm:

• rpm -ql <package name>

To determine the build host and installed size of packages:

• rpm -qi <package name>
• yum info <package name>

To verify an installed package:

• rpm -V <package name>

      The format of the output is a  string  of  9  characters,  a  possible
      attribute marker:
      c %config configuration file.
      d %doc documentation file.
      g %ghost file (i.e. the file contents are not included in the package payload).
      l %license license file.
      r %readme readme file.
      from  the  package  header,  followed by the file name.  Each of the 8
      characters denotes the result of a comparison of attribute(s)  of  the
      file  to  the value of those attribute(s) recorded in the database.  A
      single "." (period) means the test passed, while a single  "?"  (ques-
      tion  mark)  indicates the test could not be performed (e.g. file per-
      missions prevent reading). Otherwise,  the  (mnemonically  emBoldened)
      character denotes failure of the corresponding --verify test:
      S file Size differs
      M Mode differs (includes permissions and file type)
      5 MD5 sum differs
      D Device major/minor number mismatch
      L readLink(2) path mismatch
      U User ownership differs
      G Group ownership differs
      T mTime differs
      C selinux Context differs

To determine which package installed a file:

• rpm -qf <file name>
• For example, rpm -qf /etc/mail/sendmail.cf
• Output: sendmail-8.13.1-3.3.el4
• yum provides <file name>

Clean up directories recursively with SCOUR shell script

#!/bin/sh
#
# Deletes old data files.
#
# Recursively deletes files older than a specified number of days from a
# specified set of directories.  The directories, retention time in days,
# and an optional shell filename pattern appear, separated by tab characters
# one directory per line, in a configuration file named on the command line.
#
# If no files have been written under a directory since the last time scour was
# run, it will skip deleting old files and log an error message instead.
#
# WARNING: scour follows symbolic links, so don't put symbolic links to
# directories you don't want scoured under data directories.

PATH=/bin:/usr/bin
CONFFILE=/home/dblogs/bin/scour.conf                    # default, if no args
VERBOSEFLAG=
DEBUGFLAG=
ERRS=
LOGGER=echo
TZ=UTC0 export TZ

while [ "$1" != "" ]
do
        case "$1" in
        -v)
                VERBOSEFLAG=1
                ;;
        -x)
                DEBUGFLAG=1
                ;;
        -l)                                             # logs to syslogd
                LOGGER="logger -t `basename $0` -p local0.notice"
                ;;
        -*)
                $LOGGER "unrecognized flag ($1)"
                ERRS=1
                ;;
        *)
                if [ $# -ne 1 ]
                then
                        $LOGGER "only 1 conf file argument permitted"
                        ERRS=1
                fi
                CONFFILE=$1
                ;;
        esac
        shift
done

if [ "$ERRS" != "" ]
then
        $LOGGER "usage: `basename $0` [-l] [-v] [-x] [config-file]"
        exit 1
fi

if [ "$VERBOSEFLAG" != "" ]
then
        $LOGGER "Starting Up"
fi

while read dir age pattern; do
    if [ x$dir = x ]    # ignore blank lines
    then
        continue
    fi

    case $dir in
      #*)              # ignore comments
        continue
        ;;
      *)
        if [ x$pattern = x ]
        then
                pattern="*"
        fi

        # handle ~user expansion for home directory of user
        rdir=`echo echo $dir | csh -f`
        if [ $? != 0 ]; then
            $LOGGER "directory $dir does not exist, skipping"
            continue
        fi

        # expand symbolic link, since find doesn't follow symbolic links
        edir=`cd $rdir && /bin/pwd`
        if [ $? != 0 ]; then
            $LOGGER "directory $dir does not exist, skipping"
            continue
        fi

        if [ "$DEBUGFLAG" != "" ]
        then
            echo "dir=$dir age=$age pattern=$pattern edir=$edir"
        fi

        # if either "$edir/.scour$pattern" doesn't exist yet OR
        #           there are files newer than "$edir/.scour$pattern"
        # then
        #     delete old files and create "$edir/.scour$pattern"
        # else
        #     skip deletions and log message

        if  [ ! -f "$edir/.scour$pattern" ] ||
            (cd $edir; find . -newer "$edir/.scour$pattern" -print) | grep -s .
        then
            # find has a non-intuitive interpretation of age, subtract 1
            FINDAGE=`echo $age 1 - p|dc`
            if [ "$VERBOSEFLAG" != "" ]
            then
                    BEFORE=`du -s $edir|sed 's/^ *([0-9]*).*/1/'`
            fi
            (cd $edir;
             find . -mtime +$FINDAGE -name "$pattern" -exec rm -f {} ;)
             && touch "$edir/.scour$pattern"
            if [ "$VERBOSEFLAG" != "" ]
            then
                AFTER=`du -s $edir|sed 's/^ *([0-9]*).*/1/'`
                # set DELETED to max(0, BEFORE - AFTER)
                DELETED=`echo $BEFORE $AFTER|
                         sed 's/([0-9]*) ([0-9]*).*/b=1;a=2;d=0;if(a<b)d=b-a;d/'|
                             bc`
                $LOGGER "$DELETED blocks from $edir/$pattern ( > $age days old)"
            fi
        else
            $LOGGER "skipping, no recent files in $edir/$pattern"
        fi
        ;;
    esac
done < $CONFFILE

if [ "$VERBOSEFLAG" != "" ]
then
        $LOGGER exiting ...
fi


Example scour.conf file
/data/app/logs/app_inact/LOGS   7       app_act_log.[0-9]*
/data/app/logs/app_inact/LOGS   30      app_act_log.[0-9]*.gpg
/data/app/logs/app_prod1/LOGS   7       app_prod_log.[0-9]*
/data/app/logs/app_prod1/LOGS   30      app_prod_log.[0-9]*.gpg
/data/app/logs/app_web/LOGS     7       app_web_log.[0-9]*
/data/app/logs/app_web/LOGS     30      app_web_log.[0-9]*.gpg