Centralized YUM Repository Unix Patch...

Centralized YUM Repository Unix Patch Management

Repository Locations

The PVALENTINO YUM repositories live on PATCH.PVALENTINO.ORG

All repositories live under /data01/repository/. The only sub-directory currently present is patch/, which is where all normal package updates go.

The patch directory structure is demonstrated by this example:

 

/data01/repository/patch/q3/rhel5-x86_64/

Where:

/data01/repository/ is the root of all repositories

patch/ is where all system updates go

q3/ indicates that these packages are for the 3rd Quarter scheduled patching

rhel5-x86_64/ contains the packages for Red Hat Enterprise Linux 5 for the 64bit edition

 

Under rhel5-x86_64 (in the example) would contain the actual repository metadata and all the RPM package files. You cannot just place the RPM files into a directory and call it a repository -- you must set up the YUM metadata for that directory before it can be used.

Solaris patch clusters and AIX fix files should be placed in logically named directories underneath the patch date directory (q3 in this example). For Solaris, the name should be something like solaris8 for Solaris 8. For AIX, it should be aix_51 for AIX 5.1. See the Appendices for Solaris and AIX patching procedures.

 

Creating a Patch Repository

To create the repositories for a scheduled quarterly patch cycle, we will first need to obtain all the current packages for each release of Red Hat Linux and for each platform that it runs on in our environment.

Most of our systems are connected to the Red Hat Network (RHN), and one of the services it provides is to provide the up-to-date packages for download. RHN is designed to only allow each system to download packages for its own release and platform; however, there is a workaround for this problem. On PATCH there is a utility called "rhnget", which can impersonate other systems registered through RHN and download the appropriate packages for their respective release and platform.

There is a directory on PATCH, /data01/systemid/, which contains the system identification files from one representative system for each release and platform in our environment:

 

rhel4-i386.systemid RHEL 4 i386

rhel4-x86_64.systemid RHEL 4 x86-64

rhel5-x86_64.systemid RHEL 5 x86-64

rhel5-i386.systemid RHEL 5 i386

 

The rhnget utility must impersonate a system that has an active entitlement to RHN. If any of these systems change or are decommissioned, you will have to copy the system identification file from another system of the same release and platform that is also entitled with RHN. Register a system for each platform with RHN with up2date --register (if you get error follow steps to import the key into rpm then retry the registration). Once a system is registered with RHN the system identification file can be found on the system at /etc/sysconfig/rhn/systemid, simply copy this file over to PATCH in the /data01/systemid/ folder and rename it using the convention above.

 

To fetch the packages for each release/platform, log into PATCH as root and do the following:

 

Manual Process :

Create and change to the directory for the quarter:

mkdir /data01/repository/patch/q3 ; cd /data01/repository/patch/q3

Also, change the 'current' symlink, so that the we can always access the current patchset with the same URL:

rm /data01/repository/patch/current

ln -s /data01/repository/patch/q3 /data01/repository/patch/current

 

Clean up an existing repository:

yum clean all

yum clean metadata

yum clean dbcache

yum makecache

Retrieve the packages for each release/platform (the directories will be automatically created). Use this exact script, in order to get the additional package channels that some systems are subscribed to:

rhnget --systemid=/data01/systemid/rhas21-i386.systemid rhn:///redhat-advanced-server-i386 ./rhas21-i386/

rhnget --systemid=/data01/systemid/rhel3-i386.systemid rhn:///rhel-i386-as-3 ./rhel3-i386/

rhnget --systemid=/data01/systemid/rhel4-i386.systemid rhn:///rhel-i386-as-4 ./rhel4-i386/

rhnget --systemid=/data01/systemid/rhel4-i386.systemid rhn:///rhel-i386-as-4-appstk-1 ./rhel4-i386/

rhnget --systemid=/data01/systemid/rhel4-x86_64.systemid rhn:///rhel-x86_64-as-4 ./rhel4-x86_64/

rhnget --systemid=/data01/systemid/rhel4-ppc.systemid rhn:///rhel-ppc-as-4 ./rhel4-ppc/

(You will probably want to create a simple script to run the commands, as downloading all the files will take several hours. Scripting it will let you run all of them without having to check on it constantly. Make sure to check the output occasionally to make sure that there weren't any problems during the downloads.)

Once all the files are downloaded, you can turn the directories into YUM repositories. This step will scan the RPMs in the directory and collect information from them, including their versions and dependencies.

In each of the directories under where you ran the rhnget command (e.g., /data01/repository/patch/q3/) you will need to run the following three commands:

cd /data01/repository/patch/q3/[directory]/

createrepo -v /data01/repository/patch/q3/[directory]/

Or, for older versions of Linux use:

yum-arch

(Replace [directory] with the name of the directory you are working on.)

The createrepo command creates the new-style of YUM metadata, used by newer versions of YUM. The yum-arch command creates the old-style of YUM metadata, required by older versions of YUM. For instance, RHEL 3 systems will not run the newer versions of YUM.

Automated Process Script:

#!/bin/bash

if [ "$1" = "" ]

then

exit 1

fi

if [ ! -d /data01/repository/patch/$1 ]; then mkdir /data01/repository/patch/$1 ; fi

cd /data01/repository/patch/$1

rm -f /data01/repository/patch/current

ln -s /data01/repository/patch/$1 /data01/repository/patch/current

result="1"

count="0"

while [ $result != 0 ] && [ $count -lt 3 ]

do

/usr/bin/rhnget -vvv --systemid=/data01/systemid/rhel4-i386.systemid rhn:///rhel-i386-as-4 ./rhel4-i386/ 2>&1 | mail -s "$1 rhel4 patch download status" pvalentino@sysxperts.com

result=$?

count=`expr $count + 1`

done

if [ $result = 0 ]; then /usr/bin/createrepo -v ./rhel4-i386;fi

result="1"

count="0"

while [ $result != 0 ] && [ $count -lt 3 ]

do

/usr/bin/rhnget -vvv --systemid=/data01/systemid/rhel4-x86_64.systemid rhn:///rhel-x86_64-as-4 ./rhel4-x86_64/ 2>&1 | mail -s "$1 rhel4 x86_64 patch download status" pvalentino@sysxperts.com

result=$?

count=`expr $count + 1`

done

if [ $result = 0 ]; then /usr/bin/createrepo -v ./rhel4-x86_64;fi

result="1"

count="0"

while [ $result != 0 ] && [ $count -lt 3 ]

do

/usr/bin/rhnget -vvv --systemid=/data01/systemid/rhel5-x86_64.systemid rhn:///rhel-x86_64-server-5 ./rhel5-x86_64 2>&1 |mail -s "$1 rhel5-x86_64 patch download status" pvalentino@sysxperts.com

result=$?

count=`expr $count + 1`

done

if [ $result = 0 ]; then /usr/bin/createrepo -v ./rhel5-x86_64;fi

exit

Then create a cron job to run each quarter to download the patches

0 20 21 2 * /root/patchdownload.sh q1

0 20 3 6 * /root/patchdownload.sh q2

0 20 3 9 * /root/patchdownload.sh q3

0 20 3 11 * /root/patchdownload.sh q4

 

Congratulations, you have set up the repositories. The next step is to create the configuration files used to update the systems.

Creating YUM Configurations

These are the files used by YUM on the servers to define their package distribution and where to locate the packages. One is required for each release/platform defined.

The files should be located in the quarterly patch directory, and should be named after their directory, with a .conf on the end. For example, if you have:

 

/data01/repository/patch/current/rhel5-x86_64/

you would create the following configuration file:

/data01/repository/patch/current/rhel5-x86_64.conf

 

The file contents are fairly simple. Copy between the BEGIN FILE and END FILE markers:

 

-----BEGIN FILE-----

[main]

cachedir=/var/cache/yum

debuglevel=2

logfile=/var/log/yum.log

pkgpolicy=newest

distroverpkg=update

tolerant=1

exactarch=1

retries=20

#exclude=kernel*

[update]

name=PVALENTINO Patching - rhel5-x86_64

baseurl=http://PATCH.PVALENTINO.ORG/repository/patch/q3/rhel5-x86_64/

-----END FILE-----

 

Replace the quarter and release/platform information in the [update] section to reflect the correct information. The baseurl parameter is the web URL where the release/platform RPMs and repository metadata are located on the PATCH server.

 

Using the Repositories

Web Access

You can access the PATCH repositories internally via HTTP, at:

http://PATCH.PVALENTINO.ORG/repository/

 

Updating Systems

If you haven't updated a system using this method before, you will need to install YUM onto the system first for RHEL 4 or older. RHEL 5 includes Yum and necessary packages. Packages for Red Hat 2.1, 3, and 4 are available on PATCH. Install it with the command:

 

rpm -ivh http://PATCH.PVALENTINO.ORG/repository/yum/yum-package

Where yum-package is one of:

yum-1.0.3-0.1.el2.rf.noarch.rpm for RHAS 2.1

yum-2.0.8-0.1.el3.rf.noarch.rpm for RHEL 3

 

These packages are platform-independent (they're just Python scripts), so they will work on i386, PowerPC, or whatever else. The exception is for RHEL4, which has package dependencies. For RHEL4 systems, you will need to download one of the following:

 

yum-rhel4-i386.tar.gz for RHEL 4 on i386

yum-rhel4-ppc.tar.gz for RHEL 4 on PowerPC (PPC)

yum-rhel4-x86_64.tar.gz for RHEL 4 on x86_64

 

Untar the appropriate file and install all the RPMs it contains together (e.g. rpm -ivh *.rpm)

 

Once YUM is installed, you only have to issue the following command to start the update process:

yum -c http://PATCH.PVALENTINO.ORG/repository/patch/<quarter>/<release>.conf -y update

or just yum -y update with the appropriate yum.conf in /etc

 

Replace quarter with the quarter information (e.g. q3) or the current keyword, and release with the release/platform information (e.g. rhel5-x86_64)

If you want to update a system without updating its kernel packages (necessary on some systems using kernel modules tied to a specific kernel version), run the following instead:

 

yum -c http://PATCH.PVALENTINO.ORG/repository/patch/quarter/release.conf --exclude kernel* -y update

 

YUM will fetch the package information from PATCH, determine dependency requirements, and then download the packages and install them. If a new kernel version is to be installed, it will install the new kernel (not replace the old one) and make it the default kernel in /etc/grub.conf. Switching back to the old kernel is just a matter of editing /etc/grub.conf.

 

When YUM completes, reboot the system:

Init 6

 

Monitor the system console to ensure that it boots up without any issues. Make sure that all services on the system start up correctly.

If you are patching a VMWare virtual server and the kernel has been updated, you will probably need to reinstall the VMWare Tools package and reboot again in order to have the network modules properly rebuilt for the new kernel. See Appendix D for instructions on reinstalling the VMWare Tools package.

If the system is connected to the Red Hat Network, you will need to send the updated system package information to Red Hat, so it can correctly track what packages are on the system and alert us to any updates or security issues. Run the following as root on the system:

 

up2date -p

 

This completes the system patching process.

 

Troubleshooting

If, when rebooting the patched system where the kernel was upgraded, the system panics on startup, then it is possible that the system uses specific kernel modules for the disk subsystem. What you'll need to do is reboot the system to the old kernel (select it from the GRUB bootloader), and remake the ramdisk image that the new kernel uses for booting the system so that it includes the necessary kernel modules. (See also: mkinitrd man page)

Boot to the previous kernel version and log into the system as root. Go to the /lib/modules/ directory, and you'll see directories for each version of the kernel installed on the system. Basically, you need to find out what modules are in the currently booted kernel's version that aren't in the new kernel's version, and copy them over (maintaining the directory structure inside the directories.)

In general, modules built on one patch revision will work on a newer patch revision (i.e. 2.4.21-foo should work on 2.4.21-bar). A module from a 2.4 kernel, however, will almost certainly not work on a 2.6 kernel.

Once the modules are copied, change to the /boot directory. Make a backup of the initrd file for the new kernel. Then, run the mkinitrd command for the new kernel:

 

mkinitrd initrd-(kernel).img (kernel)

mkinitrd initrd-$(uname -r).img $(uname -r)

Where (kernel) is the version string of the new kernel (e.g., 2.4.21-47.0.1.ELsmp which is output with `uname –r` command). Once completed, try rebooting the system again.

 

PATCH System Build

The following are the specifications for PATCH.PVALENTINO.ORG. This is a fairly basic Red Hat installation, with only Apache as its major service.

VMWare:

One CPU

512mb memory

One network adapter

connected to VLAN1

One 16gb virtual disk

root, 4gb

swap, 1gb

/boot, 150mb

/var, 4gb

/usr, remaining space (~6.74gb)

One 100gb virtual disk

Set up entirely as a LVM volume, mounted on /data01

OS:

Running Red Hat Enterprise Linux 4 update 8

Apache:

Apache should run at startup.

 

Appendix A: AIX and Solaris Patching

 

AIX Patching

PVALENTINO has only three AIX systems: aixapp, aixtest, and aixdb. These all currently run AIX 5.1. The following describes how to determine what patches need to be applied, how to obtain them, and how to apply them.

Getting the Patches

You first need to obtain the list of latest system fileset versions from IBM. You can download the list for AIX 5.1 from:

http://www.ibm.com/eserver/support/fixes/fixcentral/fixinfofiledownload?file=LatestFixData51

Log into aixapp as root, create a temporary directory (i.e. mkdir /tmp/patchwork), and change to that directory. Copy the LatestFixData51 file that you downloaded into this directory. Then run:

/usr/sbin/compare_report -s -r /tmp/patchwork/LatestFixData51 -l

This will generate two files: /tmp/lowerthanmaint.rpt and /tmp/lowerthanlatest1.rpt. The lowerthanlatest1.rpt file is what we need. Copy it to your workstation, and then go to:

http://www.ibm.com/eserver/support/fixes/fixcentral/comparereport?system=2&type=1&package=5&release=51&tab=0

That page lets you upload the lowerthanlatest1.rpt file, and get a customized list of what fileset updates you need. Click the Browse button on the page, select the lowerthanlatest1.rpt file, and then click Submit. On the next page, make sure all three checkboxes are selected. Select the operating system revision from the dropdown (use the oslevel -r command on lmsappdev to determine this), and then click Continue.

You will now be presented with a list of fileset updates. There should be a link at the top to "Download all filesets using the ftp command". Click on that, and then click Continue in the window that pops up. Follow the instructions to download all the .bff files -- it is recommended that you do this directly on PATCH, since the files should be stored in the repository there.

 

Installing the Patches

First thing you should do on your target system is to check to make sure that any previously applied patches were committed correctly. Log into the system as root and run installp -s. This will show all software updates that are applied but not committed. If nothing is returned, then you're ready to go. Otherwise, you will need to apply the previous updates by running installp -c all (as the root user.)

Create a directory in a filesystem on the target machine that has enough space, change to that directory, and transfer the previously downloaded .bff files to there. To begin the patching, run the following command as root:

 

installp -aX -d . all

 

After all the patches are installed, the system must be rebooted. If possible, monitor the system console, and make sure the system comes back up normally. If you don't have access to the system console, be advised that it can take 10-15 minutes for the system to become accessible again.

Two weeks after the patches are applied, you need to commit them permanently to the system. Run the following as root:

 

installp -c all

 

No reboot is required for running the commit command.

 

Solaris Patching

PVALENTINO only has three Solaris systems: sun1, sun2, and sun3. These are all running Solaris 8 on the SPARC architecture.

Getting the Patch Cluster

With Solaris, you don't need to fiddle with figuring out what software is on your system: you just download a patch cluster, unzip it, and run it. The only trick is getting the patch cluster in the first place.

Go to http://sunsolve.sun.com/ and log in. If you do not have an account, you will need to create one. Sun requires that you have a SunSolve account before they'll let you download the patch cluster.

Once you're logged in, go back to the main SunSolve page. Select PatchFinder. Under 'Recommended Solaris Patch Clusters', scroll down and select 'Solaris 8' (NOT 'Solaris 8 x86' or 'Solaris 8 Sun Alert Patch Cluster'). Select the 'Download HTTP' radio button, and the click 'Go'. You will now download the 8_Recommended.zip file. Once that is downloaded, transfer that to the PATCH repository.

 

Installing the Patch Cluster

Create a directory in a filesystem on the target machine that has enough space, change to that directory, and transfer the previously downloaded 8_Recommended.zip file there. Unzip it with the command:

unzip -qq 8_Recommended.zip

Its a large file, and these aren't very fast systems, so unzipping may take a while. Once unzipped, remove the 8_Recommended.zip file.

You must have the system console (aka serial console) to patch, since you need to reboot the system to single-user mode before patching -- you will not be able to ssh into the system.

When you are ready to patch, log into the system via the system console, shutdown any applications, and reboot to single-user mode with the command:

 

reboot -- -s

 

The system will reboot, and then ask for the root password to enter maintenance mode. Enter the root password, and you will be at the shell prompt.

Run the following command to make sure all local filesystems are mounted:

 

mountall -l

 

Change to the directory where you unzipped the 8_Recommended.zip file, and then change to the 8_Recommended/ directory that was created. Run the following command to start the patch process:

 

./install_cluster | tee patchlog

 

Answer 'y' when it asks if you're ready to continue. It will then start working through patch installations.

 

You will see plenty of 'Return code 2' and 'Return code 8' messages. These are normal... return code 2 means that the patch was already installed, and return code 8 means that the patch was for a software package that wasn't installed on the system. For other patch codes, Google search for "Solaris Patch Codes" -- there are several lists out there. With the 'tee patchlog' part of the command, all output from the install_cluster script will be written out to a file named patchlog so that you can review all the messages later.

Be advised that it will usually take a good hour or more for it to run through the entire patch process.

Once script is completed, reboot the system by typing 'reboot'.