Scripts to Archive Log or Data Files

Scripts to Archive Log or Data Files

 

Script will move logs to a subdirectory based upon time, zip files after specified time period, then finally delete after a specified time period

 

Create file called archiveLog.sh in home directory of user that will be doing the archive process or in a directory owned by that user.

 

#!/bin/bash

#***************************************************************************************************#
# archiveLog.sh                                                                                #
# March, 2009                                                                                       #
#                                                                                                   #
# Archive log files                                                                           #
#                                                                                                   #
# Usage:                                                                                            #
# archiveLog.sh [-b base_dir] [dir 1] [dir2] [dir n] [dir_day_cnt [zip_day_cnt [del_day_cnt]]] #
#                                                                                                   #
# Default behavior:                                                                                 #
# Archive from base directory /log/                                                            #
# Archive all subdirectories                                                                        #
# Move to a subdirectory after 31 days                                                              #
# Move to a zip with the same name as the subdirectory 62 days after the                            #
# subdirectory was created.                                                                         #
# Remove zip files a year after the zip file was created                                            #
#                                                                                                   #
# Notes:                                                                                            #
# To flag a directory so that the archive process ignores it, put a file                            #
# in the directory with the name ".archive.ignore"                                              #
#***************************************************************************************************#

echo " log file archiving"

ERROR_GENERAL=1
ERROR_BAD_DIR=2

prog_name=" Archive Log"
base_dir="/log"
arc_script="/tmp/archlogtmp.sh"

# process command line args - last 3 (or first numeric) are day counts
# -b base_dir must be first 2 args if used
IFS=$'\x0A'
delim=$'\x0A'
dir_list=""
if [ ! -z "$1" ] ; then
   if [ "$1" = "-b" ] ; then
      base_dir=${2:-$base_dir}
      shift
      shift
   fi
   until [ -z "$1" ] || [[ $1 == [0-9]* ]]
   do
      dir_list="$dir_list$delim$base_dir/$1"
      shift
   done
fi
# Get the list of all directories off the base directory if none were specified
if [ -z "$dir_list" ] ; then
   dir_list=$(find $base_dir -type d -maxdepth 1  -mindepth 1 -printf "%p\n")
fi

dir_day_cnt=${1:-30}
zip_day_cnt=${2:-32}
del_day_cnt=${3:-365}

echo "Days to move to directory  : $dir_day_cnt"
echo "Days to zip to archive     : $zip_day_cnt"
echo "Days to remove archive     : $del_day_cnt"
echo "Base directory: $base_dir"
echo "Directories being archived : "
echo $dir_list

if [ ! -z "$4" ] ; then
   for i in [1 2 3] ; do shift; done
   echo "$prog_name : Warning : Unused command arguments: $@"
fi

# Check for error code and if it is not 0 display an error message and exit
# $1 result error code to test
# $2 exit error code if there is an error
# $3 custom error message - default is general error
check_error()
{
   if [ "$1" -ne "0" ]; then
      echo "$prog_name : Error $2 : ${3:-General Error}" >& 2
      exit $2
   fi
}

# Create a temporary script with all archiving steps plus helper functions
cat << EOF > $arc_script
#!/bin/bash

# Check for error code and if it is not 0 display an error message and exit
# \$1 result error code to test
# \$2 exit error code if there is an error
# \$3 custom error message - default is general error
check_error()
{
   if [[ \$1 == [0-9]* ]] && [ "\$1" -ne "0" ]; then
      echo "\$0 : Error \$2 : \${3:-General Error}" >& 2
      exit \$2
   fi
}

# Move the contents of a directory into a zip file and remove the directory
# Non-managed directories are ignored (don't have the .arch file)
# \$1 Directory to be zipped
zipDirectory()
{
   local tag="\$1/.arch"
   local ignore="\$1/.archive.ignore"
   if [[ -f "\$tag" ]] && [[ -f "\$ignore" ]] ; then
      echo "Compressing archive directory \$1"
      zip -m9jDq \$1-Archive \$1/*
      check_error \$? $ERROR_GENERAL "Unable to create archive '\$1-Archive.zip'"
      rm "\$tag"
      check_error \$? $ERROR_GENERAL "Unable to remove tag file '\$tag'"
      rmdir "\$1"
      check_error \$? $ERROR_GENERAL "Unable to remove directory '\$1'"
   fi
}

# Move a file from its original directory to an archive directory
# \$1 direcotory containg file
# \$2 File to move
# \$3 archive directory to move file into
moveFile()
{
   local tag="\$1/.arch"
   local arcdir="\$1/\$3"
   local ignore="\$1/.archive.ignore"
   if [[ ! -f "\$tag" ]] && [[ ! -f "\$ignore" ]] ; then
      if [[ ! -d \$arcdir ]] ; then
         /bin/mkdir "\$arcdir"
         check_error \$? $ERROR_GENERAL "Unable to create directory '\$arcdir'"
         echo " Archive" > "\$arcdir/.arch"
         echo "Creating new archive directory \$arcdir"
      fi
      /bin/mv "\$1/\$2" "\$arcdir"
      check_error \$? $ERROR_GENERAL "Unable to move file '\$2' to archive directory '\$arcdir'"
   fi
}

# Remove a file and display removal notice
# \$1 fully qualified file name to remove
removeArchive()
{
   echo "Removing archive \$1"
   rm "\$1"
   check_error \$? $ERROR_GENERAL "Unable to remove archive '\$1'"
}

EOF

# Make sure base directory exists
if [[ ! -d "$base_dir" ]] ; then
   check_error 1 $ERROR_BAD_DIR "Invalid base directory '$base_dir'"
fi

# Make sure all archive directories are valid
for dir in $dir_list
do
   if [[ ! -d "$dir" ]] ; then
      check_error 1 $ERROR_BAD_DIR "Invalid directory '$dir'"
   fi
done

# Look for date files older than one month and move them into an archiving sub-directory
for dir in $dir_list
do
   find "$dir" -type f -mtime +$dir_day_cnt  \! \( -name \*.zip -or -name \*.gzip \) -printf "moveFile \"%h\" \"%f\" \"%Tb-%TY\" \n " | sed -e 's/\$/\\\$/g' >>$arc_script
   check_error $? $ERROR_GENERAL "Error calling 'find' to move files for '$dir'"
done

# Look for archive directories that are more than two months and zip them
for dir in $dir_list
do
   find "$dir" -type d -mtime +$zip_day_cnt -name ???-???? -printf "zipDirectory \"%p\" \n " >>$arc_script
   check_error $? $ERROR_GENERAL "Error calling 'find' to zip directories for '$dir'"
done

# Look for zip archives that are one year old and delete them
for dir in $dir_list
do
   find "$dir" -type f -mtime +$del_day_cnt -name ???-????-Archive.zip -printf "removeArchive \"%p\" \n ">>$arc_script
   check_error $? $ERROR_GENERAL "Error calling 'find' to remove arhive zip files for '$dir'"
done

# Execute the archiving script
/bin/bash "$arc_script"
check_error $? $ERROR_GENERAL "Error running generated archive script"

echo "$prog_name : Archival processing complete"
exit 0
 

Create a file called archiveLogJob.sh in the same directory

 

/bin/bash /home/webadmin/archiveLog.sh dir1 dir2 30 32 365

 

dir1 and dir2 corresponds to the names of the directories you want to archive and it is assumed that the top level is /log in this script

the 30 32 and 365 correspond to days to move, days to zip, and days to delete respectively

 

Usage:                                                                                           
# archiveLog.sh [-b base_dir] [dir 1] [dir2] [dir n] [dir_day_cnt [zip_day_cnt [del_day_cnt]]]

 

Finally create a cron job to run the scripts on an appropriate schedule

 

0 15 * * 5  /home/user/scripts/archiveLogJob.sh > /home/user/archiveLogJob.log