Bulk version change tips and tricks

Ever needed to create a lot of versions as a copy of others?  Ever needed to create versions and also change data selections and processing options?  Say for example you opened another DC and wanted to copy all of the config from the previous ones – reuse all of your IP – well, do I have some good news for you..  Well indifferent news – it can be done.

The first step to getting this working is brain storming, one of my fav people to brainstorm with is Shae.  We can have quick single syllable word conversations, grunt a bit – but at the end of the day articulate an elegant solution to some amazing technical problems,  I’m very lucky to have peers like this to work with.  Shae came up with the idea of using a par file to complete this task – and that was a great idea!  I can easily create a project with SQL, populate it with all of the versions I need to copy.  I can also create all of the F983051 and central objects to create the base objects, but I’d need to use load testing or scripts to change all of the PO’s and data selection.

Shae’s idea to use the parfile was great, it seemed possible.  The client in question has about 500 versions all for a particular DC, and I needed to change names, PO;s and data selections based upon the new name change – okay – challenge accepted.

There are heaps of ways of doing this – java, node.js, lambda, vbscript – I went old school – a little bit of sed and awk.

I basically took the parfile, sftpd it to linux and then ripped it apart.

The structure was not too crazy to deal with, although it did feel like Russian dolls, where there was a zip file in a zip file in a zip file.

There was also some pretty funky things like unicode files in the middle not normal files and base64 strings for PO’s – but nothing was going to stop me.

What I’m going to do is just cut and paste the script here, you’ll get the idea of what needed to be done from the sections and the amazing comments.

In my example the version names, PO’s and data selection all changed from string TT to string GB – so it was really easy to apply these rules through the script.

At the end of the day, this created a separate par file that you can restore to a project when all of the new versions in it! Really nice.

There is a tiny bit of error handling and other things – but really just showing you what can be done.

Imagine if you needed to change the queue on 100’s of versions or anything like this.  You could use some of the logic below to get it done (or be nice to me).

if [ $# -ne 3 ]
   then
     echo 'USAGE $0 <parfile> <FROm STRING> <TO STRING>'
     exit
fi

_debug=1
workingDir=/tmp
parfile=$1
fromString=$2
toString=$3
parfileNoExt=`echo $parfile | awk -F. '{print $1}'`
expDir=$workingDir/$parfileNoExt

rm -fR $expDir

#unzip the file to the working dir
unzip $parfile -d $expDir

for file in `ls $expDir/*.par`
   do
     #echo $file
     dir=`echo $file | awk -F. '{print $1}'`
     #echo $dir
     unzip -q $file -d $dir
done

#parfile
# parfile UBEVER_R57000035_PP0001_60_99
#   F983051.xml
#   F983052.xml
#   manifest.xml
#   specs.zip
#   RDASPEC
#    R5700036.PP0002.1.0.0.0.0.xml
#

#now lets extract the specs zip file for each

find $expDir -name specs.zip -execdir unzip -q \{} \;

#now delete par files and all else

find $expDir -name '*.par' -exec rm \{} \;
find $expDir -name specs.zip -exec rm \{} \;

# now we need to rename directories
if [ $_debug = 1 ]
then
   echo "RENAME DIRS"
fi

cd $expDir
for dir in `ls -d *${fromString}*`
do
   echo $dir
   newname=`echo $dir | sed s/_${fromString}/_${toString}/g`
   newname=`basename "$newname"`
   echo $newname
   cd $expDir
   mv $dir $newname
done

#reanme files, generally in the spec dir
#for file in `find $expDir -name "*${fromString}*.xml" -type f -print`
#holy crap, that took a long time to encase this with double quotes so as not to lose the
#dodgey versions
if [ $_debug = 1 ]
then
   echo "RENAME FILES"
fi

find $expDir -name "*${fromString}*.xml" -type f |while read file; do
     newfile=`basename "$file"`
     newfile=`echo "$newfile" | sed s/${fromString}/${toString}/2`
     currDir=`dirname "$file"`
     mv "$file" "$currDir/$newfile"
     if [ $? -ne 0 ]
       then
         echo "MOVE ERROR " "FILEFROM:$file:" "FILETO:$currDir/$newfile:"
         sleep 10
         exit
     fi
done

#filelist="`find $expDir -name "*${fromString}*.xml" -type f -print`"
#echo $filelist
#for file in $filelist
   #do
     #newfile=`basename "$file"`
     #newfile=`echo "$newfile" | sed s/${fromString}/${toString}/g`
     #currDir=`dirname "$file"`
     #mv "$file" "$currDir/$newfile"
     #if [ $? -ne 0 ]
       #then
         #echo "MOVE ERROR " "FILEFROM:$file:" "FILETO:$currDir/$newfile:"
         #sleep 10
         #exit
     #fi
#done

if [ $_debug = 1 ]
then
   echo "SED CONTENTS OF FILES AND CREATE .NEW"
fi

#This is ridiculous - I need to convert manifest.xml
#from utf-16 to utf-8 and grep and then back again
# this is killing me
echo 'CONVERTING MANIFEST.XML'
for file in `find $expDir -name manifest.xml -print`
do
   echo $file
#  iconv -f utf-16 -t utf-8 $file | sed s/${fromString}[0-9]/${toString}/g > $file.utf8
   iconv -f utf-16 -t utf-8 $file | sed s/${fromString}0/${toString}0/g > $file.utf8
   iconv -f utf-8 -t utf-16 $file.utf8 > $file
   rm $file.utf8
done
  
#okay, now for the contents of the files
set -x
grep -r -l "${fromString}" $expDir | while read file; do
#for file in "`grep -R -l ${fromString} $expDir/*`"
#  do
     newfile=`echo "${file}.new"`
     echo $file "contains $fromString"
     cat "${file}" | sed s/${fromString}/${toString}/g > "${newfile}"
     #note that if you need to compare the internals of the files
     #comment out the following lines.
     rm "$file"
     mv "$newfile" "$file"
     if [ $? -ne 0 ]
       then
         echo "MOVE ERROR " "FROMFILE:$newfile:" "TOFILE:$file:"
         sleep 10
         exit
     fi
done

#Need to decode the base63 PO string and replace fromString there too
#find the F983051's
#create a variable for PODATA, check for PP
echo "Processing F983051 VRPODATA"
for file in `find $expDir -name F983051.xml -print`
do
   base64String=`cat $file | tr -s "\n" "@" | xmlstarlet sel -t -v "table/row/col[@name='VRPODATA']"  | base64 -d`
   charCount=`echo $base64String | wc -c`
   if [ $charCount -gt 1 ]
     then
     base64String=`echo $base64String | sed s/${fromString}/${toString}/g`
     echo 'changed string:' $base64String
     base64String=`echo $base64String | base64`
     xmlstarlet ed --inplace -u "table/row/col[@name='VRPODATA']" -v $base64String $file
     #just need to run the xmlstarlet ed
   fi
done

# find $expDir -name '*.new' -print | wc -l

#so now we replace the .new with the original
#job done...
#need to zip everything back up

if [ $_debug = 1 ]
then
   echo "Creating spec.zip"
fi

for dir in `ls -d $expDir/*`
   do
     cd $dir
     zip -r specs.zip ./RDASPEC
     zip -r specs.zip ./RDATEXT
     rm -fr ./RDASPEC ./RDATEXT
done

#now we create all of the par files from the dirs under expDir

for dir in `ls -d $expDir/* |grep drwx`
   do
     cd $expDir
     zip -r ${dir}.par `basename $dir`
     rm -rf $dir
done

#now the root parfile
cd $expDir
rm -rf ../${parfile}.zip
zip -r ../${parfile}.zip *