PDA

View Full Version : Test/Prod Patching Strategy



cmyslinsky
12-Feb-2014, 16:35
I have been tasked with implementing a patching strategy on our new SLES 11 SP3 boxes which starts patches in test then a month later installs those same patches in production. Rinse and repeat.

Being new to SuSE administration I've started investigating this effort with the Online Update tool. First glance it appears I can select the needed updates, export the list of updates to a text file then install. I'm guessing I can use the text file to ensure only the prior month's patches are installed on production. In the end this solution feels kludgy and error prone so I'm looking for suggestions.

I'm curious how others are pushing patches through their environments and better implementation ideas for this particular scheme of patching test, waiting a month then patching production.

Cheers
Chad

jmozdzen
12-Feb-2014, 16:58
Hi Chad,

sounds like you're running a modestly sized environment (read "no sneaker management") - have you had a look at the "SUSE Manager" product?

Regards,
Jens

cmyslinsky
12-Feb-2014, 17:11
Thanks for the fast response Jens. I should have mentioned the size of the environment in the original message. Sorry about that.

We currently have 6 boxes with 2 being prod and the remaining 4 being dev/test so not too bad even if we had to do something manual. There are also 7 Red Hat boxes which could become SuSE at some point but I don't expect the environment to grow beyond those.

Haven't investigated SuSE Manager will look that up. Would you still recommend investigating that for a 15ish box environment?

Chad

jmozdzen
12-Feb-2014, 17:17
Hi Chad,

Haven't investigated SuSE Manager will look that up. Would you still recommend investigating that for a 15ish box environment?

let me admit that I have no idea of the licensing costs - but that environment size would in my eyes justify at least evaluating that tool. SUSE Manager will handle RedHat boxes as well, so you'd profit right from the start...

Regards,
Jens

malcolmlewis
12-Feb-2014, 17:21
On Wed 12 Feb 2014 04:14:02 PM CST, cmyslinsky wrote:


Thanks for the fast response Jens. I should have mentioned the size of
the environment in the original message. Sorry about that.

We currently have 6 boxes with 2 being prod and the remaining 4 being
dev/test so not too bad even if we had to do something manual. There
are also 7 Red Hat boxes which could become SuSE at some point but I
don't expect the environment to grow beyond those.

Haven't investigated SuSE Manager will look that up. Would you still
recommend investigating that for a 15ish box environment?

Chad




Hi
If you can spin up a virtual machine with 4-8GB of RAM and say 80GB of
disk space, it would be worth running an eval version of SUSE Manager
to test, it will also support the RHEL machines.

--
Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
openSUSE 13.1 (Bottle) (x86_64) GNOME 3.10.2 Kernel 3.11.10-7-desktop
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below... Thanks!

cmyslinsky
13-Feb-2014, 20:37
It looks like for the short-term the patching strategy will need to happen without the assistance of SuSE Manager. Since the original post I realized that what is selected under summary where it says "Security update for..." or "Recommended update for..." can actually be comprised of one or more packages. It's the packages that I can export to the text file from the YaST2 Online Update. In other words, what is being selected to be installed doesn't cleanly match up with the packages to be installed. I'm also concerned a month later when production is patched that a newer version of a package could become available and supersede the one installed in test the month prior. It appears an alternative version could be selected from the Versions tab but I'm not sure what caveats that could bring. So, I'm back to looking for additional suggestions on how to patch test, wait a month than apply those exact same patches in production. Additional thoughts?

Chad

malcolmlewis
13-Feb-2014, 21:17
On Thu 13 Feb 2014 07:44:01 PM CST, cmyslinsky wrote:


It looks like for the short-term the patching strategy will need to
happen without the assistance of SuSE Manager. Since the original post
I realized that what is selected under summary where it says "Security
update for..." or "Recommended update for..." can actually be comprised
of one or more packages. It's the packages that I can export to the
text file from the YaST2 Online Update. In other words, what is being
selected to be installed doesn't cleanly match up with the packages to
be installed. I'm also concerned a month later when production is
patched that a newer version of a package could become available and
supersede the one installed in test the month prior. It appears an
alternative version could be selected from the Versions tab but I'm not
sure what caveats that could bring. So, I'm back to looking for
additional suggestions on how to patch test, wait a month than apply
those exact same patches in production. Additional thoughts?

Chad




Hi
Subscribing to the patch alert emails should provide info on what/why
the updates are there. Some are critical, and depending on your
situation may need to applied sooner than a month. Sounds like you need
some vm's and snapshots to test/check?

You can use zypper with a dry-run command to see what will be
downloaded and installed, you can also just download only and not
install....

--
Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
openSUSE 13.1 (Bottle) (x86_64) GNOME 3.10.2 Kernel 3.11.10-7-desktop
If you find this post helpful and are logged into the web interface,
please show your appreciation and click on the star below... Thanks!

Magic31
13-Feb-2014, 22:16
...I'm also concerned a month later when production is patched that a newer version of a package could become available and supersede the one installed in test the month prior. It appears an alternative version could be selected from the Versions tab but I'm not sure what caveats that could bring. So, I'm back to looking for additional suggestions on how to patch test, wait a month than apply those exact same patches in production. Additional thoughts?

What I've been doing is setup a SUSE SMT server and disable the daily smt-mirror (that command syncs the local SMT server repos with those in the outside world) and run the smt-mirror command when the patch cycle has completed. With our customer sites that can also take a month or so and this gives them the option to do a sync, start patching the less critical servers (SLES and OES) and move on to the more business critical ones using the same patch levels.

When they are ready for a new patch cycle, they first initiate a smt-mirror command.

That could be an option?

Cheers,
Willem

cmyslinsky
13-Feb-2014, 22:40
We have a VM for test and another for production. The goal is to apply a set of patches to test then a month later apply only that set of patches to production. Preferably using a semi automated method but I'm not ruling a totally manual process if it is foolproof and repeatable. The concerns I'm struggling with are two fold.

First is that the patches selected or deselected for install include one or more packages if I'm using the terminology correctly. The list of packages is what I'm able to export to a text file not the list of patches being installed.

The second area is how to address patches that have been updated to include a newer package than what was installed on test when it is time to patch production a month later. It appears different versions can be selected for install from the versions tab in YaST2 but I'm concerned about making a mess of dependencies.

Chad

cmyslinsky
14-Feb-2014, 14:27
What I've been doing is setup a SUSE SMT server and disable the daily smt-mirror (that command syncs the local SMT server repos with those in the outside world) and run the smt-mirror command when the patch cycle has completed. With our customer sites that can also take a month or so and this gives them the option to do a sync, start patching the less critical servers (SLES and OES) and move on to the more business critical ones using the same patch levels.


SMT appears to be free. Worth a look.

Thanks for the scoop.
Chad

clarkmike
26-Feb-2014, 19:25
I use a shell program to do updates via cron. It would be easy enough to run it at the beginning of the month on the test server then at the end of the month on the production server. This way I get a report of what was installed and can investigate if there are problems. Like when the SLES 11.3 update killed the BackupExec ralus agent.

#!/bin/bash
MailTo=SomeOne #put names and addresses in /etc/aliases
UpdatesOut='/tmp/updates.out'
rm $UpdatesOut

Running=`ps -ef|grep zypper|grep -v grep`
if [ -n "$Running" ];then
echo "$Running" |mail -s "ServerName Updates TROUBLE zypper is still running" $MailTo
exit
fi
zypper ref
UpDates=`zypper lu`
if [[ "$UpDates" =~ "No updates" ]];then
echo "$UpDates" > $UpdatesOut
cat $UpdatesOut | mail -s "ServerName Updates No Updates" $MailTo #</dev/null
exit
fi
/usr/bin/zypper -n up -l > $UpdatesOut
if [ $? -eq 0 ] ;then
mail -s "ServerName Updates" $MailTo < $UpdatesOut
/sbin/SuSEconfig
else
mail -s "ServerName Updates TROUBLE" $MailTo < $UpdatesOut
exit
fi

if grep -i kernel $UpdatesOut > /dev/null 2>&1; then
/sbin/init 6
#reboot if kernel update was included
fi
PRun=`zypper ps`
if [[ "$PRun" =~ "No processes" ]];then
exit
else
echo $PRun|mail -s "ServerName Updates zypper ps" $MailTo
fi
exit

clarkmike
26-Feb-2014, 19:39
I forgot to mention that with each SLES version update it's most likely you will need to change the update program name. There have been 3 different programs used for SLES updates in the last three versions, 9- I forget, 10 -rug, 11- zypper. 12 - ????

Magic31
27-Feb-2014, 21:46
I forgot to mention that with each SLES version update it's most likely you will need to change the update program name. There have been 3 different programs used for SLES updates in the last three versions, 9- I forget, 10 -rug, 11- zypper. 12 - ????

SLES 9 was red-carpet I believe.... but rug was also already in there. Long time for me too since I've used SLES 9.

From the looks of it, zypper will still be the mechanism used for SLES 12. Which is a good thing imo, as I like zypper :)

Cheers,
Willem