I recently needed to upgrade a few MX480 routers and decided that it would be a good opportunity to get some experience with Juniper’s in service software upgrade. I’d read a bit about it but I’d not had the chance to really use it. It’s pretty straightforward and it does what it claims. The following are my notes from rolling through this on my test lab MX480.
A few things are necessary to get going with ISSU, first and foremost, you need to have a box with two routing engines. Check. Second, some configuration is necessary to make this all work. The boxes need to be running nonstop-routing, they need to be syncronizing the configs, and they need to have graceful-switchover enabled.
Here are the steps I went through on my vanilla test box to make this work:
root# set chassis redundancy graceful-switchover root# set routing-options nonstop-routing root# set system commit synchronize
If you already have those options set then you don’t need to enter those commands.
With those options set, you’re ready to do the ISSU.
root> request system software in-service-upgrade /var/tmp/jinstall-10.3R2.11-domestic-signed.tgz Chassis ISSU Check Done ISSU: Validating Image Checking compatibility with configuration Initializing… Using jbase-10.1R1.8 vn_read_compressed_block: invalid block index 550 Verified manifest signed by PackageProduction_10_1_0 Verified jbase-10.1R1.8 signed by PackageProduction_10_1_0 Using /var/tmp/jinstall-10.3R2.11-domestic-signed.tgz ….
This takes a LONG time and generates a lot of scroll.
You’ll see long pauses and more text like
Saving package file in /var/sw/pkg/jinstall-10.3R2.11-domestic-signed.tgz … Saving state for rollback … Backup upgrade done Rebooting Backup RE
Rebooting re1 ISSU: Backup RE Prepare Done Waiting for Backup RE reboot
Then interesting thing start to show up:
GRES operational Initiating Chassis In-Service-Upgrade Chassis ISSU Started ISSU: Preparing Daemons ISSU: Daemons Ready for ISSU ISSU: Starting Upgrade for FRUs ISSU: Preparing for Switchover
This is where the magic starts. The nonstop-routing and hitless failover come into play as the route engines switch over. Very cool.
On the console of the backup RE (now the master) you’ll see messages like
Message from syslogd@ at Dec 29 19:11:57 … fpc0 RDP: Remote side reset connection: rdp.(fpc0:22528).(serverRouter:ppm)
Message from syslogd@ at Dec 29 19:11:57 … fpc1 RDP: Remote side reset connection: rdp.(fpc1:4096).(serverRouter:ppm)
these are normal.
Some things that I didn’t expect, but probably should have:
The old master stays the backup route engine after the ISSU The old master does NOT reboot into the new code, instead it stays on the original code requiring a manual reboot (unless, I asume, you add the “reboot” command on to the original upgrade command). On routers that have logical systems configured on them, only the master logical system supports nonstop active routing.
I did the reboot manually
root> request system reboot Reboot the system ? [yes,no] (no) yes
*** FINAL System shutdown message from root@ *** System going down IMMEDIATELY
Shutdown NOW! Reboot consistency check bypassed - jinstall 10.3R2.11 will complete installation upon reboot [pid 65006]
and then did a failover to the old master.
root> request chassis routing-engine master switch
….and thats pretty much it. Upgrade complete.
This is a really useful tool that allows for very minimal interruption during software upgrades. I’d recommend reading this white paper on ISSU if you’re interested into diving into deeper details.
Basically what ISSU does is to install junos on the backup route engine (re1) as normal, reboot it, validate and switch over to re1 and do the upgrade on the primary (now backup) route engine. The entire process took about 40 minutes on my mx480.