Juniper Upgrade Full Guide

Upgrade process is easy, innit? #

I have had bad experience with OS upgrades on Cisco Nexus devices in the past and I hopefully thought, Juniper is doing much better in how it handles its upgrade procedure. Yeah what should I say honestly, thats not the case. Now at this point I guess that every brand has its own individual challenge in upgrading there network devices.

So the question should be more like; why is it such a challenge?

• Is it the fact that the network guy did not take the time to learn about the build and structure of hardware and the software stack to do a successfuly upgrade?

• Does the brand just do weird things here in handling such a upgrade process? Order of upgrades between hardware, there drivers and the VM host? (bare-metal images won’t be supported anymore, so thats the idea of Juniper…)

Funny thing is, that I’m not alone having this issues with this stuff, when I take a short research session, I find dozens of other network guys having problems with that. But what is the difference between a network device like a MX480 model from Juniper instead of any other device which a similar OS or Unix-based system? I mean we easily upgrade Debian on thousands of different hardware devices instead of a router which in some case does not have than 5 different bare-metal models and it works quiet well.

Problem Case: MX480 #

In this post I want to cover only the MX480 router model, because this guy has more hardware inside his chassi than other models from Juniper, which is quiet more challenging to upgrade.

I do not want to go too much in detail, so I just want to cover the necessary stuff for a successful upgrade process.

Structure and Build #

Typically you want to run MX480 in following setup:

• Use two Routing Engines, in case one stuck, have a software bug or a hardware damage (trust me, I faced that several times in the past).

• In each Routing Engine should be two disks installed, which are called primary and backup disk.

• Optional but effective: try to have a backup for each FPC/linecard you installed in chassi to have a backup if one is crashing or having other troubles.

To understand the upgrade process you need to understand how the routing engines and the disk storage on this device is working.

Layers of Routing Engine #

Routing Engine is split in different layers. Important are the layer of OS & HYPERVISOR and JUNOS VM GUEST for upgrade procedure.

Iam not an OS developer, but I find it funny to run a “a construction kit to build Linux-based systems” like YOCTO Linux seems on such a more-critical infrastrcuture device, but maybe this is normal and Iam just stupid(?)

Disk Storage of Routing Engine #

Every Routing Engine has two SSD slots to run a primary and a backup disk. Both disks are build the same way with the same structure of partitions.

Important are the different SET here, which are seperated in p for primary and b for backup. On each of it a Junos Image is installed and hold. So in case of an issue its possible to switch between both sets to boot in a different Junos Image.

On Junos OS this looks like following:

Routing Engine 0 (RE0) #

root@mx480-re0> show vmhost version 
Current root details,     Device sda, Label: jrootp_P, Partition: sda3
Current boot disk: Primary
Current root set: p
UEFI  Version: NGRE_v00.53.00.01

Primary Disk, Upgrade Time: Wed Jun 12 14:41:21 GMT 2024

Version: set p
VMHost Version: 7.2570
VMHost Root: vmhost-x86_64-21.4R3-S10-20241211_2121_builder
VMHost Core: vmhost-core-x86-64-21.4R3-S10.13
kernel: 5.2.60-rt15-LTS19
Junos Disk: junos-install-mx-x86-64-21.4R3-S10.13

Version: set b
VMHost Version: 3.1917
VMHost Root: vmhost-x86_64-18.4R3-S7-20210107_0140_builder
VMHost Core: vmhost-core-x86-64-18.4R3-S9.2
kernel: 3.10.100-ovp-rt110-WR6.0.0.38_preempt-rt
Junos Disk: junos-install-mx-x86-64-18.4R3-S9.2

Routing Engine 1 (RE1) #

root@mx480-re1> show vmhost version 
Current root details,     Device sda, Label: jrootp_P, Partition: sda3
Current boot disk: Primary
Current root set: p
UEFI  Version: NGRE_v00.53.00.01

Primary Disk, Upgrade Time: Wed Jun 12 16:22:45 GMT 2024

Version: set p
VMHost Version: 7.2570
VMHost Root: vmhost-x86_64-21.4R3-S10-20241211_2121_builder
VMHost Core: vmhost-core-x86-64-21.4R3-S10.13
kernel: 5.2.60-rt15-LTS19
Junos Disk: junos-install-mx-x86-64-21.4R3-S10.13

Version: set b
VMHost Version: 3.1917
VMHost Root: vmhost-x86_64-18.4R3-S7-20210107_0140_builder
VMHost Core: vmhost-core-x86-64-18.4R3-S9.2
kernel: 3.10.100-ovp-rt110-WR6.0.0.38_preempt-rt
Junos Disk: junos-install-mx-x86-64-18.4R3-S9.2

We can see above, that we can hold in summary four Junos Images for both Primary Disk on both Routing Engines. This will be multiplied for the Backup Disk, to have in summary then eight Junos Images to work with.

Additional to that we know that currently we are running on the SET p on both Routing Engines.

Upgrade Procedures #

There are several different ways to upgrade a MX480 based on how many Routing Engines and disks are running in the chassi. All following examples are based on MX480 with two Routing Engines and 2 disks each.

Preperation #

Before you are going to install a new Junos Image it is mandatory to know, which Routing Engine is currently the Master and which is the Backup one. There are two ways to find this out:

1. Easiest way is to execute a command which is only executeable from the current Master Routing Engine, one of those commands are: show chassis alarms

2. If you have configured both Routing Engines in a group then you can check with SSH on which Routing Engine you get terminated. But to be 100% sure I would check with the above command too.

I would recommend to have a group configuration if you run with two Routing Engines, there are only advantages and no disadvantages of have this active. Its pretty easy.

set groups re0 system host-name berlin.r1-re0
set groups re0 system ports auxiliary type vt100
set groups re0 interfaces fxp0 unit 0 family inet address 10.0.0.10/24
set groups re0 snmp description berlin.r1-re0.example.de
set groups re1 system host-name berlin.r1-re1
set groups re1 system ports auxiliary type vt100
set groups re1 interfaces fxp0 unit 0 family inet address 10.0.0.20/24
set groups re1 snmp description berlin.r1-re1.example.de

set system apply-groups re0
set system apply-groups re1
set interfaces apply-groups re0
set interfaces apply-groups re1
set snmp apply-groups re0
set snmp apply-groups re1

set system commit synchronize

If you are one the Backup one, you would receive this error:

root@mx480-re1> show chassis alarms 
error: Aborted! This command can only be used on the master routing engine.

Next check how old the current image is before you going to upgrade it.

If you are between these two worlds, you have two choices:

1. Upgrade to a recent-ish Junos version (e.g. 18.4R3) that supports old & new firmware, install the new i40e NVM-Firmware and upgrade to future version that only support new firmware and do a upgrade of the underlaying OS from 11 to 12.

2. There are several Junos Images on which the i40e NVM-Firmware upgrades to 7.0 and the underlaying OS upgrade from 11 to 12 are already pre-installed.

You can check the current installed i40e NVM-Firmware with: show system firmware:

Part             Type              Tag Current   Available Status
                                       version   version
Routing Engine 0 RE BIOS           0   0.53.1    0.55.2    OK                
Routing Engine 0 RE FPGA           1   41.0.0    41.0      OK                
Routing Engine 0 RE SSD1           6   12050     12050     OK                
Routing Engine 0 RE SSD2           6   12050     12050     OK                
Routing Engine 0 RE i40e-NVM       7   4.26      0         OK                
Routing Engine 1                   0   0.53.1    0         OK

So its version 4.26, that means you need to upgrade it.

I definetly recommend the second option, I ran several times with the first option and fucked it completly up. Trust me, its not funny, its disgusting.

Scenario #1 - Under 18.4R1 + manually upgrading firmware + underlaying OS #

Okay buddy, thats the hard way, wish you luck, could be challenging. Here you go.

… but do you have all the mandatory files for that!?

• You need an Junos Image like junos-vmhost-install-mx-x86-64-18.4R3-S9.2.tgz
• You need the i40e NVM-Firmware Image which is exactly the same junos version, same release version and same maintenance version without that it will not work. So with the image above you need: jfirmware-vmhost-x86-64-18.4R3-S9.2.tgz
• You need the OS upgrade package like os-package-20230925.211220_builder_stable_12.tgz

If this is the case you can continue, if not stop here!

1. Check which Routing Engine is the current Master, take a look here. In this example current Master Routing Engine is re0.

2. Copy the Junos Image junos-vmhost-install-mx-x86-64-18.4R3-S9.2.tgz to the router, it doesn’t matter if you copy it to the Master or Backup one, you need it on both.

3. Copy it internally to the Backup one: file copy /var/tmp/junos-vmhost-install-mx-x86-64-18.4R3-S9.2.tgz re1:/var/tmp.

4. Connect to the Backup one: request routing-engine login other-routing-engine.

5. (Optional) You should have a console connection to see the boot process and in case you have no connection via IP to the Routing Engine.

6. Check the current SET status. Image is getting installed always on the currently not used one! So if you are running on SET p it is going to install it on SET b.

7. (Optional) If you do not want to install it on the not currently used SET then you need to switch it with: request vmhost software rollback re1 and reboot the Routing Engine.

8. If it is rebooted in the correct SET then you start the upgrade to 18.4R3 with: request vmhost software add re1 /var/tmp/junos-vmhost-install-mx-x86-64-18.4R3-S9.2.tgz no-validate.

9. Reboot with request vmhost reboot re1.

10. Connect to the Backup one: request routing-engine login other-routing-engine.

11. After the upgrade succeeded continue with the i40e NVM-Firmware upgrade: request vmhost software add /var/tmp/jfirmware-vmhost-x86-64-18.4R1.8.tgz.

12. After loading it, trigger the upgrade process with the next reboot: request system firmware upgrade re i40nvm. (This command is hidden, so no auto completion!)

13. You should see now PROGRAMMING (0%) in the output of: show system firmware.

14. Reboot the Routing Engine: request vmhost reboot re1 to install the new firmware while booting.

15. Routing Engine will reboot and upgrade firmware, it’ll say again to power cycle several times, each time it asks: request chassis cb slot 1 offline and request chassis cb slot 1 online.

16. Connect to the Backup one: request routing-engine login other-routing-engine.

17. Check the firmware again, it should now show 6.01 or higher: show system firmware.

18. Check the current version of FreeBSD on the underlaying OS: start shell and then uname -a, if the output includes something like: FreeBSD JNPR-11.0-20220930.99a5417_buil FreeBSD JNPR-11.0-20220930.99a5417_builder_stable_11-204ab then you need to upgrade the OS.

19. (Optional) If the OS needs to be upgraded then execute: request system software add /var/tmp/os-package-20230925.211220_builder_stable_12.tgz, output should be something like this: Verified os-package-20230925.211220_builder_stable_12 signed by PackageProductionECP256_2023 method ECDSA256+SHA256. So you triggered the upgrade process which takes in place with the next reboot.

20. If your i40e NVM-Firmware upgraded to 7.0 you can start copying the firmware higher or equal 21.4R1 on the Routing Engine. If you are on 6.01 you should go with 19.x or 20x. We assume its 7.0 now.

21. Copy the Junos Image junos-vmhost-install-mx-x86-64-18.4R3-S9.2.tgz to the router, it doesn’t matter if you copy it to the Master or Backup one, you need it on both.

22. Copy it internally to the Backup one: file copy /var/tmp/junos-vmhost-install-mx-x86-64-21.4R3-S5.4.tgz re1:/var/tmp.

23. Check SET again and repeat steps 6 - 8 above.

24. Install new image: request vmhost software add re1 /var/tmp/junos-vmhost-install-mx-x86-64-21.4R3-S5.4.tgz no-validate. You need to set no-validate because there was an upgrade in the Kernel-Space below JunOS (BSD-Upgrade) and since an older version cannot correctly verify a new version this message is expected and can safely be used.

25. Reboot with request vmhost reboot re1.

26. Connect to the Backup one: request routing-engine login other-routing-engine.

27. After rebooting and upgrade you can check underlaying OS with: start shell and uname -a output should be something like: FreeBSD JNPR-12.1-20230821.5fbe894_buil FreeBSD JNPR-12.1-20230821.5fbe894_builder_stable_12_214

28. Check Junos Version: show version output should be: Junos: 21.4R3-S5.4,os-package-20230925.

29. At this point we finished the upgrade for the Backup Routing Engine. To do the same for the current Master one, we need to switchover request chassis routing-engine master switch.

    Warning! All following shown examples are done in a defined maintenance range. Upgrading this with two Routing Engines in a full productive setup with active customer traffic is not recommended, a switchover could take some seconds and this is enough to impact active traffic!

30. After switchover you can repeat steps 1 - 28 above.

Done.

Scenario #2 - Under 18.4R1 + you don’t want to upgrade manually #

Well, thats the best choice you decided to use the safer way.

There are several Junos Images on which the i40e NVM-Firmware upgrades to 7.0 and the underlaying OS upgrade from 11 to 12 are already pre-installed:

• 21.4R3-S5 and higher

• 22.1R3-S3 and higher

• 22.2R3-S1 and higher

• 22.3R3 and higher

• 22.4R3 and higher

• 23.1R2 and higher

• 23.2R2 and higher

• 23.3R1 and higher

• 23.4R1 and higher

I would recommend not skipping all major releases to install the newest Junos Image. In this scenario we install first 18.4R43 and at the end I would upgrade directly from 18.4R1 to 21.x for safer reasons.

1. Check which Routing Engine is the current Master, take a look here. In this example current Master Routing Engine is re0.

2. Copy the Junos Image junos-vmhost-install-mx-x86-64-18.4R3-S9.2.tgz to the router, it doesn’t matter if you copy it to the Master or Backup one, you need it on both.

3. Copy it internally to the Backup one: file copy /var/tmp/junos-vmhost-install-mx-x86-64-18.4R3-S9.2.tgz re1:/var/tmp.

4. Connect to the Backup one: request routing-engine login other-routing-engine.

5. (Optional) You should have a console connection to see the boot process and in case you have no connection via IP to the Routing Engine.

6. Check the current SET status. Image is getting installed always on the currently not used one! So if you are running on SET p it is going to install it on SET b.

7. (Optional) If you do not want to install it on the not currently used SET then you need to switch it with: request vmhost software rollback re1 and reboot the Routing Engine.

8. If it is rebooted in the correct SET then you start the upgrade to 18.4R3 with: request vmhost software add re1 /var/tmp/junos-vmhost-install-mx-x86-64-18.4R3-S9.2.tgz no-validate.

9. Reboot with request vmhost reboot re1.

10. Connect to the Backup one: request routing-engine login other-routing-engine.

11. Check Junos Version: show version output should be: Junos: 18.4R3-S9.2.

12. Copy the Junos Image junos-vmhost-install-mx-x86-64-18.4R3-S9.2.tgz to the router, it doesn’t matter if you copy it to the Master or Backup one, you need it on both.

13. Copy it internally to the Backup one: file copy /var/tmp/junos-vmhost-install-mx-x86-64-21.4R3-S5.4.tgz re1:/var/tmp.

14. Check SET again and repeat steps 6 - 8 above.

15. Install new image: request vmhost software add re1 /var/tmp/junos-vmhost-install-mx-x86-64-21.4R3-S5.4.tgz no-validate.

16. Reboot with request vmhost reboot re1.

17. Connect to the Backup one: request routing-engine login other-routing-engine.

18. Check Junos Version: show version output should be: Junos: 21.4R3-S5.4,os-package-20230925.

19. Check the firmware again, it should now show 6.01 or higher: show system firmware.

20. After rebooting and upgrade you can check underlaying OS with: start shell and uname -a output should be something like: FreeBSD JNPR-12.1-20230821.5fbe894_buil FreeBSD JNPR-12.1-20230821.5fbe894_builder_stable_12_214

21. At this point we finished the upgrade for the Backup Routing Engine. To do the same for the current Master one, we need to switchover request chassis routing-engine master switch.

    Warning! All following shown examples are done in a defined maintenance range. Upgrading this with two Routing Engines in a full productive setup with active customer traffic is not recommended, a switchover could take some seconds and this is enough to impact active traffic!

22. After switchover you can repeat steps 1 - 20 above.

Done.

To conclude #

Hopefully this will help someone with all the upgrade pain. I faced many issues while I was testing and trying to fix stuff…

I would say above two scenarios should do the job for more than 75% all upgrade cases. I will update this file, if I will find another scenario which is particularly different to the mentioned one.

Finaly done.

Cheers mate,