Skip to main content
Version: 1.2.2.0

ODP Stack Upgrade

This Page covers the differents way to upgraded, prerequesites, as the complete process.

Latest Ambari Version

You need to have upgraded to latest 2.7.9.0.0-110 version before starting the ODP upgrade procedure.

Prerequesites​

Operating System​

You need to make sure that your Operating System is compatible with the version of ODP stack that you are going to install.

Disk Space Requirement​

You need at least 30 Go of disk available as you need space for both source and target version (a version need about 15Go of space).

Ambari Agent Heartbeats​

All Ambari Agent must be communicating with Ambari Server correctly. If a host is not heartbeating, you need to put in into Maintenance Mode.

Host Maintenance Mode​

Hosts that are in Maintenance Mode should not be hosting any Service Master Components. It’s possible for a host in Maintenance Mode to not host Master Components, but be aware that you’ll receive a warning. You can continue with your upgrade, but these hosts won’t be upgraded. Before finalizing the upgrade, you’ll need to delete these hosts from the cluster.

Service Maintenance Mode​

Service Maintenance Mode is not allowed unless on Ambari Metrics service

Cluster up and running​

All Services must be started, except for Ambari Metrics service

Cluster health​

Every service Check must pass. For this you need to go on each service > Actions section and run the service check.
You need to do it prior to install ODP target packages.
on every services.

Service Auto Start​

Ambari Service Auto Start feature need to be disabled. Log in Ambari Web UI Click Dashboard > Service Auto Start next to Auto Start Settings disable by clicking on the button. Service Auto Start disabling

Backup before upgrading ODP​

Databases Backup​

We recommend that you perform backups of all your databases before beginning the upgrade process.

HDFS Backup​

You need to execute the following action on the NameNode Host . If you have enabled Hive Availability for HDFS, you can find the Active Namenode by connecting to Ambari > HDFS > Summary section.

Connect the NameNode (Active) host:

su -l hdfs

mkdir [NN_UPGRADE_DIR]
cd [NN_UPGRADE_DIR]

# Check that the FileSystem is healthy
hdfs fsck / -files -blocks -locations > backup-hdfs-nn-fsck.log

# List DataNodes
hdfs dfsadmin -report >  backup-hdfs-dn-list.log

Save the Namespace

su -l hdfs

mkdir [NN_UPGRADE_DIR]
cd [NN_UPGRADE_DIR]

# Check that the FileSystem is healthy
hdfs fsck / -files -blocks -locations > backup-hdfs-nn-fsck.log

# List DataNodes
hdfs dfsadmin -report >  backup-hdfs-dn-list.log

Checkpoint HDFS by generating a snapshotimage

hdfs dfsadmin -safemode enter
hdfs dfsadmin -saveNamespace
hdfs dfsadmin -safemode leave
note

Entering Safemode makes HDFS being unaccessible by other service and will cause Cluster Downtime.

Register new ODP Version​

We need to tell Ambari what is the target version and what are the associated repositories. For this we are going to use the ODP 1.2.2.0 VDF file, and give it to Ambari.

Collect the repositories​

For Ubuntu 22:

wget -nv https://archive.clemlab.com/ubuntu22/ambari-release/2.7.9.0.0-110/ODP-VDF.xml apt update -y apt upgrade ambari-agent

For CentOS/RedHat 7:

wget -nv https://archive.clemlab.com/centos7/ambari-release/2.7.9.0.0-110/ODP-VDF.xml apt update -y apt upgrade ambari-agent

For CentOS/RedHat/RockyLinux/AlmaLinux 8:

wget -nv https://archive.clemlab.com/centos8/ambari-release/2.7.9.0.0-110/ODP-VDF.xml apt update -y apt upgrade ambari-agent

Register Procedure​

  • Log in to Ambari Web UI.
  • Click on Cluster Admin > Stack and Version
  • Click on the Versions tab, you should see the current running version Ambari Running version
note

The version displayed must be the source version, and should be supported in upgrade path. The version should be the full ODP version including the build number. In the picture example the full version is ODP 1.2.1.0-134.

  • Click on Manage Versions button. Valid the popup. Now you are redirected to Admin / Versions page.

  • Click on Register.
    On this page you can select the stack Major Stack Version button. For example the button ODP-1.2. Register new version button

    Click on Add Version.
    You have two methods to upload the ODP VDF file:

  • Click on Upload Version Definition File to use a local file

  • Click on Version Definition File URL to use a remote url (Ambari Server needs to be able to communicate with the remote url).

    The VDF must represent the ODP software packages delivery method (ie Operating System version, url repositories). Select the previously VDF file. new version displayed

note

Check very carefully that the Target Version, OS and reporitories match the current 1.2.2.0 documentation version. Request support if any trouble on supported migration path.

Click Save and Dashboard .

Install Software Package​

Now You can go back on Stack and version

  • Log in to Ambari Web UI.

  • Click on Cluster Admin > Stack and Version The new version should be displayed with an Install Packages button.

  • Click on Install Packages and wait for completion. Install Packages


    Once completed the Upgrade button will appear instead of Install Packages

    Upgrade Packages

Perform Upgrade ODP Stack Cluster​

Now You can perform the upgrade procedure. You have two scenarios:

  • Express Upgrade
  • Rolling Upgrade

Performing an Express Upgrade​

Downtime

Services are stopped while the upgrade is performed. Incurs downtime, but faster upgrade. The total time to upgrade depends on several factors: number of node, number of services...

Click Upgrade and select Express .

Before starting the Upgrade procedure Ambari will check that you have performed checks and that your cluster validates the prerequesites.

  • If any of the required prerequesites is not validated, the dialog will display an error. You can see in details what checks failed. required check error required check error details

  • If any of the optional prerequesites is not validated, the dialog will just print a warning. You can proceed with the upgrade procedure.

For information, the upgrade is divided into multiple stages, containing multiple steps. Some of the steps requires administrator to tacke action, such as Yarn application kill, Database Backup. If any of the steps fail, you are prompt for action and can retry afterwards. You can also pause upgrade if you need to execute some longer actions.

  • Prepare Upgrade
  • Stop Components for High-Level Services
  • Perform Backups
  • Stop Components for Core Service
  • Update Target Respository
  • Update Service Configs
  • Restart Components
  • All Service Checks
  • Restart Components
  • Set Version on All Hosts
  • Finalize Upgrade Pre-Check
  • Finalize Upgrade

Once the upgrade is finished you may choose Finalize Upgrade or Finalize Later. You can choose Finalizing later in order to perform deeper checks in order to verify that everything works fine.

Once you have made every validation click on Finalize Upgrade.

Once Finalized, Ambari will remove the starting Version from Stack and Versions tab, and the new version will be displayed as Current

Performing a Rolling Upgrade​

Downtime

Services remain running while the upgrade is performed. Minimized disruption but slower upgrade. The total time to upgrade depends on several factors: number of node, number of services...

Minimal Disruption

The Rolling Upgrade is not a Zero Downtime Upgrade, while it does take longer to perform than an Express Upgrade. You should consider going for an Express Upgrade if time to perform the upgrade is important for you.

Click Upgrade and select Rolling .

Before starting the Upgrade procedure Ambari will check that you have performed checks and that your cluster validates the prerequesites.

  • If any of the required prerequesites is not validated, the dialog will display an error. You can see in details what checks failed. required check error

  • If any of the optional prerequesites is not validated, the dialog will just print a warning. You can proceed with the upgrade procedure.

For information, the upgrade is divided into multiple stages, containing multiple steps. Some of the steps requires administrator to tacke action, such as Yarn application kill, Database Backup. If any of the steps fail, you are prompt for action and can retry afterwards. You can also pause upgrade if you need to execute some longer actions.

  • Prepare Upgrade
  • Stop Components for High-Level Services
  • Perform Backups
  • Stop Components for Core Service
  • Update Target Respository
  • Update Service Configs
  • Restart Components
  • All Service Checks
  • Restart Components
  • Set Version on All Hosts
  • Finalize Upgrade Pre-Check
  • Finalize Upgrade

Once the upgrade is finished you may choose Finalize Upgrade or Finalize Later. You can choose Finalizing later in order to perform deeper checks in order to verify that everything works fine.

Once you have made every validation click on Finalize Upgrade.

Once Finalized, Ambari will remove the starting Version from Stack and Versions tab, and the new version will be displayed as Current