Release Notes
Python2 and Rhel 7 EOL
OpenSource Data Platform (ODP) 1.3.1.0 dropped support of python2 and RedHat 7. Rehdat7 / Python2 built can be requested from the support team. For upgrade scenarios, please refer to the [Upgrade Guide](/docs/1.3.1.0/upgrade/upgrade-introduction)
ODP 1.3 is the next major version of ODP Stack after 1.2. It includes new features, improvements, and bug fixes.
[New Features]
JDK 8 and JDK 17 Support
ODP 1.3 provides complete support of JDK 8 and JDK 17 runtime support for all stack components. ODP core components are still build with JDK 8 to provide backward compatibility with existing clusters running with JDK 8.
Ambari 2.8.1.0 requires JDK 17 to run and has a new option to configure separately the JAVA JDK used of Ambari server/agent and the JAVA JDK used by the ODP stack components.
The --ambari-java-home options has been added to ambari-server program to setup Ambari server JAVA_HOME separately from the ODP stack components JAVA_HOME.
To know more about you can read the ambari-server setup documentation.
The upgrade documentation explain also how to setup both JDK separately.
Additional Operating System Support
Ubuntu 24.04 has been added to the list of supported operating systems starting with Ambari 2.8.1.0 and ODP 1.3.1.0.
Python version
Python3.12 has been added to the list of supported Python versions for Ambari server and agent.
[Behavioral Changes]
Ambari
Ambari 2.8.1.0 introduces the following behavioral changes:
- The Ambari server and agent now run with Python 3. Python 2 is no longer supported.
- The Ambari server now requires JDK 17 to run. JDK 8 is no longer supported for Ambari server.
- The default JAVA_HOME for ODP stack components is still JDK 8, but can be changed to JDK 17 if needed.
- The Ambari server and agent now run with JDK 17. JDK
Apache Iceberg Support
Iceberg has been introduced and is enabled natively in Apache Hive 4.0.1 and Apache Spark 3.5.6.
Aarch64 Support
aarch64 build of ODP is now production ready and supported on all ODP stack components (it was introduced as a tech-preview in ODP 1.2.2.0)
Version Upgrade
Major upgrade of Stack components version
- Upgrade Apache Atlas version to 2.4.0
- Upgrade Apache Flink version to 1.20.0
- Upgrade Apache Hadoop 3 version to 3.4.1
- Upgrade Apache HBase 3 version to 2.6.4
- Upgrade Apache Hive 3 version to 4.0.1
- Upgrade Apache Knox version to 2.1.0
- Upgrade Apache Livy version to 0.8.0
- Upgrade Apache Ozone to version 2.0.0
- Upgrade Apache Ranger version to 2.6.0
- Upgrade Apache Spark 3 version to 3.5.6
- Upgrade Apache Tez version to 0.10.4
- Upgrade Apache Zeppelin version to 0.10.2
- Upgrade Apache ZooKeeper version to 3.9.3
[Changelog]
Apache Ambari
| Issue Type | Key | Summary |
|---|---|---|
| Improvement | AMBARI-295 | Improve YARN NM and MR JHS daemon stop starting on ODP 1.3 |
| Task | AMBARI-294 | Use Execute function to backup knox data instead of os |
| Task | AMBARI-293 | Update YARN, MAPREDUCE, TEZ service check to use default config dir during rolling upgrades |
| Bug | AMBARI-290 | Fix word breaking on hive configuration page after jquery update |
| Bug | AMBARI-289 | Fix hive-env in ODP hive preventing Hive LLAP to start due to missing credentials |
| Bug | AMBARI-288 | Fix hive llap python2 to python3 migration |
| Bug | AMBARI-287 | Fix template ambari-env preventing ambari from starting after multiple upgrades |
| Bug | AMBARI-286 | Handle missing RCA tables during cleanup |
| Sub-task | AMBARI-284 | TEZ update service advisor to support JDK 11/17 runtime |
| Task | AMBARI-281 | Harden existence check in apt manager using package name and output |
| Task | AMBARI-280 | Improve spark3 service advisor on spark.yarn.historyServer.address |
| Task | AMBARI-279 | Migrate SPARK3 from log4j to log4j2 |
| New Feature | AMBARI-278 | Fix quicklinks.json and alerts.json for SPARK3 |
| Sub-task | AMBARI-277 | Add regex raw attribute in ambari-agent |
| Sub-task | AMBARI-276 | Rename getargspec to getfullargspec for python 3.12 compatibility |
| Task | AMBARI-275 | Handle hadoop conf dir not existing by deb/rpm |
| Bug | AMBARI-274 | Metrics not displaying in HDFS, HBASE, YARN, AMS widgets |
| Sub-task | AMBARI-273 | Refactor hadoop-env write to add odp.version and write date |
| Sub-task | AMBARI-272 | Add raw attribute in regexp to remove syntax warning in ambari-common |
| Sub-task | AMBARI-271 | Wrap socket in ssl for ws4py client |
| Bug | AMBARI-269 | Use new read_string method instead of readfp to support python 3.12 |
| New Feature | AMBARI-267 | Add support on ubuntu24 |
| New Feature | AMBARI-266 | Separate core-site from HDFS to allow Apache Ozone powered cluster |
| Improvement | AMBARI-265 | First step on initial working deployment |
| Epic | AMBARI-264 | Update ozone-env for compatibility with JDK 11+ JVM options |
| Epic | AMBARI-263 | Create an ambari view to manage and upload helm releases |
| Task | AMBARI-262 | Add new supported components to stack upgrade from ODP 1.2 to 1.x |
| Bug | AMBARI-261 | Set RANGER_LOG4J_SUPPORT max version to ODP 1.2 |
| Bug | AMBARI-259 | Correct UpgradeHistory to Upgrade entity mapping |
| Task | AMBARI-258 | Add ODP 1.3 stack upgrade files |
| Bug | AMBARI-257 | Update pre-requisite DTOs for Jackson 2 & JDK 17 compatibility |
| Sub-task | AMBARI-256 | Execute DML update on ambari-server to support INFRA SOLR ENV java version before stack upgrade |
| Sub-task | AMBARI-255 | Execute DML update on ambari-server to support AMS java version before stack upgrade |
| Bug | AMBARI-254 | Fix Ranger Dashboard Ranger Tagsync link after JDK 11/17 runtime upgrade |
| Bug | AMBARI-253 | Fix hdp.version still present in tez-site and mapred-site instead of odp.version |
| Bug | AMBARI-252 | Fix admin_log4j property not picked up |
| Sub-task | AMBARI-251 | Add upgrade scripts on Env to support JDK from 2.7.x to 2.8.1 |
| Task | AMBARI-248 | Move from flink-conf.yaml to config.yaml with ODP 1.3 |
| Sub-task | AMBARI-244 | Support JDK 11/17 runtime on Hive |
| Sub-task | AMBARI-243 | Support JDK 11/17 runtime on Ambari Infra Solr |
| Sub-task | AMBARI-242 | Support JDK 11/17 runtime on ODP Ranger KMS |
| Sub-task | AMBARI-241 | Support JDK 11/17 runtime on ODP Ranger |
| Sub-task | AMBARI-240 | Support JDK 11/17 runtime on ODP YARN/MAPREDUCE |
| Sub-task | AMBARI-239 | Support JDK 11/17 runtime on ODP HDFS |
| Task | AMBARI-235 | Remove EventCounter class from log4j properties starting with ODP 1.3 |
| Task | AMBARI-233 | Add janus kerberos property when kerberos is enabled |
| Task | AMBARI-232 | Remove jmx handler from ranger solr config xml |
| Task | AMBARI-231 | Create atlas user if atlas_server does not exist |
| Bug | AMBARI-229 | Improve setup credential logic when checking existing alias |
| Bug | AMBARI-228 | Fix ClusterDAO returning empty collection after JDK upgrade |
| Bug | AMBARI-226 | Fix hue setup command running as hue user when ambari runs as non-root |
| Bug | AMBARI-225 | Fix import on knox service advisor and flink incorrect access to properties |
| Task | AMBARI-224 | Set ambari_java_version and align it with ambari_java_home |
| Bug | AMBARI-222 | Detect java version from JDK home when using --java-home option |
| Bug | AMBARI-220 | Add hbase rest support and normalize scripts with hbase thrift |
| Task | AMBARI-219 | Replace log4j.properties with logback.xml |
| Bug | AMBARI-215 | Move ranger log4j to logback |
ODP Stack
| Issue Type | Key | Summary |
|---|---|---|
| Bug | ODP-328 | Update service-dep.tar.gz to fix LLAP start |
| Task | ODP-327 | Update Ozone to 2.0.0 |
| Task | ODP-326 | Fix Apache Knox Ranger plugin unable to download policies or write audit since Knox 2.1 |
| Task | ODP-325 | Update Knox to 2.1.0 in ODP 1.3 |
| New Feature | ODP-324 | Spark: Replace SQLContext with SparkSession in STS |
| New Feature | ODP-323 | Initial release of KEDA component in ODP 1.3.1.0 |
| Bug | ODP-322 | Fix python2 dependency in Spark Debian packages |
| Task | ODP-313 | Set -Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true in knox-functions.sh |
| Task | ODP-312 | Port KNOX-2946: Cookie path scoping does not work with default topology URL |
| Sub-task | ODP-311 | Remove Tez embedded EventCounter |
| Sub-task | ODP-310 | Remove Jersey dependencies in Ranger HBase plugin |
| Sub-task | ODP-309 | Update managed-schema to match Solr 8.11.4 |
| Sub-task | ODP-308 | Apply RANGER-4821 patch |
| Task | ODP-307 | Update Ranger to 2.6.0 in ODP 1.3 |
| Bug | ODP-306 | Enable ATS hook on Hive 4 |
| Sub-task | ODP-303 | Update Kafka symlink build on Hadoop-shaded Guava and Protobuf |
| Bug | ODP-302 | Add JAVA_HOME option in Knox to support JDK 8 for CLIs and service start |
| Sub-task | ODP-300 | Add hadoop-shaded guava jar in Ranger service libs (1.3.0 for Hadoop 3.4.1) |
| Sub-task | ODP-299 | Update commons-cli to match Hadoop version |
| Task | ODP-297 | Exclude org.jetbrains.pty4j from ranger-knox-plugin |
| Task | ODP-296 | Create symlinks on slf4j instead of real jar |
| Task | ODP-294 | Exclude Atlas jars from hive-exec |
| Bug | ODP-293 | Hadoop does not compile on x86_64: update Protobuf to 3.21.12 |
| Epic | ODP-292 | Update NiFi to 1.28 in ODP 1.3.1.0 |
| Sub-task | ODP-290 | HIVE-27828: Enable copy-on-write update for Iceberg without breaking Atlas hooks |
| Bug | ODP-289 | Update Atlas to 2.4.0 |
| Epic | ODP-287 | Update Apache Hive to 4.0.1 |
| Sub-task | ODP-286 | Apply RANGER-4952 to support Hive 4.0.1 |
| Sub-task | ODP-285 | Fix port of RANGER-2324 |
| Task | ODP-284 | Update Apache Ranger to 2.5.0 |
| Task | ODP-283 | Update Phoenix to 5.2.1 and phoenix-queryserver to phoenix-db-1.2.2 |
| Task | ODP-282 | Update Sqoop dependencies to ODP 1.2.2.0-138 |
| Sub-task | ODP-281 | Adapt HIVE-27925: Unify HiveConf ConfVars enum and improve readability |
| Task | ODP-280 | Undo private access on cleanupLogs for Spark 3 |
| Sub-task | ODP-279 | Remove test on clean logs on Windows as cleanerLogs is private |
| Sub-task | ODP-278 | Adapt KAFKA-7632: Support compression levels (KIP-390) |
| Sub-task | ODP-277 | Backport SPARK-47738: Upgrade Kafka to 3.7.0 |
| Sub-task | ODP-276 | Backport SPARK-47679: Use HiveConf.getConfVars or direct conf names |
| Task | ODP-275 | Update Hive to 4.0.1 in ODP |
| Sub-task | ODP-274 | Backport SPARK-46697: Eliminate Hive configuration time variables conversion |
| Bug | ODP-273 | Update Spark 3 to 3.5.4 in ODP 1.3.1.0 |
| Bug | ODP-272 | Port HIVE-20065 to build Hive 3.1.3 after Hadoop 3.4.1 upgrade |
| Sub-task | ODP-270 | Update existing patch on husky build and releasing script |
| Improvement | ODP-269 | Update Flink to 1.20.0 |
| Improvement | ODP-268 | Update Kafka version to 3.8.1 |
| Improvement | ODP-267 | Port HBASE-28939: Change default Hadoop 3 version to 3.4.1 |
| Bug | ODP-266 | Port HBASE-29021 to fix upgrade-related issues |
| Improvement | ODP-265 | Adapt existing patch on HBASE-28153 |
| Epic | ODP-264 | Update HBase to 2.6.1 in ODP 1.3.1.0 |
| Task | ODP-263 | Update Tez to 0.10.4 in ODP 1.3.1.0 |
| Improvement | ODP-261 | Update HTTPFS gateway code to match Hadoop 3.4.x (HDFS-16129) |
| Task | ODP-260 | Implement WriteUnlock and ReadUnLock from HDFS-16434 |
| Improvement | ODP-259 | Update Hadoop version to 3.4.1 |
| Improvement | ODP-258 | Port HDDS-11617 to build 1.4.1 with Hadoop 3.4.1 |
| Epic | ODP-257 | Update Ozone to 1.4.1 |
| Sub-task | ODP-229 | Update Kafka build to use Scala 2.13 binary |
| Sub-task | ODP-223 | [LIVY] Update Scalatra test to 2.7.2 for Scala 2.13 compatibility |
| Sub-task | ODP-222 | Update Flink to support Scala 2.13 |
| Sub-task | ODP-221 | Update Spark 3 with Scala 2.13 binary |
| Sub-task | ODP-220 | Update Ranger Scala binary version |
| Sub-task | ODP-219 | Update Hive binary version to 2.13.11 |
| Sub-task | ODP-217 | Update Livy to support Scala 2.13.11 (patch + component build) |