RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker#660
RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker#660
Conversation
… docker Signed-off-by: Ramesh Mani <rmani@apache.org>
There was a problem hiding this comment.
Pull Request Overview
This PR integrates Apache Tez as the processing framework for the ranger-hive Docker setup to enable faster data processing through DAG execution and resolve issues with INSERT commands in beeline.
- Adds Tez binary distribution and configuration files for Hive integration
- Updates Hadoop YARN configuration to support Tez execution
- Creates comprehensive Tez configuration across all Hive database variants
Reviewed Changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tez-site.xml | New Tez configuration template with memory and execution settings |
| ranger-hive-setup.sh | Adds Tez setup, YARN configuration, and HDFS directory creation |
| ranger-hadoop-setup.sh | Enhances YARN configuration and installs Tez JARs for NodeManager |
| hive-site-*.xml | Adds Tez execution engine configuration to all database variants |
| hive-site-metastore-mysql.xml | New metastore-specific configuration with Tez support |
| create-users.sh | New script for creating test users (alice, abram) |
| download-archives.sh | Adds Tez binary download support |
| docker-compose files | Updates build arguments and environment variables for Tez |
| Dockerfiles | Integrates Tez installation and user creation across containers |
| .env | Updates Hadoop version compatibility and adds Tez version |
Comments suppressed due to low confidence (1)
dev-support/ranger-docker/.env:1
- The KAFKA_VERSION line appears to be missing after the HIVE_HADOOP_VERSION change. This could break Kafka-related builds that depend on this environment variable.
BUILD_HOST_SRC=true
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
Thank you @rameeshm for the patch, I believe this is tested with Ubuntu base image, please see if this can be tested with UBI base image as well, this change needs to made in |
… docker - changes to use ranger base image for user creation, fix issue with usage of ranger base image in other containers
@kumaab current patch with the review comments tested with RANGER_BASE_VERSION=[20250712-1-ubi-8] |
… docker - addressed review comment on issue related to base immage
… docker - address review comment on the switch user statement
kumaab
left a comment
kumaab
left a comment
There was a problem hiding this comment.
Looks good, few minor changes.
… docker - address review comment
|
@kumaab Addressed the review comments. Thanks |
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated 1 comment.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The docker build seems to be failing, please retain |
… docker - fixed Ozone version
kumaab
left a comment
kumaab
left a comment
There was a problem hiding this comment.
Looks good, tested the changes on my local as well as the docker build was failing in CI. thank you!
…ve docker (apache#660) * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker Signed-off-by: Ramesh Mani <rmani@apache.org> * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - Review comments adddress, hadoop and hive ssh issue while startup addressed, removed not need configs * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - changes to use ranger base image for user creation, fix issue with usage of ranger base image in other containers * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - addressed review comment on issue related to base immage * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - address review comment on the switch user statement * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - address review comment * Update dev-support/ranger-docker/scripts/hive-site-oracle.xml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - fixed Ozone version --------- Signed-off-by: Ramesh Mani <rmani@apache.org> Co-authored-by: Ramesh Mani <rmani@apache.org> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
| # Zookeeper Configuration | ||
| ZK_VERSION=3.8.4 | ||
|
|
||
| # Database Versions |
There was a problem hiding this comment.
Looks like the DB versions were altered in this PR for all DB flavors, with some versions going back:
Earlier:
MARIADB_VERSION=10.7.3
ORACLE_VERSION=23.6
Now:
MARIADB_VERSION=10.6
ORACLE_VERSION=21.3.0-xe
Is this intentional ? thanks.
There was a problem hiding this comment.
@kumaab Those worked without issue. I shall check on with what was there, if needed I shall upgrade it, test it and send a new patch
There was a problem hiding this comment.
Sure, please check.
While testing upgrades from 2.6 and 2.7 to 2.8.0-SNAPSHOT, the upgrade with Oracle failed: https://github.com/kumaab/ranger/actions/runs/20984849291
…ve docker (apache#660) * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker Signed-off-by: Ramesh Mani <rmani@apache.org> * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - Review comments adddress, hadoop and hive ssh issue while startup addressed, removed not need configs * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - changes to use ranger base image for user creation, fix issue with usage of ranger base image in other containers * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - addressed review comment on issue related to base immage * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - address review comment on the switch user statement * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - address review comment * Update dev-support/ranger-docker/scripts/hive-site-oracle.xml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * RANGER-5310:Include Apache Tez as the process framework for ranger-hive docker - fixed Ozone version --------- Signed-off-by: Ramesh Mani <rmani@apache.org> Co-authored-by: Ramesh Mani <rmani@apache.org> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
What changes were proposed in this pull request?
Include Apache Tez as the process framework for ranger-hive docker
How was this patch tested?
Testing in Docker running HiveServer 2 beeline and execute "INSERT" statement for DAG.
