Cloud-Native Apache Hive: A Deep Dive into the New Kubernetes Operator, Autoscaling, and Multi-Tenancy

Ayush Saxena

14 min read

·

Jun 23, 2026

--

Listen

Share

Press enter or click to view image in full size

Apache Hive has taken a massive leap forward in its cloud-native journey. With the introduction of the new Kubernetes Operator, managing a Highly Available (HA) Apache Hive cluster on Kubernetes is no longer an exercise in writing endless YAML manifests. It is now entirely declarative.

A single HiveCluster Custom Resource Definition (CRD) now manages the entire lifecycle of the Hive Metastore (HMS), HiveServer2 (HS2), LLAP daemons, and Tez Application Masters (TezAM).

But the operator goes far beyond simple deployment. By orchestrating the latest releases of Apache Hive, it unlocks native zero-dependency autoscaling, scale-to-zero hibernation, multi-tenant LLAP clusters, and out-of-the-box Apache Iceberg support.

In this deep dive, we will explore the theory behind these features, explain the CRD configuration parameters in detail, and provide real-world examples for deploying via Helm.

1. Prerequisites & Dependencies

Before deploying the Hive Operator, you need a few core dependencies in your Kubernetes cluster.

• Database: A backend database for the Hive Metastore. The operator supports PostgreSQL, MySQL, Derby, Oracle, and MSSQL. (Note: You can also skip deploying a database entirely and connect to an existing external HMS).
• ZooKeeper: Required for LLAP service discovery and Tez session registries.
• Storage: Any Hadoop-compatible file system (Apache Ozone, AWS S3, GCS, ABFS, HDFS).

2. Storage Configurations, Hadoop Overrides, and External HMS

The Hive Operator is completely storage-agnostic. It seamlessly integrates with any Hadoop-compatible file system by injecting overrides directly into core-site.xml and mounting the necessary credentials.

Storage & Native Hive Overrides

You can pass custom properties to core-site.xml using coreSiteOverrides, and to hive-site.xml or llap-daemon-site.xml using configOverrides.

# values.yaml
cluster:
  database:
    type: postgres
    url: "jdbc:postgresql://postgres-postgresql:5432/metastore"
    driver: "org.postgresql.Driver"
    username: hive
    passwordSecretRef:
      name: hive-db-secret
      key: password
    driverJarUrl: "https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar"
  zookeeper:
    quorum: "zookeeper:2181"
  storage:
    coreSiteOverrides:
      fs.defaultFS: "s3a://hive"
      fs.s3a.endpoint: "http://ozone-s3g-rest:9878"
      fs.s3a.path.style.access: "true"
    envVars:
      - name: HADOOP_OPTIONAL_TOOLS
        value: "hadoop-aws"
      - name: AWS_ACCESS_KEY_ID
        value: "ozone"
      - name: AWS_SECRET_ACCESS_KEY
        value: "ozone"
  # Native Hive Overrides
  hiveServer2:
    replicas: 2
    configOverrides:
      hive.server2.enable.doAs: "false"

External Metastore (Bring Your Own HMS)

If your organization already has a central Hive Metastore running elsewhere, you don’t have to deploy a new one. You can instruct the operator to skip the HMS deployment and point HiveServer2 directly to your existing endpoint by setting metastore.enabled: false and providing an externalUri:

cluster:
  database: {} # Not needed when metastore is external
  zookeeper:
    quorum: "zookeeper:2181"
  metastore:
    enabled: false
    externalUri: "thrift://my-external-metastore.company.internal:9083"

3. Intelligent Autoscaling: Zero-Dependency Elasticity

In the cloud, provisioning for “peak load” means you are burning money at 3:00 AM when your cluster is idle.

Traditionally, autoscaling on Kubernetes requires a massive operational stack: Prometheus to scrape metrics, Prometheus Adapter to translate them, and Horizontal Pod Autoscalers (HPAs) to act on them.

The new Hive Operator throws all of that out the window. It features a native, operator-driven control loop. When you enable autoscaling, the operator automatically injects a JMX Exporter agent into your Hive pods. The operator itself scrapes these pods and scales the StatefulSets and Deployments dynamically.

The Autoscaling Parameters Explained

Every component (HiveServer2, Metastore, LLAP, TezAM) accepts an autoscaling block that lets you fine-tune the operator's responsiveness:

• enabled: Activates autoscaling for the component.
• minReplicas: The absolute floor for scaling. Set to 0 for LLAP and TezAM to enable scale-to-zero capabilities. HS2 and HMS enforce a minimum of 1 to ensure the cluster remains reachable.
• scaleUpThreshold: The metric target per pod that triggers scale-up. For HS2, this is the number of open sessions per pod. For HMS, it is the API request rate. For LLAP, it is the number of busy task slots. (TezAM scales purely on session demand, so it ignores this).
• metricsScrapeIntervalSeconds: How frequently the operator's internal control loop polls the JMX endpoint on your active pods. Lower values (e.g., 10s) provide faster response times.
• scaleUpStabilizationSeconds: A smoothing buffer to prevent flapping. The operator looks at the highest replica recommendation over this time window before committing to a scale-up.
• scaleDownStabilizationSeconds: The cooldown period. Traffic must remain consistently low for this duration before the operator safely removes pods.
• gracePeriodSeconds: The maximum safety cap for the graceful preStop drain. The operator gracefully waits for sessions/queries to finish before killing the pod. If active sessions drop to 0 before this timer expires, the pod terminates immediately.

Dual-Signal Scaling (Metrics + CPU)

What if only a few Data Engineer submits a monstrous ETL join? The session count is low, but the CPU is pinned at 100%.

To solve this, the operator uses Dual-Signal Scaling for HS2 and HMS. It calculates desired replicas based on session/API metrics and CPU utilization, taking the max() of both signals.

• cpuScaleUpThreshold: The CPU percentage (0-100) that triggers scale-up.
• cpuScaleDownThreshold: The CPU percentage below which scale-down is considered.

Crucial Tip: Always set resources.limitsCpu in your Kubernetes spec when using CPU-based autoscaling! The JMX metric (ProcessCpuLoad) calculates CPU usage as a fraction of available processors. Without CPU limits, the JVM looks at the underlying physical Kubernetes Node. If your pod is on a 16-core node and maxes out 1 core, JMX will report 6.25% CPU usage, meaning your autoscaler may never trigger.

Example YAML Configuration for Autoscaling:

cluster:
  hiveServer2:
    replicas: 10 # Acts as the MAXIMUM ceiling when autoscaling is on
    resources:
      limitsCpu: "2"
    autoscaling:
      enabled: true
      minReplicas: 1
      scaleUpThreshold: 80
      metricsScrapeIntervalSeconds: 10
      scaleUpStabilizationSeconds: 60
      scaleDownStabilizationSeconds: 600
      cpuScaleUpThreshold: 90
      cpuScaleDownThreshold: 30

4. Auto-Suspend: Full Cluster Hibernation

Autoscaling LLAP to 0 is great, but HiveServer2 and the Metastore still run 24/7. For development, testing, or intermittent analytics environments, the operator introduces Auto-Suspend.

When enabled, the operator monitors the cluster. If all components are sitting at their minReplicas and HiveServer2 registers 0 open sessions for a configured duration, the operator force-scales the entire cluster (including HS2 and HMS) to 0.

Auto-Suspend Parameters

• enabled: Activates the hibernation routine.
• idleTimeoutMinutes: Minutes of 0 active sessions before the cluster auto-suspends.
• includeMetastore: If true, the HMS also scales to 0. If false, HMS stays alive at minReplicas to serve external metadata requests.

Observing the Hibernation Lifecycle

You can watch the operator transition through the suspension lifecycle in real time using the kubectl get command:

# 1. Active cluster processing queries
$ kubectl get hivecluster hive
NAME   PHASE     IDLE (MIN)
hive   Running   

# 2. Queries have finished, and the cluster begins waiting for new work
$ kubectl get hivecluster hive 
NAME   PHASE     IDLE (MIN)
hive   Idle      10

# 3. The idle timeout is reached; the cluster hibernates to save resources
$ kubectl get hivecluster hive 
NAME   PHASE       IDLE (MIN) 
hive   Suspended

To wake the cluster up manually, simply patch the CRD:

# Wake the cluster back up
kubectl patch hivecluster hive --type=merge -p '{"spec":{"suspend":false}}'
# Force hibernation immediately (bypassing the idle timer)
kubectl patch hivecluster hive --type=merge -p '{"spec":{"suspend":true}}'

5. Multi-Tenant LLAP: Beating the “Noisy Neighbor”

Press enter or click to view image in full size

In a traditional big data environment, sharing a single compute pool causes the dreaded “Noisy Neighbor” problem. If Alice runs a massive ETL pipeline, she consumes all the LLAP executors. When Bob (a BI Analyst) opens his executive dashboard, his sub-second ad-hoc queries queue behind Alice’s massive joins.

Multi-Tenant LLAP solves this by allowing you to define multiple, completely independent LLAP clusters within a single Hive deployment. Each cluster gets its own StatefulSet, autoscaling rules, and dedicated Tez Application Masters.

This is how the magnified flow chart looks looks like

Press enter or click to view image in full size

Server-Side Routing

The operator uses Server-Side Routing via the llapClusterRouting property. Clients require zero configuration—Hive routes them automatically based on their username or group.

Format: user:<name>=<cluster>,group:<name>=<cluster>,default=<cluster>

Multi-Tenant YAML Example

Here is how you isolate Alice from Bob:

cluster:
  # Map users/groups to specific compute pools
  llapClusterRouting: "user:alice=etl,group:analysts=bi,default=dev"
  
  llapClusters:
    # Pool 1: ETL (Alice) - Large memory, highly scalable
    - name: etl
      replicas: 6
      executors: 4
      memoryMb: 8192
      autoscaling:
        enabled: true
        minReplicas: 2
      tezAm:
        autoscaling:
          enabled: true
          minReplicas: 2
    
    # Pool 2: BI (Analysts) - Smaller tasks, scales to zero aggressively
    - name: bi
      replicas: 4
      executors: 2
      memoryMb: 4096
      autoscaling:
        enabled: true
        minReplicas: 0
      tezAm:
        autoscaling:
          enabled: true
          minReplicas: 0

When Bob connects via Beeline (-n bob), HiveServer2 intercepts the connection, evaluates the routing rules, and seamlessly points his session to the isolated bi compute pool. His queries trigger scale-up events for that specific LLAP cluster, completely unaffected by Alice's ETL workload!

6. Connecting via Beeline & Running Apache Iceberg

With the new operator, HiveServer2 runs in HTTP transport mode by default. Because routing is handled server-side, the client requires no complex JDBC strings.

Because the operator deploys the latest Apache Hive, you get powerful modern table formats right out of the box. You can create standard native Hive tables, or seamlessly dive into Apache Iceberg.

Let’s port-forward to test the connection:

# Terminal 1: Forward the HTTP port
kubectl port-forward svc/hive-hiveserver2 10001:10001

# Terminal 2: Connect using Beeline
beeline -u "jdbc:hive2://localhost:10001/;transportMode=http;httpPath=cliservice" -n alice

The Iceberg Experience

Once connected, you can immediately start creating and querying Iceberg tables, as well as inspecting Iceberg’s powerful hidden metadata tables:

0: jdbc:hive2://hive-hiveserver2:10001/> create table emp (id int) stored by iceberg;
0: jdbc:hive2://hive-hiveserver2:10001/> insert into emp values (1),(2),(3),(4);
0: jdbc:hive2://hive-hiveserver2:10001/> select * from emp;
+---------+
| emp.id  |
+---------+
| 1       |
| 2       |
| 3       |
| 4       |
+---------+
4 rows selected (1.125 seconds)
0: jdbc:hive2://hive-hiveserver2:10001/> select * from default.emp.files;
+--------------------+----------------------------------------------------+------------------------+--------------------+
| emp.files.content  |                 emp.files.file_path                | emp.files.file_format  | emp.files.spec_id  | 
+--------------------+----------------------------------------------------+------------------------+--------------------+
| 0                  | s3g://ayush-ozone-bucket/hive/warehouse/emp/data/...  | PARQUET                | 0                  | 
+--------------------+----------------------------------------------------+------------------------+--------------------+
1 row selected (2.675 seconds)

7. Observability: Real-World Operator Status

The operator provides extensive visibility into the state of your cluster. To see exactly what the autoscaler is doing under the hood, describe the output as YAML:

$ kubectl get hivecluster hive -o yaml

You will see rich status blocks tracking precise autoscaling metrics for each individual component and tenant. Notice how it tracks the currentMetricValue, currentCpuPercent, and exactly what the autoscaler's proposedReplicas are:

status:
  clusterPhase: Running
  conditions:
  - lastTransitionTime: "2026-06-22T08:00:31.996154087Z"
    message: All Hive components are ready
    reason: AllComponentsReady
    status: "True"
    type: Ready
  hiveServer2:
    currentReplicas: 2
    maxReplicas: 2
    minReplicas: 2
    phase: Running
    readyReplicas: 2
  llapClusters:
    llap0:
      autoscaling:
        currentMetricValue: 0
        lastScaleTime: "2026-06-22T08:16:34.000537797Z"
        proposedReplicas: 0
        scaleUpThreshold: 1
      currentReplicas: 0
      maxReplicas: 2
      minReplicas: 0
      phase: Idle
      readyReplicas: 0
  metastore:
    autoscaling:
      cpuProposedReplicas: 1
      cpuScaleUpThreshold: 90
      currentCpuPercent: 0.2079002079002079
      currentMetricValue: 0
      lastScaleTime: "2026-06-22T08:09:37.666901757Z"
      proposedReplicas: 1
      scaleUpThreshold: 5
    currentReplicas: 1
    maxReplicas: 2
    minReplicas: 1
    phase: Running
    readyReplicas: 1
  observedGeneration: 1

8. The Configuration Reference (The Ultimate CRD YAML)

Instead of a fragmented list of API properties, the following represents a comprehensive HiveCluster Custom Resource YAML via a values file.

This single file replaces dozens of Deployments, ConfigMaps, and Services. It provides a blueprint for overriding Hadoop configurations (core-site.xml), injecting Hive and Tez parameters, defining database connections, and establishing multi-tenant autoscaling.

# full-stack-values.yaml
cluster:
  image: "ayushtkn/hive:4.3.0-SNAPSHOT"
  
  # ---------------------------------------------------------------------------
  # 1. DATABASE & METASTORE CONFIG
  # ---------------------------------------------------------------------------
  database:
    type: postgres
    url: "jdbc:postgresql://postgres-postgresql:5432/metastore"
    driver: "org.postgresql.Driver"
    username: hive
    passwordSecretRef:
      name: hive-db-secret
      key: password
    driverJarUrl: "https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar"
metastore:
    enabled: true
    replicas: 2
    autoscaling:
      enabled: true
      minReplicas: 1
      scaleUpThreshold: 5                 # API calls per second per pod
      cpuScaleUpThreshold: 90             # Scale up if CPU hits 90%
      cpuScaleDownThreshold: 30
  # ---------------------------------------------------------------------------
  # 2. ZOOKEEPER & STORAGE OVERRIDES
  # ---------------------------------------------------------------------------
  zookeeper:
    quorum: "zookeeper:2181"
  storage:
    coreSiteOverrides:
      fs.defaultFS: "s3a://hive"
      fs.s3a.endpoint: "http://ozone-s3g-rest:9878"
      fs.s3a.path.style.access: "true"
    envVars:
      - name: HADOOP_OPTIONAL_TOOLS
        value: "hadoop-aws"
      - name: AWS_ACCESS_KEY_ID
        value: "ozone"
      - name: AWS_SECRET_ACCESS_KEY
        value: "ozone"
  # ---------------------------------------------------------------------------
  # 3. HIVESERVER2 & HIBERNATION
  # ---------------------------------------------------------------------------
  autoSuspend:
    enabled: true
    idleTimeoutMinutes: 15    
    includeMetastore: true 
  hiveServer2:
    replicas: 2
    configOverrides:
      hive.server2.enable.doAs: "false"
    resources:
      limitsCpu: "2"          # CRITICAL for accurate CPU autoscaling!
    autoscaling:
      enabled: true
      minReplicas: 1
      scaleUpThreshold: 80    # Active sessions per pod
  # ---------------------------------------------------------------------------
  # 4. MULTI-TENANT LLAP & TEZ
  # ---------------------------------------------------------------------------
  llapClusterRouting: "user:alice=llap0,user:bob=llap1,default=llap2"
  
  llapClusters:
    - name: llap0
      enabled: true
      replicas: 2
      executors: 1
      memoryMb: 1024
      configOverrides:
        hive.llap.daemon.task.scheduler.wait.queue.size: "1"
      autoscaling:
        enabled: true
        minReplicas: 0       # Scale to zero when idle
        scaleUpThreshold: 1
      tezAm:
        replicas: 2
        autoscaling:
          enabled: true
          minReplicas: 0
    
    - name: llap1
      enabled: true
      replicas: 2
      executors: 1
      autoscaling:
        enabled: true
        minReplicas: 0
        scaleUpThreshold: 1
      tezAm:
        replicas: 2
        autoscaling:
          enabled: true
          minReplicas: 0
          
    - name: llap2
      enabled: true
      replicas: 2
      executors: 1
      autoscaling:
        enabled: true
        minReplicas: 0
        scaleUpThreshold: 1
      tezAm:
        replicas: 2
        autoscaling:
          enabled: true
          minReplicas: 0

Deploying this is as simple as:

helm install hive oci://registry-1.docker.io/ayushtkn/hive-operator \
  --version 4.3.0-SNAPSHOT \
  -f full-stack-values.yaml

9. The Quick-Start Helm Commands

Disclaimer: The following commands utilize a custom OCI registry image (ayushtkn/hive) containing a patched operator for testing purposes. For production deployments, ensure you are using the official Apache Hive Docker images once released.

If you prefer deploying without a values.yaml file, you can pass all configurations via the Helm CLI using the multi-stage quick-start below.

Step 1: Install Core Cluster Dependencies

Every Hive deployment requires a central coordination service and a metadata catalog database. Run these commands to spin up ZooKeeper and PostgreSQL.

# 1. Install ZooKeeper for service discovery and session management
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install zookeeper bitnami/zookeeper \
  --set replicaCount=1 --set auth.enabled=false \
  --set image.repository=bitnamilegacy/zookeeper \
  --set image.tag=3.9.3-debian-12-r21 \
  --set global.security.allowInsecureImages=true --wait

# 2. Install PostgreSQL to serve as the Hive Metastore database
helm install postgres bitnami/postgresql \
  --set auth.username=hive --set auth.password=hive123 \
  --set auth.database=metastore --wait

# 3. Securely store the database credentials in a Kubernetes Secret
kubectl create secret generic hive-db-secret --from-literal=password=hive123

Step 2: Install Storage (Apache Ozone)

If you are testing locally and do not have an existing cloud object store (like Apache Ozone, AWS S3 or GCS), execute the following commands to provision Apache Ozone as your local S3-compatible file system. Skip this step entirely if you plan to target existing or native cloud storage buckets.

# 1. Install the local Apache Ozone cluster
helm repo add ozone https://apache.github.io/ozone-helm-charts/
helm install ozone ozone/ozone --version 0.2.0 --wait
sleep 50

# 2. Provision the storage volume and target data bucket inside Ozone
kubectl exec statefulset/ozone-om -- ozone sh volume create /s3v
kubectl exec statefulset/ozone-om -- ozone sh bucket create /s3v/hive

Step 3: Deploy the Hive Operator (Choose Your Scenario)

(Notice how backslashes \ are used to escape dots in coreSiteOverrides, preventing Helm from interpreting them as nested YAML objects).

Scenario A: Auto-Scaling Disabled (Apache Ozone)

Sets up a static, single-tenant Hive cluster pointing to our local Apache Ozone bucket and Postgres database.

helm install hive oci://registry-1.docker.io/ayushtkn/hive-operator \
  --version 4.3.0-SNAPSHOT \
  --set operator.image.repository="ayushtkn/hive" \
  --set operator.image.tag="operator-4.3.0-SNAPSHOT" \
  --set cluster.image="ayushtkn/hive:4.3.0-SNAPSHOT" \
  --set cluster.database.type=postgres \
  --set cluster.database.url="jdbc:postgresql://postgres-postgresql:5432/metastore" \
  --set cluster.database.driver="org.postgresql.Driver" \
  --set cluster.database.username=hive \
  --set cluster.database.passwordSecretRef.name=hive-db-secret \
  --set cluster.database.passwordSecretRef.key=password \
  --set cluster.database.driverJarUrl="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar" \
  --set cluster.zookeeper.quorum="zookeeper:2181" \
  --set cluster.storage.coreSiteOverrides."fs\.defaultFS"="s3a://hive" \
  --set cluster.storage.coreSiteOverrides."fs\.s3a\.endpoint"="http://ozone-s3g-rest:9878" \
  --set-string cluster.storage.coreSiteOverrides."fs\.s3a\.path\.style\.access"=true \
  --set 'cluster.storage.envVars[0].name=HADOOP_OPTIONAL_TOOLS' \
  --set 'cluster.storage.envVars[0].value=hadoop-aws' \
  --set 'cluster.storage.envVars[1].name=AWS_ACCESS_KEY_ID' \
  --set 'cluster.storage.envVars[1].value=ozone' \
  --set 'cluster.storage.envVars[2].name=AWS_SECRET_ACCESS_KEY' \
  --set 'cluster.storage.envVars[2].value=ozone'

Scenario B: Auto-Scaling Enabled, Single Tenant (Apache Ozone)

Activates intelligent autoscaling for HiveServer2, Metastore, LLAP, and TezAM, while turning on the Auto-Suspend hibernation feature.

helm install hive oci://registry-1.docker.io/ayushtkn/hive-operator \
  --version 4.3.0-SNAPSHOT \
  --set operator.image.repository="ayushtkn/hive" \
  --set operator.image.tag="operator-4.3.0-SNAPSHOT" \
  --set cluster.image="ayushtkn/hive:4.3.0-SNAPSHOT" \
  --set cluster.database.type=postgres \
  --set cluster.database.url="jdbc:postgresql://postgres-postgresql:5432/metastore" \
  --set cluster.database.driver="org.postgresql.Driver" \
  --set cluster.database.username=hive \
  --set cluster.database.passwordSecretRef.name=hive-db-secret \
  --set cluster.database.passwordSecretRef.key=password \
  --set cluster.database.driverJarUrl="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar" \
  --set cluster.zookeeper.quorum="zookeeper:2181" \
  --set cluster.storage.coreSiteOverrides."fs\.defaultFS"="s3a://hive" \
  --set cluster.storage.coreSiteOverrides."fs\.s3a\.endpoint"="http://ozone-s3g-rest:9878" \
  --set-string cluster.storage.coreSiteOverrides."fs\.s3a\.path\.style\.access"=true \
  --set 'cluster.storage.envVars[0].name=HADOOP_OPTIONAL_TOOLS' \
  --set 'cluster.storage.envVars[0].value=hadoop-aws' \
  --set 'cluster.storage.envVars[1].name=AWS_ACCESS_KEY_ID' \
  --set 'cluster.storage.envVars[1].value=ozone' \
  --set 'cluster.storage.envVars[2].name=AWS_SECRET_ACCESS_KEY' \
  --set 'cluster.storage.envVars[2].value=ozone' \
  --set cluster.hiveServer2.autoscaling.enabled=true \
  --set cluster.metastore.autoscaling.enabled=true \
  --set cluster.autoSuspend.enabled=true \
  --set 'cluster.llapClusters[0].name=llap0' \
  --set 'cluster.llapClusters[0].autoscaling.enabled=true' \
  --set 'cluster.llapClusters[0].tezAm.autoscaling.enabled=true'

Scenario C: Multi-Tenant Autoscaling (Apache Ozone)

Our ultimate architecture. Injects server-side routing rules and provisions three fully independent LLAP and TezAM clusters, configured to auto-scale independently.

helm install hive oci://registry-1.docker.io/ayushtkn/hive-operator \
  --version 4.3.0-SNAPSHOT \
  --set operator.image.repository="ayushtkn/hive" \
  --set operator.image.tag="operator-4.3.0-SNAPSHOT" \
  --set cluster.image="ayushtkn/hive:4.3.0-SNAPSHOT" \
  --set cluster.database.type=postgres \
  --set cluster.database.url="jdbc:postgresql://postgres-postgresql:5432/metastore" \
  --set cluster.database.driver="org.postgresql.Driver" \
  --set cluster.database.username=hive \
  --set cluster.database.passwordSecretRef.name=hive-db-secret \
  --set cluster.database.passwordSecretRef.key=password \
  --set cluster.database.driverJarUrl="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar" \
  --set cluster.zookeeper.quorum="zookeeper:2181" \
  --set cluster.storage.coreSiteOverrides."fs\.defaultFS"="s3a://hive" \
  --set cluster.storage.coreSiteOverrides."fs\.s3a\.endpoint"="http://ozone-s3g-rest:9878" \
  --set-string cluster.storage.coreSiteOverrides."fs\.s3a\.path\.style\.access"=true \
  --set 'cluster.storage.envVars[0].name=HADOOP_OPTIONAL_TOOLS' \
  --set 'cluster.storage.envVars[0].value=hadoop-aws' \
  --set 'cluster.storage.envVars[1].name=AWS_ACCESS_KEY_ID' \
  --set 'cluster.storage.envVars[1].value=ozone' \
  --set 'cluster.storage.envVars[2].name=AWS_SECRET_ACCESS_KEY' \
  --set 'cluster.storage.envVars[2].value=ozone' \
  --set cluster.hiveServer2.autoscaling.enabled=true \
  --set cluster.metastore.autoscaling.enabled=true \
  --set cluster.autoSuspend.enabled=true \
  --set-string 'cluster.llapClusterRouting=user:alice=llap0\,user:bob=llap1\,default=llap2' \
  --set 'cluster.llapClusters[0].name=llap0' \
  --set 'cluster.llapClusters[0].autoscaling.enabled=true' \
  --set 'cluster.llapClusters[0].tezAm.autoscaling.enabled=true' \
  --set 'cluster.llapClusters[1].name=llap1' \
  --set 'cluster.llapClusters[1].autoscaling.enabled=true' \
  --set 'cluster.llapClusters[1].tezAm.autoscaling.enabled=true' \
  --set 'cluster.llapClusters[2].name=llap2' \
  --set 'cluster.llapClusters[2].autoscaling.enabled=true' \
  --set 'cluster.llapClusters[2].tezAm.autoscaling.enabled=true'

Scenario D: AWS S3 (Single Tenant)

Connects directly to native AWS S3. Here, we create a Kubernetes Secret for our AWS credentials and securely map them to the operator using valueFrom.secretKeyRef.

# First, create your AWS credentials secret
kubectl create secret generic aws-s3-creds \
  --from-literal=accessKey="<YOUR_ACCESS_KEY>" \
  --from-literal=secretKey="<YOUR_SECRET_KEY>"

helm install hive oci://registry-1.docker.io/ayushtkn/hive-operator \
  --version 4.3.0-SNAPSHOT \
  --set operator.image.repository="ayushtkn/hive" \
  --set operator.image.tag="operator-4.3.0-SNAPSHOT" \
  --set cluster.image="ayushtkn/hive:4.3.0-SNAPSHOT" \
  --set cluster.database.type=postgres \
  --set cluster.database.url="jdbc:postgresql://postgres-postgresql:5432/metastore" \
  --set cluster.database.driver="org.postgresql.Driver" \
  --set cluster.database.username=hive \
  --set cluster.database.passwordSecretRef.name=hive-db-secret \
  --set cluster.database.passwordSecretRef.key=password \
  --set cluster.database.driverJarUrl="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar" \
  --set cluster.zookeeper.quorum="zookeeper:2181" \
  --set cluster.storage.coreSiteOverrides."fs\.defaultFS"="s3a://ayush-k8s-bucket" \
  --set 'cluster.storage.envVars[0].name=HADOOP_OPTIONAL_TOOLS' \
  --set 'cluster.storage.envVars[0].value=hadoop-aws' \
  --set 'cluster.storage.envVars[1].name=AWS_ACCESS_KEY_ID' \
  --set 'cluster.storage.envVars[1].valueFrom.secretKeyRef.name=aws-s3-creds' \
  --set 'cluster.storage.envVars[1].valueFrom.secretKeyRef.key=accessKey' \
  --set 'cluster.storage.envVars[2].name=AWS_SECRET_ACCESS_KEY' \
  --set 'cluster.storage.envVars[2].valueFrom.secretKeyRef.name=aws-s3-creds' \
  --set 'cluster.storage.envVars[2].valueFrom.secretKeyRef.key=secretKey'

Scenario E: Google Cloud Storage (GCS)

Connects directly to GCP storage. We map a JSON service account keyfile as a Secret, instruct the operator to mount it as a volume, and inject the GCS connector JAR.

# First, create your GCS secret from a JSON keyfile
kubectl create secret generic gcs-creds  --from-file=key.json=/path/to/key.json

helm install hive oci://registry-1.docker.io/ayushtkn/hive-operator \
  --version 4.3.0-SNAPSHOT \
  --set operator.image.repository="ayushtkn/hive" \
  --set operator.image.tag="operator-4.3.0-SNAPSHOT" \
  --set cluster.image="ayushtkn/hive:4.3.0-SNAPSHOT" \
  --set cluster.database.type=postgres \
  --set cluster.database.url="jdbc:postgresql://postgres-postgresql:5432/metastore" \
  --set cluster.database.driver="org.postgresql.Driver" \
  --set cluster.database.username=hive \
  --set cluster.database.passwordSecretRef.name=hive-db-secret \
  --set cluster.database.passwordSecretRef.key=password \
  --set cluster.database.driverJarUrl="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.7.5/postgresql-42.7.5.jar" \
  --set cluster.zookeeper.quorum="zookeeper:2181" \
  --set 'cluster.storage.coreSiteOverrides.fs\.defaultFS=gs://ayush-gcs-bucket' \
  --set 'cluster.storage.coreSiteOverrides.fs\.gs\.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem' \
  --set 'cluster.storage.coreSiteOverrides.fs\.gs\.auth\.type=SERVICE_ACCOUNT_JSON_KEYFILE' \
  --set 'cluster.storage.coreSiteOverrides.fs\.gs\.auth\.service\.account\.json\.keyfile=/etc/gcs/key.json' \
  --set-string 'cluster.storage.coreSiteOverrides.fs\.gs\.reported\.permissions=777' \
  --set 'cluster.storage.externalJars[0]=https://repo1.maven.org/maven2/com/google/cloud/bigdataoss/gcs-connector/hadoop3-2.2.25/gcs-connector-hadoop3-2.2.25-shaded.jar' \
  --set 'cluster.storage.volumes[0].name=gcs-key' \
  --set 'cluster.storage.volumes[0].secret.secretName=gcs-creds' \
  --set 'cluster.storage.volumeMounts[0].name=gcs-key' \
  --set 'cluster.storage.volumeMounts[0].mountPath=/etc/gcs' \
  --set 'cluster.storage.volumeMounts[0].readOnly=true'

The AWS S3 & GCS examples are for Single Tenant architecture but they support AutoScaling & Multi Tenancy similar to the Ozone examples and can be configured accordingly

10. The Road Ahead: Tech Preview & Future Work

The Kubernetes operator is officially arriving as part of the upcoming Apache Hive 4.3 release and Apache Tez 1.0.0 as an exciting Tech Preview.

This foundational release shifts Hive from a core on-premises architecture to a deeply integrated, highly dynamic elastic resource provider. But the community isn’t stopping there. Here is a sneak peek at the features slated for upcoming versions of the operator:

• Persistable Sessions in Apache Hive: Session recovery will soon allow execution context to persist across transient networking disconnects or container restarts, insulating long-running analytical jobs from platform-level failures.
• AI-Enabled UI for the Hive Operator: An intelligent dashboard is in development to provide visual tracing for data pool routing, real-world metrics parsing, and AI-driven autoscaler optimization recommendations.

A Note on the Testing Artifacts

The Helm charts (ayushtkn/hive-operator) and container images used in the quickstart examples are part of a personal testing registry compiled directly from the latest master branch source code. They are provided to allow early experimentation prior to the formal Apache releases and aren’t meant for production use case nor are replacement to official Apache Hive releases.