Kubernetes with Data Engineering approach — 5 Authorization and Data Masking with Trino and Apache Ranger
Security at Schema, Table, and Column Level
In big data platforms, data security is crucial for both regulatory compliance and preventing unauthorized access. Trino provides a powerful tool for data analytics with its high-performance distributed query execution capabilities, while Apache Ranger simplifies security management through centralized authorization and auditing mechanisms.
In this study, we explore how to implement schema, table, and column-level access control and data masking using Trino and Apache Ranger. User and group-based authorization is managed, while Hive Metastore is used for data cataloging. MinIO is considered as an object storage solution, and integration with modern data formats like Apache Iceberg facilitates efficient large-scale data management. All these components are orchestrated within a Kubernetes environment to ensure scalability and security.
Before starting this article, you must have a Kubernetes cluster. If you do not have a Kubernetes cluster, proceed to our article titled 1 Create a Kubernetes cluster (with all its dependencies), which is the first article of the series.
In this article, I will use the Kubernetes Cluster that I created with Docker Desktop.
Our Roadmap:
- Installation and configuration of MinIO and Hive Metastore.
- PostgreSQL installation for Secondary Catalog.
- Apache Ranger installation.
- Apache Ranger configuration and Ranger API.
- Trino installation and Configuration.
- Trino and Apache Ranger connection
- Cloudbeaver and Trino connection.
- Creating schemas and tables for Iceberg and PostgreSQL catalogs and adding sample data.
- Apache Ranger Access definitions and Tests.
- Apache Ranger masking and test.
- Tables summaries
- Final: A Secure and Manageable Data Ecosystem.
Installation and configuration of MinIO and Hive Metastore.
In this section, we will install MinIO, then we will provide MinIO information to the Hive Metastore configs and install Hive.
MinIO Installation
MinIO is a high-performance and scalable object storage solution. It provides Amazon S3-compatible API support and is optimized for big data and AI workloads. It is widely used in Kubernetes and cloud environments.
We will set up MinIO using the Helm Chart below.
Run the commands one by one and complete the installation.
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install minio bitnami/minio --version 15.0.7 -n minio --create-namespace
Then learn the MinIO admin information from the Secrets section.
Login to the UI with this information.
If you want, you can perform the operations from the UI. or you can perform the operations using mc (MinIO Client). We will continue using the UI.
Create a bucket named “iceberg” as below.
After creating the Bucket, we will create an Access Key.
This is the Access and Secret Key for this article.
Access Key: iceberg-s3
Secret Key: xoxq9UMdnPH9qO7LUbckldh4CIWEBfN3mWiYr5jv
Now that we have finished the MinIO side, we can move on to the Hive side.
Hive Metastore Installation
We will use the following GitHub repo for the Hive Metastore installation.
https://github.com/AhmetFurkanDEMIR/minio-hive-example
Pull the repo to your local.
git clone https://github.com/AhmetFurkanDEMIR/minio-hive-example.gitEdit the hivems-hadoop-metastore-config.yaml file according to your MinIO address and keys.
<property>
<name>fs.s3a.endpoint</name>
<value>http://10.96.213.145:9000</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<value>iceberg-s3</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>xoxq9UMdnPH9qO7LUbckldh4CIWEBfN3mWiYr5jv</value>
</property>Then let’s create a namespace called hive-metastore.
kubectl create namespace hive-metastoreLet’s apply the yaml files one by one.
bash run.shLet’s complete the Hive Metastore installation with the bash run.sh command.
Thus, we have completed the hive-metastore installation.
PostgreSQL installation for Secondary Catalog
In this step, we will install PostgreSQL, which is our second catalog.
PostgreSQL is an advanced, open-source, object-relational database management system. It is ACID-compliant, highly scalable, and supports extensive SQL features, making it ideal for large-scale data processing and enterprise applications. It also supports JSON, XML, and extensible plugins for modern data management.
We will use the Helm Chart below for PostgreSQL installation.
Run the commands below and complete the installation.
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install postgresql bitnami/postgresql --version 16.5.0 -n postgresql --create-namespace
You can enter PostgreSQL with the following commands.
export POSTGRES_PASSWORD=$(kubectl get secret - namespace postgresql postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)
kubectl run postgresql-client --rm --tty -i --restart='Never' --namespace postgresql --image docker.io/bitnami/postgresql:17.4.0-debian-12-r4 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
--command -- psql --host postgresql -U postgres -d postgres -p 5432
Thus, we have completed the PostgreSQL installation.
Apache Ranger installation
Apache Ranger is an open-source security framework designed to provide centralized access control and auditing for big data environments. It allows administrators to define, enforce, and monitor access policies across various data services like HDFS, Hive, HBase, Kafka, and YARN.
Key Features:
- Centralized Policy Management: Define fine-grained access policies for users and groups.
- Granular Authorization: Supports column-level, row-level filtering, and dynamic data masking.
- Audit and Monitoring: Keeps track of who accessed what data and when.
- External Authentication Support: Integrates with LDAP, Active Directory, and Kerberos.
- REST API Support: Provides APIs for automation and integration.
Apache Ranger is an essential tool for securing data access in enterprise-scale big data platforms.
Create the ranger namespac with the following command.
kubectl create namespace rangerThen apply the following yaml files to your K8S cluster.
apiVersion: v1
kind: ConfigMap
metadata:
name: ranger-db-init-script
namespace: ranger
data:
init_postgres.sh: |
#!/bin/bash
set -e
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
CREATE USER rangeradmin WITH PASSWORD 'rangerR0cks!';
CREATE DATABASE ranger;
GRANT ALL PRIVILEGES ON DATABASE ranger TO rangeradmin;
CREATE USER hive WITH PASSWORD 'rangerR0cks!';
CREATE DATABASE hive;
GRANT ALL PRIVILEGES ON DATABASE hive TO hive;
CREATE ROLE root WITH SUPERUSER LOGIN PASSWORD 'rangerR0cks!';
CREATE DATABASE root OWNER root;
EOSQLapiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ranger-pvc
namespace: ranger
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5GiapiVersion: apps/v1
kind: Deployment
metadata:
name: ranger-db
namespace: ranger
spec:
replicas: 1
selector:
matchLabels:
app: ranger-db
template:
metadata:
labels:
app: ranger-db
spec:
containers:
- name: ranger-db
image: apache/ranger-db:2.5.0
ports:
- containerPort: 5432
env:
- name: RANGER_DB_TYPE
value: "postgres"
volumeMounts:
- name: ranger-data
mountPath: /var/lib/postgresql/data
- name: init-script
mountPath: /docker-entrypoint-initdb.d/init_postgres.sh
subPath: init_postgres.sh
volumes:
- name: ranger-data
persistentVolumeClaim:
claimName: ranger-pvc
- name: init-script
configMap:
name: ranger-db-init-script
---
apiVersion: v1
kind: Service
metadata:
name: ranger-db
namespace: ranger
spec:
selector:
app: ranger-db
ports:
- protocol: TCP
port: 5432
targetPort: 5432
type: ClusterIPapiVersion: apps/v1
kind: Deployment
metadata:
name: ranger-zk
namespace: ranger
spec:
replicas: 1
selector:
matchLabels:
app: ranger-zk
template:
metadata:
labels:
app: ranger-zk
spec:
containers:
- name: ranger-zk
image: apache/ranger-zk:2.5.0
ports:
- containerPort: 2181
---
apiVersion: v1
kind: Service
metadata:
name: ranger-zk
namespace: ranger
spec:
selector:
app: ranger-zk
ports:
- protocol: TCP
port: 2181
targetPort: 2181
type: ClusterIPapiVersion: apps/v1
kind: Deployment
metadata:
name: ranger-solr
namespace: ranger
spec:
replicas: 1
selector:
matchLabels:
app: ranger-solr
template:
metadata:
labels:
app: ranger-solr
spec:
containers:
- name: ranger-solr
image: apache/ranger-solr:2.5.0
ports:
- containerPort: 8983
---
apiVersion: v1
kind: Service
metadata:
name: ranger-solr
namespace: ranger
spec:
selector:
app: ranger-solr
ports:
- protocol: TCP
port: 8983
targetPort: 8983
type: ClusterIPapiVersion: apps/v1
kind: Deployment
metadata:
name: ranger
namespace: ranger
spec:
replicas: 1
selector:
matchLabels:
app: ranger
template:
metadata:
labels:
app: ranger
spec:
containers:
- name: ranger
image: apache/ranger:2.5.0
ports:
- containerPort: 6080
env:
- name: RANGER_DB_TYPE
value: "postgres"
---
apiVersion: v1
kind: Service
metadata:
name: ranger
namespace: ranger
spec:
selector:
app: ranger
ports:
- protocol: TCP
port: 6080
targetPort: 6080
type: NodePortSummary
- All components are running in the ranger namespace.
- Persistent Volume Claim (PVC) was used for PostgreSQL.
- Init script (init_postgres.sh) for PostgreSQL was defined as ConfigMap.
- Zookeeper, Solr and Ranger components were defined with their own services.
Apache Ranger image is pulled from DockerHub and this image has ARM architecture, use another image for AMD64.
admin user: admin
admin password: rangerR0cks!
Apache Ranger configuration and Ranger API
Apache Ranger provides APIs for managing security in a big data environment. These APIs help automate user and group management tasks, allowing centralized control of access policies.
User and Group Access Control APIs:
- User Management: Apache Ranger APIs allow you to create new users, update existing ones, and delete users. These users can be granted access to specific data sources. Access permissions for users to particular resources and operations are defined via Apache Ranger policies.
- Group Management: Groups allow you to bundle users who share similar access rights. Apache Ranger APIs can be used for creating groups, managing group memberships, and assigning access policies to these groups. Each group can have access policies for one or more data sources.
- Access Control: With Apache Ranger APIs, you can manage which users or groups have access to certain resources, and what actions they can perform (e.g., read, write, delete). Access control policies can be fine-grained and highly specific.
How User and Group APIs Work:
- Add User: To add a new user, the
POST /service/xusers/secure/usersendpoint is used. - Add Group: Similarly, groups can be created using the
POST /service/xusers/secure/groups/endpoint. - Access Control: While defining access policies, users and groups are assigned specific permissions to data sources. The APIs allow monitoring and controlling how these policies are enforced.
- Edit and Delete: Modifications to users and groups can be made using
PUTandDELETEmethods.
In short, Apache Ranger’s APIs provide a comprehensive solution for managing user access and group-based permissions in big data environments.
I will create Group and User using APIs, you can do it via UI if you want.
For my example, I will open groups and users as in the tables below.
Groups Create
Let’s create the groups in order.
DE (Data Engineering)
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/groups" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "DE",
"description": "The Data Engineering team, responsible for managing data pipelines and infrastructure.",
"status": 1
}'
DS (Data Scince)
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/groups" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "DS",
"description": "The Data Science team, responsible for analysis and modeling.",
"status": 1
}'
Inters
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/groups" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "Interns",
"description": "Interns, who temporarily assist with various projects.",
"status": 1
}'
Users Create
After the groups, let’s create the users.
Get AHMET FURKAN DEMIR’s stories in your inbox
Join Medium for free to get updates from this writer.
DE Users
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/users" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "elif.arslan",
"firstName": "Elif",
"lastName": "Arslan",
"emailAddress": "elif.arslan@company.com",
"password": "ElifPassword123!",
"description": "Data Engineer at DE team",
"status": 1,
"isVisible": 1,
"userSource": 0,
"userRoleList": ["ROLE_USER"],
"groupIdList": [2]
}'
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/users" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "mustafa.kaya",
"firstName": "Mustafa",
"lastName": "Kaya",
"emailAddress": "mustafa.kaya@company.com",
"password": "MustafaPassword123!",
"description": "Data Engineer at DE team",
"status": 1,
"isVisible": 1,
"userSource": 0,
"userRoleList": ["ROLE_USER"],
"groupIdList": [2]
}'
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/users" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "ahmet.furkan.demir",
"firstName": "Ahmet Furkan",
"lastName": "Demir",
"emailAddress": "ahmet.furkan.demir@company.com",
"password": "AhmetPassword123!",
"description": "Data Engineer at DE team",
"status": 1,
"isVisible": 1,
"userSource": 0,
"userRoleList": ["ROLE_USER"],
"groupIdList": [2]
}'
DS Users
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/users" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "mehmet.yilmaz",
"firstName": "Mehmet",
"lastName": "Yılmaz",
"emailAddress": "mehmet.yilmaz@company.com",
"password": "MehmetPassword123!",
"description": "Data Scientist at DS team",
"status": 1,
"isVisible": 1,
"userSource": 0,
"userRoleList": ["ROLE_USER"],
"groupIdList": [3]
}'
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/users" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "hakan.ozdemir",
"firstName": "Hakan",
"lastName": "Özdemir",
"emailAddress": "hakan.ozdemir@company.com",
"password": "HakanPassword123!",
"description": "Data Scientist at DS team",
"status": 1,
"isVisible": 1,
"userSource": 0,
"userRoleList": ["ROLE_USER"],
"groupIdList": [3]
}'
Interns Users
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/users" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "ali.tuna",
"firstName": "Ali",
"lastName": "Tuna",
"emailAddress": "ali.tuna@company.com",
"password": "AliPassword123!",
"description": "Intern at the company",
"status": 1,
"isVisible": 1,
"userSource": 0,
"userRoleList": ["ROLE_USER"],
"groupIdList": [4]
}'
curl -X POST "http://10.96.89.224:6080/service/xusers/secure/users" \
-u admin:rangerR0cks! \
-H "Content-Type: application/json" \
-d '{
"name": "zeynep.sahin",
"firstName": "Zeynep",
"lastName": "Şahin",
"emailAddress": "zeynep.sahin@company.com",
"password": "ZeynepPassword123!",
"description": "Intern at the company",
"status": 1,
"isVisible": 1,
"userSource": 0,
"userRoleList": ["ROLE_USER"],
"groupIdList": [4]
}'
The Ranger installation and user and group creation steps are complete. After the Trino connection, we will return to set up the policies.
Trino installation
Trino (formerly known as PrestoSQL) is a distributed SQL query engine designed for fast querying of large datasets. It is commonly used for big data analytics, ETL (Extract, Transform, Load) processes, and integrates with data warehouses. Trino’s primary purpose is to query data from various data sources (e.g., HDFS, S3, Kafka, MySQL, PostgreSQL) quickly and efficiently.
Key Features of Trino:
- Distributed Architecture: Trino performs parallel processing on data, making queries faster.
- Multi-Source Support: Trino allows you to query multiple data sources in a single query.
- High Performance: Trino enables high-performance SQL queries, even on large data sets.
- Open Source: Trino is an open-source project developed by the community.
Use Cases:
- Data Warehouses: Trino can query data from multiple data warehouses and combine them for analysis.
- Big Data Analytics: Trino integrates with big data systems, allowing for quick data analysis.
- ETL: Trino can be used in ETL processes to move data from one system to another.
Benefits of Trino:
- Fast Querying: Trino’s parallel execution allows for fast query processing, even on large datasets.
- Flexibility: It supports querying from multiple data sources and combining them in one query.
- Ease of Use: Trino uses SQL as its query language, making it easy for SQL users to get started.
Trino SSL Cert Create
In order to log in to Trino with your username and password, we need to activate the https, or SSL, protocol, and for this we will create a certificate.
Enter your own load balancer IP in the SAN=IP: section.
keytool -genkeypair -alias trino -keyalg RSA -keysize 2048 -keystore trino_cert.jks -validity 365 -storepass abc123 -ext "SAN=dns:trino.trino"
keytool -exportcert -alias trino -keystore trino_cert.jks -file trino.cert -storepass abc123
We will use this certificate file later, now let’s move on to the trino configuration.
Trino Configs
values.yaml file
image:
repository: trinodb/trino
tag: "472"
service:
type: LoadBalancer
loadBalancerIP: 10.96.113.8
port: 8080
ports:
- name: http
port: 8080
targetPort: 8080
server:
workers: 1
config:
https:
enabled: true
port: 8443
keystore:
path: "/etc/trino/ssl/trino_cert.jks"
key: "abc123"
authenticationType: "PASSWORD"
coordinatorExtraConfig: |
password-authenticator.config-files=/etc/trino/password_db.properties
accessControl:
type: properties
properties: |
access-control.name=ranger
ranger.service.name=dev_trino
ranger.plugin.config.resource=/etc/trino/ranger-trino-security.xml,/etc/trino/ranger-trino-audit.xml,/etc/trino/ranger-policymgr-ssl.xml
additionalConfigProperties:
- internal-communication.shared-secret=trino-minio-ranger-example
- internal-communication.https.required=true
- internal-communication.http2.enabled=false
- http-server.https.enabled=true
- http-server.https.keystore.path=/etc/trino/ssl/trino_cert.jks
- http-server.https.keystore.key=abc123
worker:
additionalVolumes:
- name: trino-certs
persistentVolumeClaim:
claimName: trino-certs-pvc
additionalVolumeMounts:
- name: trino-certs
mountPath: /etc/trino/ssl
coordinator:
additionalExposedPorts:
https:
servicePort: 8443
name: https
port: 8443
protocol: TCP
additionalVolumes:
- name: trino-certs
persistentVolumeClaim:
claimName: trino-certs-pvc
additionalVolumeMounts:
- name: trino-certs
mountPath: /etc/trino/ssl
additionalConfigFiles:
password_db.properties: |
password-authenticator.name=file
file.password-file=/etc/trino/password.db
password.db: |
admin:$2y$10$HNvott.IRJ1X6crH2i6R3udsLwUZqSOlb1PveErCuxUlZV.F368SO
mehmet.yilmaz:$2y$10$HNvott.IRJ1X6crH2i6R3udsLwUZqSOlb1PveErCuxUlZV.F368SO
hakan.ozdemir:$2y$10$HNvott.IRJ1X6crH2i6R3udsLwUZqSOlb1PveErCuxUlZV.F368SO
elif.arslan:$2y$10$HNvott.IRJ1X6crH2i6R3udsLwUZqSOlb1PveErCuxUlZV.F368SO
mustafa.kaya:$2y$10$HNvott.IRJ1X6crH2i6R3udsLwUZqSOlb1PveErCuxUlZV.F368SO
ahmet.furkan.demir:$2y$10$HNvott.IRJ1X6crH2i6R3udsLwUZqSOlb1PveErCuxUlZV.F368SO
ali.tuna:$2y$10$HNvott.IRJ1X6crH2i6R3udsLwUZqSOlb1PveErCuxUlZV.F368SO
zeynep.sahin:$2y$10$HNvott.IRJ1X6crH2i6R3udsLwUZqSOlb1PveErCuxUlZV.F368SO
# ranger.plugin.trino.policy.rest.url: http://10.96.89.224:6080
# ranger.plugin.trino.use.rangerGroups: true
# ranger.plugin.trino.super.users: admin
ranger-trino-security.xml: |
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration xmlns:xi="http://www.w3.org/2001/XInclude">
<property>
<name>ranger.plugin.trino.policy.rest.url</name>
<value>http://10.96.89.224:6080</value>
<description>MANDATORY: a comma separated list of URLs to Apache Ranger instances in a deployment</description>
</property>
<property>
<name>ranger.plugin.trino.access.cluster.name</name>
<value></value>
<description>Name to identify the cluster running the Trino instance. This is recorded in audit logs generated by the plugin</description>
</property>
<property>
<name>ranger.plugin.trino.use.rangerGroups</name>
<value>true</value>
<description>Boolean flag to specify whether user-to-groups mapping should be obtained from in Apache Ranger. Default: false</description>
</property>
<property>
<name>ranger.plugin.trino.use.only.rangerGroups</name>
<value>false</value>
<description>Boolean flag. true: use only user-to-groups mapping from Apache Ranger; false: use user-to-groups mappings from Apache Ranger and Trino. Default: false</description>
</property>
<property>
<name>ranger.plugin.trino.super.users</name>
<value>admin</value>
<description>Comma separated list of user names. Superusers will be authorized for all accesses, without requiring explicit policy grants.</description>
</property>
<property>
<name>ranger.plugin.trino.super.groups</name>
<value></value>
<description>Comma separated list of group names. Users in supergroups will be authorized for all accesses, without requiring explicit policy grants</description>
</property>
</configuration>
ranger-trino-audit.xml: |
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration xmlns:xi="http://www.w3.org/2001/XInclude">
<property>
<name>xasecure.audit.is.enabled</name>
<value>true</value>
<description>Boolean flag to specify if the plugin should generate access audit logs. Default: true</description>
</property>
<property>
<name>xasecure.audit.solr.is.enabled</name>
<value>false</value>
<description>Boolean flag to specify if audit logs should be stored in Solr. Default: false</description>
</property>
<property>
<name>xasecure.audit.solr.solr_url</name>
<value></value>
<description>URL to Solr deployment where the plugin should send access audits to</description>
</property>
</configuration>
ranger-policymgr-ssl.xml: |
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration xmlns:xi="http://www.w3.org/2001/XInclude">
<!-- properties used for 2-way SSL between the Trino plugin and Apache Ranger server -->
<property>
<name>xasecure.policymgr.clientssl.keystore</name>
<value></value>
<description>Path to keystore file. Only required for two-way SSL. This property should not be included for one-way SSL</description>
</property>
<property>
<name>xasecure.policymgr.clientssl.keystore.type</name>
<value>jks</value>
<description>Type of keystore. Default: jks</description>
</property>
<property>
<name>xasecure.policymgr.clientssl.keystore.credential.file</name>
<value></value>
<description>Path to credential file for the keystore; the credential should be in alias sslKeyStore. Only required for two-way SSL. This property should not be included for one-way SSL</description>
</property>
<property>
<name>xasecure.policymgr.clientssl.truststore</name>
<value></value>
<description>Path to truststore file</description>
</property>
<property>
<name>xasecure.policymgr.clientssl.truststore.type</name>
<value>jks</value>
<description>Type of truststore. Default: jks</description>
</property>
<property>
<name>xasecure.policymgr.clientssl.truststore.credential.file</name>
<value></value>
<description>Path to credential file for the truststore; the credential should be in alias sslTrustStore</description>
</property>
</configuration>
catalogs:
iceberg: |
connector.name=iceberg
fs.native-s3.enabled=true
hive.metastore.uri=thrift://10.96.112.6:9083
s3.path-style-access=true
s3.endpoint=http://10.96.213.145:9000
s3.aws-access-key=iceberg-s3
s3.aws-secret-key=xoxq9UMdnPH9qO7LUbckldh4CIWEBfN3mWiYr5jv
s3.socket-connect-timeout=10m
s3.socket-read-timeout=10m
s3.region=us-east-1
hive.metastore.thrift.client.connect-timeout=3m
hive.metastore.thrift.client.read-timeout=3m
postgresql: |
connector.name=postgresql
connection-url=jdbc:postgresql://10.96.22.113:5432/postgres
connection-user=postgres
connection-password=2RZxcKZMT7
resourceGroupsConfig: |-
{
"rootGroups": [
{
"name": "global",
"softMemoryLimit": "80%",
"hardConcurrencyLimit": 100,
"maxQueued": 100,
"schedulingPolicy": "fair",
"jmxExport": true
},
{
"name": "admin",
"softMemoryLimit": "10%",
"hardConcurrencyLimit": 20,
"maxQueued": 10
},
{
"name": "ds",
"softMemoryLimit": "20%",
"hardConcurrencyLimit": 30,
"maxQueued": 20
},
{
"name": "de",
"softMemoryLimit": "25%",
"hardConcurrencyLimit": 40,
"maxQueued": 30
},
{
"name": "inters",
"softMemoryLimit": "5%",
"hardConcurrencyLimit": 10,
"maxQueued": 5
}
],
"selectors": [
{
"user": "admin",
"group": "admin"
},
{
"user": "mehmet.yilmaz",
"group": "ds"
},
{
"user": "hakan.ozdemir",
"group": "ds"
},
{
"user": "elif.arslan",
"group": "de"
},
{
"user": "mustafa.kaya",
"group": "de"
},
{
"user": "ahmet.furkan.demir",
"group": "de"
},
{
"user": "ali.tuna",
"group": "inters"
},
{
"user": "zeynep.sahin",
"group": "inters"
}
]
}Password for all users: rangerR0cks!
Let’s create a PVC with the command below, this pvc will be our certificate.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: trino-certs-pvc
namespace: trino
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path
resources:
requests:
storage: 1GiLet’s create a temp pod to copy file (cert) into this PVC.
apiVersion: v1
kind: Pod
metadata:
name: temp-pod
namespace: trino
labels:
app: temp-pod
spec:
containers:
- name: busybox
image: busybox
command: [ "sleep", "3600" ] # Pod'un sürekli çalışması için
volumeMounts:
- mountPath: /mnt/data
name: trino-certs-pvc
volumes:
- name: trino-certs-pvc
persistentVolumeClaim:
claimName: trino-certs-pvc
restartPolicy: NeverFinally, copy the file into the PVC.
kubectl cp trino_cert.jks trino/temp-pod:/mnt/dataNow that the certificate process is complete, we can deploy Trino. To do this, use the following command.
helm upgrade --install trino trino/trino --version 1.37.0 -n trino --create-namespace -f values.yaml
Trino and Apache Ranger connection
Let’s edit the dev_trino service.
Enter the information as in the picture below and take the test.
Cloudbeaver and Trino connection
CloudBeaver is a web-based database management tool. It is open-source and allows you to connect to various databases, execute SQL queries, view, and manage data. It can be considered an extension of DBeaver, but it stands out with its browser-based accessibility.
First, let’s create a namespace called cloudbeaver.
kubectl create namespace cloudbeaverand apply the following yaml file to the cluster.
apiVersion: apps/v1
kind: Deployment
metadata:
name: cloudbeaver
namespace: cloudbeaver
spec:
replicas: 1
selector:
matchLabels:
app: cloudbeaver
template:
metadata:
labels:
app: cloudbeaver
spec:
containers:
- name: cloudbeaver
image: dbeaver/cloudbeaver:25.0.0
ports:
- containerPort: 8978
volumeMounts:
- name: cloudbeaver-workspace
mountPath: /opt/cloudbeaver/workspace
volumes:
- name: cloudbeaver-workspace
persistentVolumeClaim:
claimName: cloudbeaver-pvc
- name: cloudbeaver-config
configMap:
name: cloudbeaver-configmap
---
apiVersion: v1
kind: Service
metadata:
name: cloudbeaver-service
namespace: cloudbeaver
spec:
selector:
app: cloudbeaver
ports:
- protocol: TCP
port: 8978
targetPort: 8978
type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cloudbeaver-pvc
namespace: cloudbeaver
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: local-path
Copy the Trino cert file into the PVC with the following command.
cp trino_cert.jks cloudbeaver/cloudbeaver-69dfb6d8d9-g995q:/opt/cloudbeaver/workspace
Then go to cloudbevaer and complete the installation.
After completing the installation, let’s move on to connecting the Trino.
SSL: true
SSLKeyStorePath: /opt/cloudbeaver/workspace/trino_cert.jks
SSLKeyStorePassword: abc123
Once you have completed everything you should see the PostgreSQL and Iceberg catalogs like below.
Now we can create schemas and tables under these catalogs. We can enter data into the tables.
Creating schemas and tables for Iceberg and PostgreSQL catalogs and adding sample data
Let’s fill in the Iceberg side first, then move on to fill in the PostgreSQL side.
Iceberg Create Schemas
CREATE SCHEMA iceberg.gold
WITH (location = 's3a://iceberg/gold.db/');
CREATE SCHEMA iceberg.stagging
WITH (location = 's3a://iceberg/stagging.db/');
The schemas have been created, now let’s create the tables and insert data.
Iceberg Create Tables and Insert Data (Gold)
CREATE TABLE iceberg.gold.orders (
order_id BIGINT,
customer_id BIGINT,
order_date TIMESTAMP,
total_amount DOUBLE,
status VARCHAR,
processed_at TIMESTAMP
)
WITH (
format = 'PARQUET',
partitioning = ARRAY['order_date'],
sorted_by = ARRAY['total_amount'],
location = 's3a://iceberg/gold.db/orders/'
);
CREATE TABLE iceberg.gold.customers (
customer_id BIGINT,
name VARCHAR,
email VARCHAR,
created_at TIMESTAMP,
status VARCHAR,
processed_at TIMESTAMP
)
WITH (
format = 'PARQUET',
partitioning = ARRAY['created_at'],
sorted_by = ARRAY['customer_id'],
location = 's3a://iceberg/gold.db/customers/'
);
INSERT INTO iceberg.gold.orders VALUES
(1, 101, TIMESTAMP '2025-03-17 12:00:00', 250.75, 'Completed', TIMESTAMP '2025-03-17 12:45:00'),
(2, 102, TIMESTAMP '2025-03-18 14:15:00', 150.50, 'Processing', TIMESTAMP '2025-03-18 14:50:00'),
(3, 103, TIMESTAMP '2025-03-19 16:30:00', 300.00, 'Completed', TIMESTAMP '2025-03-19 17:00:00');
INSERT INTO iceberg.gold.customers VALUES
(101, 'John Doe', 'john.doe@example.com', TIMESTAMP '2023-01-01 10:00:00', 'Active', TIMESTAMP '2023-01-01 10:05:00'),
(102, 'Jane Smith', 'jane.smith@example.com', TIMESTAMP '2023-02-15 11:30:00', 'Inactive', TIMESTAMP '2023-02-15 11:35:00'),
(103, 'Robert Brown', 'robert.brown@example.com', TIMESTAMP '2023-03-01 09:45:00', 'Active', TIMESTAMP '2023-03-01 09:50:00');
Iceberg Create Tables and Insert Data (Staging)
CREATE TABLE iceberg.stagging.orders (
order_id BIGINT,
total_amount DOUBLE,
order_date TIMESTAMP
)
WITH (
format = 'PARQUET',
partitioning = ARRAY['order_date'],
sorted_by = ARRAY['total_amount'],
location = 's3a://iceberg/stagging.db/orders/'
);
CREATE TABLE iceberg.stagging.customers (
customer_id BIGINT,
name VARCHAR,
status VARCHAR
)
WITH (
format = 'PARQUET',
partitioning = ARRAY['status'],
sorted_by = ARRAY['customer_id'],
location = 's3a://iceberg/stagging.db/customers/'
);
INSERT INTO iceberg.stagging.orders VALUES
(1, 250.75, TIMESTAMP '2025-03-17 12:00:00'),
(2, 150.50, TIMESTAMP '2025-03-18 14:15:00'),
(3, 300.00, TIMESTAMP '2025-03-19 16:30:00');
INSERT INTO iceberg.stagging.customers VALUES
(101, 'John Doe', 'Active'),
(102, 'Jane Smith', 'Inactive'),
(103, 'Robert Brown', 'Active');
PostgreSQL Create Table and Insert Data
CREATE TABLE postgresql.public.products (
id INTEGER,
product_name VARCHAR,
category VARCHAR,
price DOUBLE PRECISION
);
INSERT INTO postgresql.public.products (id, product_name, category, price)
VALUES
(1, 'Product A', 'Electronics', 199.99),
(2, 'Product B', 'Clothing', 49.99),
(3, 'Product C', 'Books', 15.99);
And so, the tables under all our Catalogs were created on the Trino side and sample data was entered into them. Now let’s look at how we can manage group-based access control of these tables on the Apache Ranger side.
Apache Ranger Access definitions and Tests
In this step, we will do the group-based table restriction and data masking parts from the Ranger UI.
We need to go to the policies and add all the groups to all the marked places, this is so that users can see the catalogs and run queries.
Add as in the example above.
Open the policy named all — catalog, schema, table, column and make the following edit. Thanks to this edit, all other catalogs will be closed but the system catalog will remain open.
Finally, it should look like the one below.
Now we can move on to Group-based access settings.
DE (Data Engineering) Group
Access permissions
PostgreSQL: Full access (Select, Insert, Create)
Iceberg: Full access (Select, Insert, Create)
DS (Data Science) Group
Access permissions
PostgreSQL: No access
Iceberg: Gold and Stagging access (Select)
Interns Group
Access permissions
PostgreSQL: No access
Iceberg: iceberg.stagging.orders access (Select)
We have completed all the access permissions for the groups, now let’s look at the masking part.
Apache Ranger masking and test
You can use the masking section for data within the scope of KVVK and hide sensitive information such as users’ email and phone numbers.
Open the policy for Masking and make your configurations as in the picture above.
The image below shows how the email data looks like for the admin user.
Another image below shows what the email data looks like for the DE group.
In this step, we talked about how we mask important data.
Tables summaries
Final: A Secure and Manageable Data Ecosystem
By following this roadmap, we successfully set up MinIO, Hive Metastore, PostgreSQL, Apache Ranger, Trino, and CloudBeaver, creating a secure, scalable, and manageable data ecosystem.
Key Achievements:
✅ Secure Access Management: Implemented Apache Ranger to enforce centralized authorization policies and protect sensitive data.
✅ Data Masking: Configured Apache Ranger’s masking features to control data visibility based on user roles.
✅ Flexible and Powerful Querying: Enabled fast and scalable queries on PostgreSQL and Iceberg tables using Trino.
✅ User-Friendly Interface: Integrated CloudBeaver to provide an intuitive SQL interface for data analysts and engineers.
With this setup, we have established a modern data architecture that prioritizes security and efficiency. While Apache Ranger ensures robust access control, Trino enables high-performance querying across multiple data sources.
The next step is to expand this infrastructure by incorporating advanced analytics and automation processes. 🚀
