Setting Up the DataHub Development Environment for Connector Development

Jishanahmed AR Shaikh (JARS)

9 min read

·

Sep 21, 2026

--

Listen

Share

Press enter or click to view image in full size

DataHub is an open-source context platform for AI agents, bringing together metadata, lineage, data quality, and real-time context so organizations can build trusted data and AI systems.

If you’re planning to build a DataHub connector, the first step is getting a clean DataHub development environment running.

This guide walks through setting up a dedicated development machine, building the DataHub repository, preparing the metadata-ingestion development environment, starting a local DataHub instance, and verifying that everything is ready for connector development.

Prerequisites

Operating System

DataHub connector development requires a Unix-based development environment such as:

• Linux
• macOS

Native Windows development is not officially supported. If your primary workstation runs Windows, you can use a Linux environment through WSL2, a virtual machine (Not Recommended)

or a remote Linux machine such as an AWS EC2 instance. (Recommended)

If you already have a Linux or macOS development machine, you can use it directly, you do not need an EC2 instance unless you want to burn 💰

Since I am natively on windows, For this guide, we will use an AWS EC2 Linux instance as the dedicated development machine. This allows the development environment to remain isolated from the local workstation while still being accessible remotely through SSH.

DataHub Repository

You should have your own fork of the DataHub repository.

For example:

datahub-project/datahub
        ↓
your-account/datahub

If you already have an existing personal fork that you want to keep for other work, create a separate organization or repository for connector development.

For this setup, I created:

jars-demo/datahub-demo

ref. link (https://github.com/jars-demo/datahub-demo)

from the upstream:

datahub-project/datahub

1. Create the Development Machine

You can use any Linux machine with enough resources to build DataHub and run its development stack.

For this setup, an AWS EC2 instance was used. (t3.xlarge)

Example:

OS:       Ubuntu 24.04 LTS
CPU:      4 vCPU
RAM:      16 GB
Storage:  100 GB gp3

The exact machine is not important as long as it has enough resources for the DataHub build and local services.

2. Connect to the Machine

From your workstation, connect to the Linux machine using SSH.

For example:

ssh ubuntu@<your-server-ip>

You can work from any editor or terminal that supports SSH/remote development:

• Kiro by AWS
• VS Code
• Cursor
• Antigravity
• terminal
• other SSH-compatible development environments

The important part is that the DataHub repository, build tools, Python environment, and DataHub services are running on the Linux machine.

3. Verify the Base Environment

Once connected:

uname -a

Check the available resources:

nproc
free -h
df -h

Check the main tools:

git --version
docker --version
docker compose version

You should also verify Java and Python:

java -version
python3 --version

4. Clone Your DataHub Fork

Move to your home directory:

cd ~

Clone your fork:

git clone https://github.com/jars-demo/datahub.git

Then:

cd ~/datahub

Verify:

git remote -v

You should have your fork configured as origin.

It is also useful to add the official DataHub repository as upstream:

git remote add upstream https://github.com/datahub-project/datahub.git

Verify:

git remote -v

You should now have:

origin    → your fork
upstream  → datahub-project/datahub

5. Install the Repository Toolchain

DataHub uses repository-managed tooling.

Install mise:

curl https://mise.run | sh

Add it to your shell:

echo 'eval "$(~/.local/bin/mise activate bash)"' >> ~/.bashrc

Reload the shell:

exec bash

Verify:

mise --version

From the DataHub repository:

cd ~/datahub

Trust the repository configuration:

mise trust

Install the required toolchain:

mise install

Verify:

mise ls --local

This allows the DataHub repository to provide the versions of its required development tools instead of relying on arbitrary system-wide versions.

Bonus: Run this in the Shell before moving ahead

echo "=== Python ==="
which python3
python3 --version

echo "=== Java ==="
which java
java -version

echo "=== Node ==="
node --version

echo "=== Yarn ==="
yarn --version

Expected output:

=== Python ===
/home/ubuntu/.local/share/mise/installs/python/3.11/bin/python3
Python 3.11.16
=== Java ===
/home/ubuntu/.local/share/mise/installs/java/25/bin/java
openjdk version "25.0.2" 2026-01-20
OpenJDK Runtime Environment (build 25.0.2+10-69)
OpenJDK 64-Bit Server VM (build 25.0.2+10-69, mixed mode, sharing)
=== Node ===
v22.23.2
=== Yarn ===
1.22.22

6. Build the DataHub Development Environment

Move into metadata-ingestion:

cd ~/datahub/metadata-ingestion

Build the development environment:

../gradlew :metadata-ingestion:installDev

This prepares the Python ingestion development environment and builds the required DataHub components.

The first build can take some time because the repository needs to download dependencies and build its required components.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

It might not complete the build successfully in the first run itself, it might fail with some dependency issues

Some Key Failures and Commands to solve them are as follows

1. Packaging / Dependencies Issue : “kerberos==1.3.1 is being compiled from source, and the current machine doesn’t have the native build/Kerberos development tools installed.

• Key Commands to solve the `kerberos` Issue

sudo apt update
sudo apt install -y build-essential libkrb5-dev

2. Python LDAP Dependency Issue: `python-ldap` (v3.4.5) was included because `acryl-datahub[dev]` (v1!0.0.0.dev0) depends on `python-ldap` This error likely indicates that you need to install a library that provides “lber.h” for `python-ldap@3.4.5`

• Key Commands to solve this Python LDAP Dependency Issue

sudo apt install -y libldap2-dev

ls -l /usr/include/lber.h

3. sasl/sasl.h → missing

• Key Commands to solve this Missing sasl/sasl.h issue

sudo apt install -y libsasl2-dev

ls -l /usr/include/sasl/sasl.h

After resolving these dependencies, rerun the build:

./gradlew :metadata-ingestion:installDev

If everything is installed correctly, you should now see:

BUILD SUCCESSFUL

7. Activate the Metadata Ingestion Environment

Once installDev completes:

source venv/bin/activate

Verify that the environment is active:

which python

You should see the Python executable coming from:

~/datahub/metadata-ingestion/venv/

Now verify the DataHub CLI:

datahub version

A development installation may report:

DataHub CLI version: unavailable (installed in develop mode)

That is expected for the development installation.

The important part is that the datahub command is available.

8. Verify the Ingestion Environment

Run:

python --version

Then:

pip --version

And:

datahub --help

At this point, the metadata-ingestion environment is ready.

9. Run a Baseline Validation

Before developing a connector, verify that the DataHub repository itself is healthy.

Run lint:

../gradlew :metadata-ingestion:lint

You can automatically fix applicable issues with:

../gradlew :metadata-ingestion:lintFix

Run the fast test suite:

../gradlew :metadata-ingestion:testQuick

And verify dependency lock files:

../gradlew :metadata-ingestion:checkLockFile

The purpose of this step is simple:

Make sure the development environment works before introducing connector-specific code.

10. Start the DataHub Development Stack

Move back to the repository root:

cd ~/datahub

Starting the local DataHub environment

DataHub provides a development helper workflow through scripts/dev/datahub-dev.sh setup followed by
scripts/dev/datahub-dev.sh start

For this connector-development environment, we use the Gradle quickstartDebug workflow instead:

./gradlew quickstartDebug

This brings up the local debug/quickstart DataHub stack, including GMS, the frontend, MySQL, Kafka, and OpenSearch.

Note:

Why quickstartDebug here?
The goal of this environment is to have a reproducible local DataHub instance against which the Python metadata-ingestion connector can be developed and tested. quickstartDebug was the workflow validated for this setup.

Press enter or click to view image in full size

Build Successful ✅

Check the running containers:

docker ps

You should see the DataHub development services running.

Press enter or click to view image in full size

12. Access DataHub From Your Local Machine

If you are connected to the EC2 instance through SSH, forward the DataHub ports to your local machine.

For example:

Local machine
    │
    │ localhost:9002
    ▼
SSH tunnel
    │
    ▼
EC2
    │
    └── DataHub :9002

Similarly, forward GMS:

Local machine
    │
    │ localhost:8080
    ▼
SSH tunnel
    │
    ▼
EC2
    │
    └── DataHub GMS :8080

With SSH port forwarding configured, you can open the DataHub UI from your normal local browser:

http://localhost:9002

You do not need to expose the DataHub UI publicly on the EC2 instance.

13. Verify GMS From Both Machines

DataHub GMS exposes the REST API used by DataHub clients and ingestion.

From the EC2 machine, test:

curl http://localhost:8080

You may receive: (If you don’t receive any such warning, still follow the guide to generate a $TOKEN which is eventually required)

{"timestamp":...,"status":401,"error":"Unauthorized","path":"/"}

This is expected when authentication is enabled. The 401 Unauthorized response confirms that the GMS service is reachable, but the request requires authentication.

You can also inspect the response headers using:

curl -i http://localhost:8080

You should see:

HTTP/1.1 401 Unauthorized

Initialize DataHub Authentication

The datahub CLI is installed inside the metadata-ingestion virtual environment, so activate the environment before running DataHub CLI commands.

cd ~/datahub/metadata-ingestion
source venv/bin/activate

You should now see the virtual environment in your shell prompt:

(venv) ubuntu@...:~/datahub/metadata-ingestion$

Initialize the DataHub CLI configuration and generate a GMS access token:

datahub init \
  --host http://localhost:8080 \
  --username datahub \
  --password datahub

This generates a token and stores the DataHub configuration in:

/home/ubuntu/.datahubenv

You should see output similar to:

✓ Generated token
✓ Configuration written to /home/ubuntu/.datahubenv

Load the generated token into an environment variable:

TOKEN=$(python -c 'import yaml; print(yaml.safe_load(open("/home/ubuntu/.datahubenv"))["gms"]["token"])')

Export it as DATAHUB_TOKEN so it can be used by the ingestion recipe later:

export DATAHUB_TOKEN="$TOKEN"

To verify that the token is accepted by GMS, make an authenticated request:

curl -i \
  -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/aspects

The /aspects endpoint does not support a GET request, so an authenticated request may return:

HTTP/1.1 400 Bad Request

with a message similar to:

GET operation not supported for URI: '/aspects'

This is expected. The important difference is that the request is no longer rejected with 401 Unauthorized, confirming that the bearer token is being accepted by GMS.

Note: Keep the DataHub token out of source control. The token should be supplied through an environment variable or another appropriate secret-management mechanism when configuring the ingestion pipeline.

Now, run the command given below in the EC2 Shell

echo $TOKEN

Copy the long string similar to eyJh........gshr

Then, from your local workstation:

curl -i -H "Authorization: Bearer YOUR_TOKEN" http://localhost:8080/aspects

Paste the copied token into YOUR_TOKEN place

Because the SSH tunnel forwards the port, both commands should reach the same DataHub GMS instance.

This is a useful sanity check that the SSH forwarding is working correctly.

14. Verify the DataHub UI

Open your local browser:

http://localhost:9002

You can log in with the demo credentials

username: datahub
password: datahub

You should see the DataHub interface.

Press enter or click to view image in full size

At this point you have verified both sides:

EC2
 │
 ├── metadata-ingestion venv
 │       └── datahub CLI
 │
 ├── DataHub GMS
 │       └── localhost:8080
 │
 └── DataHub UI
         └── localhost:9002

And from your workstation:

Browser
   │
   └── localhost:9002
           │
           └── SSH tunnel
                   │
                   └── EC2 DataHub

15. Final Development Environment Check

Before starting connector development, verify all of the following.

Repository

cd ~/datahub
git status
git remote -v

Toolchain

mise --version
mise ls --local
java -version
python3 --version

Metadata ingestion

cd ~/datahub/metadata-ingestion
source venv/bin/activate

which python
datahub version

DataHub

"curl http://localhost:8080" or "curl -i -H "Authorization: Bearer YOUR_TOKEN" http://localhost:8080/aspects"

Containers

docker ps

Local browser

Open:

http://localhost:9002

Press enter or click to view image in full size

16. You’re Ready to Build a Connector

The development environment should now look like this:

                     Your Workstation
               ┌─────────────────────────┐
               │ VS Code / Cursor / Kiro │
               │ Terminal / Browser      │
               └────────────┬────────────┘
                            │
                       SSH / Tunnel
                            │
                            ▼
               ┌─────────────────────────┐
               │      EC2 / Linux        │
               │                         │
               │  ~/datahub              │
               │       │                 │
               │       └── metadata-     │
               │           ingestion     │
               │              │          │
               │           venv          │
               │              │          │
               │        datahub CLI      │
               │                         │
               │  DataHub Development    │
               │       Stack             │
               │       │                 │
               │       ├── GMS :8080     │
               │       └── UI  :9002     │
               └─────────────────────────┘

You now have:

• a DataHub fork ready for development
• the required repository toolchain
• a working metadata-ingestion development environment
• the DataHub CLI available inside the ingestion environment
• a running DataHub development stack
• GMS accessible on port 8080
• DataHub UI accessible through SSH port forwarding
• a Linux environment ready for connector development

The next step is no longer environment setup.

It is building the connector.

Continued ⬇️

Building a Production Ready Datahub Connector in Under 30 Minutes