Treating Openflow Like Code: A Git-Backed Workflow for Process Groups
If you have worked with any GUI-driven pipeline tool, you have probably lived through some version of this: you build a pipeline visually, and when it is time to move it to another environment, you export it as a file, hand that file to someone else, and hope it imports cleanly on the other side. Maybe you rename it something like pipeline_final_v3_actually_final.xml and drop it in a shared folder, and hope nobody opens it before you do.
Openflow is Snowflake’s managed service built on Apache NiFi, where a pipeline goes by the name Process Group, a term you will see throughout this post. But the underlying problem here is not specific to NiFi or Openflow. It is the export-and-reimport pattern itself.
That pattern has three problems that never really go away, regardless of which tool you are exporting from. The exported file does not capture runtime state, so anything stateful comes back cold on import. There is no diff. You cannot look at two exported files and see what actually changed between them, so review means eyeballing a canvas and hoping you notice the one property someone edited. And the whole thing is manual. A human exports, a human renames, a human imports, a human promotes. Every one of those steps is a place to fumble a change between dev and prod.
Openflow does not solve that by giving you a better export button. It solves it by giving process groups an actual relationship with Git: branches, commits, diffs, pull requests, the works. This month is about setting that up end to end, from a bare GitHub repo to a real dev-to-prod promotion with parameters handled properly.
What We’re Building
- A GitHub-backed Registry Client wired into Controller Settings, so process groups can start version control against a real repo
- A small but real transform flow, version-controlled from the canvas, with a look at what actually lands in Git as JSON
- A feature-branch workflow: change on canvas, commit to a branch, open a PR, review the diff, merge
- A promotion path where a separate prod runtime pulls the merged flow from
main - Parent and child Parameter Contexts that share common values but override per environment
Mindset Shift: The Process Group Is the Unit of Change, Not the Export
The instinct coming from template-based NiFi is to think in terms of files you move around. Export this, import that, keep a folder of versions somewhere safe.
With Git-backed process groups, the process group itself is the thing under version control. It has a working state on the canvas and a committed state in Git, the same way a working directory relates to a Git repo. You do not export when you want a snapshot. You commit. You do not compare two exported files by hand. You open a PR and read the diff. The unit of change is the process group’s configuration, not an artifact you generate from it.
This matters most when you get to promotion. Moving a change to prod is not “import this file into a different environment.” It is “point the prod process group at a newer commit,” which is a much smaller and much safer operation.
Architecture Overview
Dev work happens on a feature branch inside the Registry Client’s GitHub repo. A PR merges that branch into main. A prod process group, versioned against the same repo, updates to pull the latest commit from main when you are ready to promote.
The dev canvas and the prod canvas are two different process groups pointed at the same bucket in the same registry. Neither one owns the flow definition. Git does. What differs between them is which commit they are currently synced to, and which Parameter Context resolves the environment-specific values at runtime.
Prerequisites
- A GitHub repository to act as flow storage, plus a Personal Access Token with
reposcope for Openflow to authenticate against it - Openflow
ACCOUNTADMIN-equivalent access to Controller Settings, to add a Registry Client - A dev and a prod Openflow runtime (or two environments you treat as such) with network access to GitHub
If your organization runs on Azure DevOps or GitLab instead of GitHub, Openflow also ships an Azure DevOps/ GitLab Registry Client that works the same way, repo and PAT included. Everything past the Registry Client setup in this post applies the same way regardless of which one you pick.
Setup
Create the GitHub repo and PAT
Create an empty repository to hold flow definitions. It does not need a README or a .gitignore, Openflow will populate it. Generate a PAT scoped to repo access on that repository and store it somewhere your Secrets Manager or Openflow can read it from securely. Do not paste it directly into a processor property.
Add a GitHub Registry Client
In Controller Settings, go to Registry Clients and add a new client.
The Repository Branch here is the default branch Openflow reads from when you first browse the registry. It is not a lock. Individual process groups can and will version against feature branches once you start making changes.
Starting Version Control on a Process Group
I built a small but real transform flow rather than a GenerateFlowFile toy, since a two-processor loop does not tell you anything useful about what actually lands in Git. Here is how to reproduce it:
- Create a new process group on the dev canvas and name it
json-transform-router. - Drag in a
GenerateFlowFileprocessor as the source. Set Custom Text to a small order payload,{ "id": "ORD-1042", "email": "jane@example.com" }, and add arecord.typeattribute to it (via the Insert Header / dynamic property fields on the processor) so downstream routing has something to key off. - Drag in a
JoltTransformJSONprocessor. Set Jolt Transform tochainand Jolt Specification to the shift spec below, which renames the two fields into the shape downstream systems expect. - Drag in a
RouteOnAttributeprocessor. Add a dynamic property,record.type, with the value${record.type:equals('order')}, so FlowFiles carrying an order record route down a separate relationship from everything else. - Wire the processors together:
GenerateFlowFile'ssuccessrelationship connects intoJoltTransformJSON, whosesuccessrelationship connects intoRouteOnAttribute. Terminate or route the unmatched relationship onRouteOnAttributeso the canvas has no dangling relationships.
Three processors, one JOLT spec, one routing decision. Small enough to read in a screenshot, real enough that the diff later in this post means something.
[
{
"operation": "shift",
"spec": {
"id": "order_id",
"email": "customer_email"
}
}
]A FlowFile shaped like { "id": "ORD-1042", "email": "jane@example.com" } comes out the other side as { "order_id": "ORD-1042", "customer_email": "jane@example.com" }. Nothing nested, nothing clever, just enough of a rename to prove the transform did something when you look at the diff later.
With the process group built on the dev canvas, right-click it and choose Version → Start Version Control.
Once it commits, the process group’s canvas header shows a version badge instead of the usual “unversioned” state. This is the part most tutorials skip. Here is what actually lands in the repo:
{
"flowContents": {
"processGroups": [],
"processors": [
{
"identifier": "5e2c...",
"name": "JoltTransformJSON",
"type": "org.apache.nifi.processors.jolt.JoltTransformJSON",
"properties": {
"Jolt Transform": "chain",
"Jolt Specification": "[ { \"operation\": \"shift\", \"spec\": { \"id\": \"order_id\", \"email\": \"customer_email\" } } ]"
},
"position": { "x": 320.0, "y": 128.0 }
},
{
"identifier": "8a91...",
"name": "RouteOnAttribute",
"type": "org.apache.nifi.processors.standard.RouteOnAttribute",
"properties": {
"record.type": "${record.type:equals('order')}"
}
}
],
"connections": [ /* processor-to-processor wiring, by identifier */ ]
},
"parameterContexts": {},
"flowEncodingVersion": "1.0"
}Processor properties, canvas positions, and connection wiring all serialize into that JSON. Processor identifiers are stable within a registry, which is what makes diffs meaningful. Change a JOLT spec and the diff shows exactly that string changing, not a wall of noise from every processor shifting position by a pixel.
The Dev → Review → Prod Loop
Feature branches for changes made on canvas
For the next change, I didn’t need to touch GitHub at all first. Right-click the versioned process group and choose Version → Create Branch, which pops a dialog showing the current branch (main) and a field for the new branch name, feature/add-error-routing. Openflow creates the branch in GitHub and switches the process group to version against it in one step. Every commit from the canvas after that lands on the feature branch, not on main directly.
Get DW Conradie’s stories in your inbox
Join Medium for free to get updates from this writer.
Openflow will not let you commit straight to a protected branch if you have branch protection turned on in GitHub, which is worth enabling for main on this repo the same way you would for application code.
With the branch created, I made the actual change on canvas: a LogAttribute processor wired to RouteOnAttribute's unmatched relationship, so rows that fail the record.type check get logged instead of silently dropping off the canvas. The process group's status bar picks up the change immediately, showing a count of local modifications not yet in Git.
To commit, right-click the process group again and choose Version → Commit local changes. That opens a dialog to write a commit message, same as the initial version control commit, and pushes the result straight to feature/add-error-routing.
Opening a PR and reviewing the diff
With the branch pushed, open a pull request in GitHub as usual. This is where the “no diff” problem from template exports actually gets solved, but it needs one piece of setup first: drop Snowflake’s Flow Diff GitHub Action into .github/workflows/flowdiff.yml in the flow storage repo. Once that workflow exists, every PR against the repo gets a visual, human-readable flow diff rendered directly into the PR conversation, added processors, changed properties, and rewired connections shown as a real diagram instead of a raw JSON blob.
A reviewer who has never opened the canvas can look at that diff and understand exactly what changed: one new RouteOnAttribute relationship, one new LogAttribute processor wired to it, no changes to the existing transform logic. That is the review that template exports never gave you.
Merge to main
Approve, merge, done. main now has the failure-routing change committed.
Promoting to Prod
The prod environment has its own process group, versioned against the same bucket and flow name, currently synced to the commit that predates the failure-routing change. Promotion is not a redeploy. It is telling that process group to change its version.
Right-click the prod process group and choose Version → Change Version, then pick the latest commit on main.
Openflow diffs the currently running configuration against the target commit and applies the difference. It does not tear down and rebuild the whole process group. Only what changed on canvas changes in prod.
Parameter Contexts Across Environments
The flow itself should be identical between dev and prod. What differs is configuration: which database to hit, which bucket to read from, what the retry count should be. That is exactly what Parameter Contexts are for, and Openflow supports inheritance between them.
Each Parameter Context maps one-to-one with a process group, which is why the inheritance model matters here. You are not sharing one giant context across every flow. You are giving each process group its own context and letting the parent-child relationship carry the values that should stay identical across environments.
I set up a parent context, shared-pipeline-params, holding values that are the same everywhere: retry counts, batch sizes, timezone settings. Each environment gets its own child context that inherits from the parent and overrides only what actually differs.
The dev environment gets its own child context, dev-params, inheriting the same retry.count and batch.size, overriding target.database to DEV_DB and target.warehouse to a smaller warehouse. The process group references the child context appropriate to its environment. Nobody has to remember to keep the shared values in sync by hand, since they only exist in one place.
Limitations and Caveats
- Flow IDs are not portable across registries. A process group versioned against one Registry Client cannot simply be repointed at a different registry and expect history to carry over. If you are migrating registries, plan on a fresh
Start Version Controland treat history in the old registry as an archive, not something you drag forward. - Rolling back does not restore processor state. Changing version to an older commit restores the configuration, not in-flight state like a stateful processor’s last watermark or cursor. Know what state your processors hold before you roll back, and be ready to reset or reseed it.
- Parameter changes are not versioned the same way flow structure is. A recent release note flagged this explicitly: editing a Parameter Context’s values does not itself generate a flow commit, only structural changes to the process group do. It is easy to assume a parameter edit is tracked the same way a processor change is. It is not. Treat parameter changes as their own change management concern, not something Git-backed versioning covers for free.
- Branch protection is your responsibility, not Openflow’s. Nothing stops someone from versioning a process group straight against
mainif the repo does not enforce protection rules. Set them up the same way you would for any other repo you care about.
Wrapping Up
Export as template, re-import is gone from this workflow entirely. What replaced it is a process group with a real relationship to Git: branches for in-progress work, diffs a reviewer can actually read, a PR before anything reaches prod, and a promotion step that is a version change instead of a file shuffle. Parameter Contexts and a Secrets Manager handle the part that version control alone never could, letting the exact same flow definition run correctly in two environments without carrying a single secret in the JSON.
Checklist for setting this up end to end:
- [ ] Create a GitHub repo for flow storage and a PAT scoped to it
- [ ] Add a GitHub (or Azure DevOps) Registry Client in Controller Settings
- [ ] Start version control on a process group, confirm the JSON lands in the repo as expected
- [ ] Branch, edit on canvas, commit to the branch, open a PR, review with Flow Diff, merge
- [ ] Point the prod process group at the merged commit via Change Version
- [ ] Set up a parent Parameter Context for shared values and child contexts per environment
- [ ] Enable branch protection on
mainin the flow storage repo
Reference: Version control for custom flows in Openflow — the official Snowflake documentation for Registry Clients, the Flow Diff GitHub Action, and Parameter Context inheritance covered in this post.
Next steps:
- Wire sensitive parameters through a Secrets Manager (AWS Secrets Manager, Azure Key Vault, or similar) instead of storing them as literal values in a Parameter Context, so credentials never touch the JSON that gets committed to Git
- Now that flow structure and environment config both promote cleanly, look at what a CI check on that PR could actually validate before a human ever opens the diff
This is part of an ongoing series on building data pipelines with Snowflake Openflow.
Next Steps
- Learn more on docs.snowflake.com
- Sign up for a Snowflake Trial
If you found this useful, follow me on LinkedIn for more Data Engineering and Snowflake AI Data Cloud use cases.
