Process Functions and UDFs in PyFlink — Unlocking Flink’s Real Power

A practical guide to user-defined functions and process functions that help you build real production-grade streaming logic.

Niteesh K R

4 min read

·

Jun 29, 2026

--

Listen

Share

If our previous article taught us how to shape streams using transformations, this one is about something even more powerful:

Writing your own logic.

Built-in transformations like map(), flat_map(), and filter() are fantastic.

But let’s be honest.

Sooner or later, every Flink developer reaches that point where they think:

“This is nice… but what if I want to do something custom?”

Exactly.

That is where User-Defined Functions (UDFs) and Process Functions enter the picture.

And trust me — this is where Flink starts feeling less like a framework and more like a superpower.

Press enter or click to view image in full size

Why Do We Even Need UDFs?

Let’s say you’re processing a stream of e-commerce orders.

Each event looks like this:

{
    "user": "Tony",
    "amount": 2500,
    "city": "Bengaluru"
}

Now suppose you want to:

• Assign a risk score
• Categorise orders into Gold/Silver/Bronze
• Detect suspicious purchases
• Trigger alerts for large transactions

Can plain map() do this?

Technically, yes.

Should you write everything as inline lambdas?

Absolutely not.

At some point, your lambdas start looking like this:

stream.map(lambda x: some_insane_business_logic_with_20_conditions(x))

That’s not elegant. That’s chaos.

UDFs solve this problem.

What Are UDFs?

A User-Defined Function (UDF) is simply a custom function or class that tells Flink:

“Here’s my logic. Run this for every relevant record.”

Think of Flink as the engine.

Think of UDFs as the rules you plug into that engine.

Simple.

The Common UDFs in PyFlink

We’ll focus on five important ones:

• MapFunction
• FlatMapFunction
• ReduceFunction
• ProcessFunction
• Plain Python lambda / callable UDFs

Notice what’s missing?

KeyedProcessFunction.

Yes, intentionally.

That one deserves its own article because once state enters the chat, things get serious.

MapFunction — When map() Grows Up

You’ve already seen:

stream.map(lambda x: x.upper())

Nice for simple cases.

But what if transformation logic becomes large?

Instead of cramming logic inside lambdas, create a proper function.

from pyflink.datastream.functions import MapFunction

class RiskClassifier(MapFunction):
    def map(self, value):
        if value > 10000:
            return "HIGH RISK"
        elif value > 5000:
            return "MEDIUM RISK"
        return "LOW RISK"

Use it like:

stream.map(RiskClassifier())

Input:

2000
7000
12000

Output:

LOW RISK
MEDIUM RISK
HIGH RISK

Much cleaner.

And far easier to test.

When Should You Use MapFunction?

Use it when:

• Logic becomes complex
• Reusability matters
• You want cleaner code
• Business rules keep changing

Basically…

If your lambda starts needing line breaks, it’s time.

FlatMapFunction — One Input, Many Outputs

We already learned that flat_map() can split one input into multiple outputs.

But with UDFs, it becomes much more powerful.

Suppose you receive comma-separated product names:

laptop,mouse,keyboard

You want each item separately.

from pyflink.datastream.functions import FlatMapFunction

class ProductSplitter(FlatMapFunction):
    def flat_map(self, value):
        for item in value.split(","):
            yield item.strip()

Use:

stream.flat_map(ProductSplitter())

Output:

laptop
mouse
keyboard

Neat.

No unnecessary nesting. No ugly loops outside Flink.

Real Use Cases for FlatMapFunction

This one is surprisingly useful.

You’ll use it for:

• Tokenising text
• Exploding arrays
• Splitting CSV-like payloads
• Parsing nested JSON lists

Or, as I like to call it:

The “one becomes many” operator.

Very dramatic. Very accurate.

ReduceFunction — Incremental Aggregation

Now things get interesting.

Suppose you’re processing sensor readings:

(sensor_1, 20)
(sensor_1, 25)
(sensor_1, 30)

You want a running total.

from pyflink.datastream.functions import ReduceFunction

class SumReadings(ReduceFunction):
    def reduce(self, value1, value2):
        return (value1[0], value1[1] + value2[1])

Pipeline:

stream \
    .key_by(lambda x: x[0]) \
    .reduce(SumReadings())

Output:

(sensor_1, 45)
(sensor_1, 75)

Every new event updates the aggregate.

No waiting.

No batch jobs.

Pure streaming.

Why Reduce Is Powerful

Because it avoids recomputing everything.

Without reduce:

• Read all records
• Aggregate again
• Waste resources

With reduce:

• Maintain running state
• Update incrementally
• Much faster

Flink loves incremental computation.

And your CPU will thank you.

ProcessFunction — The Ultimate Escape Hatch

Now we reach the crown jewel.

If transformations are tools…

ProcessFunction is the entire toolbox.

This is Flink’s most flexible low-level operator.

You use it when built-in transformations are not enough.

What Makes ProcessFunction Special?

Unlike map/filter/reduce, ProcessFunction gives you:

• Full control over each event
• Access to event metadata
• Ability to emit custom outputs
• Timer support (later article!)
• Side outputs (later article too!)

This is where production logic lives.

Simple ProcessFunction Example

Suppose we want to detect large transactions.

Input:

("Tony", 2500)
("Steve", 12000)
("Bruce", 500)

Logic:

• If amount > 10000 → ALERT
• Else → NORMAL

from pyflink.datastream.functions import ProcessFunction

class FraudDetector(ProcessFunction):
    def process_element(self, value, ctx):
        user, amount = value        if amount > 10000:
            yield f"ALERT: {user} made suspicious transaction of {amount}"
        else:
            yield f"Normal transaction by {user}"

Use:

stream.process(FraudDetector())

Output:

Normal transaction by Tony
ALERT: Steve made suspicious transaction of 12000
Normal transaction by Bruce

That’s already far beyond what simple map/filter pipelines usually do.

Where ProcessFunction Shines

This is where things get real.

Use ProcessFunction for:

• Alerting systems
• Fraud detection
• Routing events
• Custom enrichment
• Event correlation
• Complex business rules

Basically:

If you think “I need more control,” you probably need ProcessFunction.

UDFs vs Inline Lambdas

A common beginner question:

Should I always use UDF classes?

No.

Use lambdas when logic is tiny:

stream.map(lambda x: x.upper())

Use UDF classes when logic grows:

stream.process(CustomBusinessLogic())

Rule of thumb:

• 1 line → lambda
• 10 lines → UDF
• 50 lines → probably ProcessFunction

And 500 lines?

You may need therapy.

Wrapping Up

You’ve now unlocked one of the most powerful parts of PyFlink.

We covered:

• Why UDFs matter
• MapFunction
• FlatMapFunction
• ReduceFunction
• ProcessFunction

These functions let you go beyond simple transformations and start writing real production logic.

In earlier articles, we learned how to:

• Set up PyFlink
• Build DataStreams
• Apply transformations

In this article, we learned how to make Flink behave exactly the way we want.

And this is only the beginning.

What’s Next?

In the next article, we’ll dive into something Flink is famous for:

Stateful Stream Processing

We’ll explore:

• How Flink remembers information
• Why state matters
• ValueStateDescriptor
• ListStateDescriptor
• MapStateDescriptor
• How state is managed internally

Because once state enters the picture…

That’s when streaming pipelines stop being smart and start becoming intelligent.

Until then, happy streaming ✨