VPC Flow Logs Ingestion to Amazon OpenSearch

Aaditya Paliwal

4 min read

·

Feb 13, 2026

--

Listen

Share

How We Ingested 500+ GB/Day of VPC Flow Logs into OpenSearch (Across 20+ AWS Accounts) using OpenSearch Ingestion Pipeline

When your AWS Organization crosses 20 accounts and spans multiple regions, VPC Flow Logs stop being a feature and start becoming a system.

At 500+ GB per day — and growing — logging becomes an engineering problem. Not a checkbox. Not a dashboard toggle. A distributed systems problem.

This is how we built a centralized, cross-account, event-driven pipeline into OpenSearch that survives scale, disk pressure, and rapid growth.

And yes — we hit cluster RED along the way.

The Reality of Scale

Our environment looked like this:

• 20+ AWS accounts
• Multiple regions per account
• All VPC Flow Logs enabled
• Central S3 consolidation
• Separate analytics account running OpenSearch
• 90-day retention
• Okta SSO

At 500 GB per day, you are ingesting roughly 15 TB per month.

That’s not logs. That’s infrastructure.

Architecture: Designed for Growth

We intentionally separated responsibilities:

• Spoke Accounts → Generate logs
• Logging Account → Stores logs in S3
• Analytics Account → Ingests and indexes logs
• Okta → Handles authentication

Press enter or click to view image in full size

The ingestion path looks like this:

Spoke Accounts

→ Central S3 Bucket

→ S3 Event

→ SNS

→ SQS

→ OpenSearch Ingestion (OSI)

→ OpenSearch Cluster

This event-driven design prevents polling chaos and gives us buffering via SQS.

Buffers save distributed systems.

Why Event-Driven Instead of Polling?

Polling S3 at TB scale means:

• Excess API calls
• Delayed ingestion
• Inefficient scaling

Instead, S3 publishes object-created events to SNS.

SNS delivers to SQS in the analytics account.

OpenSearch Ingestion reads from SQS and pulls the object from S3.

This gives us:

• Near real-time ingestion
• Backpressure handling
• Failure tolerance
• Cross-account isolation

It also means ingestion won’t collapse if OpenSearch slows down temporarily.

IAM: The Part Everyone Underestimates

Cross-account ingestion requires clean trust boundaries.

The analytics account hosts an OSI role that can:

• Read from the central S3 bucket
• Consume messages from SQS

The logging account’s bucket policy explicitly allows that role.

We restrict access using:

• AWS Organizations condition keys
• Prefix-level permissions

Loose IAM in logging pipelines becomes a security incident waiting to happen.

OpenSearch Cluster Design (Where Things Get Serious)

At 500+ GB/day, cluster design is not optional tuning. It is survival.

We run:

• 3 Dedicated Master Nodes
• SSD-backed Hot Data Nodes
• HDD-backed Warm Nodes
• Cold Tier

Dedicated masters prevent ingestion load from destabilizing cluster state.

Disk Strategy (And the Cluster RED Incident)

OpenSearch protects itself using disk watermarks:

• 75% → Low watermark
• 85% → High watermark
• 95% → Flood-stage

At flood-stage:

• Indices become read-only
• Writes fail
• Cluster turns RED

We hit flood-stage when ingestion outpaced scaling.

That was our lesson in physics.

To recover:

1. Deleted old indices
2. Cleared read-only flags
3. Added data nodes
4. Increased EBS volumes (If possible)

Long-term, we:

• Alert at 70% disk
• Enforced strict index lifecycle policies
• Forecast storage quarterly

You don’t “fix” disk pressure. You engineer around it.

Daily Indexing: Non-Negotiable

Each day creates its own index:

vpc-flowlogs-YYYY.MM.DD

Why daily?

Because 500 GB in a single rolling index is reckless.

Daily indexing:

• Limits shard growth
• Reduces corruption blast radius
• Enables clean lifecycle transitions
• Simplifies reprocessing

Shard size is maintained between 30–50 GB.

Too small → too many shards

Too large → slow recovery

Shards are memory consumers, not just files.

Lifecycle Management: Hot → Warm → Cold → Delete

Logs live in stages:

• 0–7 days → Hot (SSD)
• 8–30 days → Warm (HDD)
• 31–90 days → Cold
• 90 days → Delete

Without lifecycle management:

500 GB/day becomes 45 TB in three months.

Storage grows quietly. Then suddenly.

Authentication via Okta

We integrated OpenSearch Dashboards with Okta using SAML.

User → Okta → SAML Assertion → OpenSearch

Benefits:

• MFA enforced
• Centralized identity
• Role mapping via Okta groups
• No local OpenSearch users

Security and audit teams appreciate that design.

Let’s Talk Money

At 500 GB/day (~15 TB/month):

OpenSearch infrastructure typically lands around:

$10K–$12K per month

CloudWatch Logs:

$10K–$13K per month

Splunk:

$75K–$100K per month

Splunk is powerful. It is priced like it knows that.

What Happens If Volume Triples?

Let’s say we grow to 1.5 TB/day.

Now we’re ingesting 45 TB/month.

OpenSearch scales roughly with infrastructure:

$25K–$30K per month

CloudWatch scales directly with ingestion pricing:

$30K–$40K per month

Splunk licensing at 2 TB/day:

$200K+ per month

At scale, ingestion-based pricing becomes painful.

Infrastructure-based pricing becomes predictable.

That difference matters when forecasting growth.

Lessons Learned

1. Centralize storage. Decouple processing.
2. Use event-driven ingestion with SQS buffering.
3. Daily indexes are mandatory at this volume.
4. Disk watermarks are not suggestions.
5. Capacity planning must stay ahead of ingestion growth.
6. Cost modeling should assume next year’s volume — not today’s.

Final Thought

VPC Flow Logs are often treated as compliance exhaust.

At 500 GB/day, they become a living map of your network’s behavior.

When designed properly, this pipeline doesn’t just store traffic records.

It turns distributed chaos into searchable intelligence.

And the difference between those two states is architecture.