“I Have a 10GB CSV File and Only 512MB RAM” — The Interview Question That Stumped Me

Sarveshkhamkar

5 min read

·

May 27, 2026

--

178

Listen

Share

And how a desperate mention of Kafka saved my answer (sort of).

Press enter or click to view image in full size

I was in a Java developer interview. Things were going well — I had answered questions about Spring Boot, microservices, and even a few LeetCode-style problems.

Then the interviewer leaned back and asked:

“Suppose you have a very large file — say 10GB of CSV data. You have very limited computational resources — maybe just 512MB of RAM and a single CPU core. How would you process this file to pre‑process the data and store it in a different format?”

I paused. My mind raced.

“Just read the file line by line?” — too obvious, probably wrong.
“Use multithreading?” — but resources are limited.
“Load it into a database?” — but then the database would also need memory.

I knew I had to say something intelligent.

My First Answer (Spoiler: It Wasn’t Great)

I asked for clarification: “What kind of data? Is it structured?”

Interviewer: “It’s a CSV file — rows and columns. You need to clean it, transform some columns, and save it as JSON or Parquet.”

After a few seconds of thinking, I answered:

“We can divide the rows into chunks and process each chunk iteratively. We’ll keep track of the byte offset of each chunk — for example, read 10,000 rows at a time, process them in memory, write the result to a new file, then move to the next offset. That way, we never load the whole file at once.”

The interviewer nodded slowly, but his face said: “That’s still too naive.”

He pushed back: “Even reading 10,000 rows at once might be heavy if each row is large. And what about fault tolerance? What if the process fails in the middle? How do you resume?”

I was stuck. I knew my answer was incomplete.

Then, out of desperation, I said:

“I’m not 100% sure about the low‑level implementation, but I know this kind of problem is typically solved with stream processing frameworks like Apache Kafka or Apache Flink. They handle exactly‑once semantics, partitioning, and fault tolerance out of the box.”

The interviewer’s eyebrows raised slightly. He didn’t say I was wrong. But he also didn’t say I was right.

I left the room feeling that I had survived, but not truly answered the question.

What I Learned After the Interview

After the interview, I went home and researched. The problem is actually a classic one: processing large files with limited memory.

The ideal answer is not about Kafka or Flink — those are overkill for a single file on a single machine. The real solution is much simpler and more elegant.

Here’s what I should have said.

The Ideal Answer: Streaming + Chunking + Offset Tracking

Step 1: Never load the whole file into memory

Use a streaming CSV reader that reads one row at a time.
In Java, libraries like OpenCSV or Super CSV support this. Even plain BufferedReader with String.split(",") works for simple cases.

java

try (BufferedReader reader = new BufferedReader(new FileReader("large.csv"))) {
    String line;
    while ((line = reader.readLine()) != null) {
        processLine(line);   // transform, validate, etc.
        writeToOutput(line); // append to a new file
    }
}

This uses almost no memory — just enough for one line.

Step 2: If you must batch (for performance), do it smartly

If the CSV parsing itself is expensive (e.g., complex validations), you can batch 1000–5000 rows at a time. But never more than a small fraction of your RAM.

java

List<String> batch = new ArrayList<>(5000);
while ((line = reader.readLine()) != null) {
    batch.add(line);
    if (batch.size() == 5000) {
        processBatch(batch);
        batch.clear(); // free memory
    }
}

Step 3: Track progress for fault tolerance

If the process fails, you don’t want to restart from the beginning. Keep a record of the last processed line number or byte offset.

You can write a tiny checkpoint file:

text

offset = 1048576   # bytes processed so far

If the program crashes, read the checkpoint, seek to that offset, and resume.

In Java, you can use RandomAccessFile to seek to a specific byte position.

java

RandomAccessFile raf = new RandomAccessFile("large.csv", "r");
raf.seek(lastKnownOffset);   // resume from here

Step 4: Use producer‑consumer pattern (optional)

If you have a single CPU but still want to overlap I/O and processing, use two threads:

• Thread 1 (Producer) — reads lines from the file and puts them into a small queue.
• Thread 2 (Consumer) — takes lines from the queue and processes them.

This keeps the disk busy while the CPU works. But with very limited resources, even this may be overkill.

Wait — Could Kafka or Flink Be the Right Answer?

Yes, but only if the problem is distributed.

If the “limited computational resource” means a single small machine, Kafka and Flink are not the answer — they add overhead.

However, if the file is on a distributed file system (HDFS) and you have a cluster of small nodes, then Kafka + Flink (or Spark Streaming) makes perfect sense. They partition the data, process in parallel, and provide exactly‑once guarantees.

The interviewer might have been hinting at stream processing as a concept, not necessarily the specific tools.

My mistake was mentioning tools without explaining the principles:

• Partitioning
• Checkpointing
• Exactly‑once semantics

The Correct One‑Sentence Answer

If I could go back, I would say:

“I would use a streaming CSV reader to process the file line by line, batching only a few thousand rows at a time to avoid memory overload. I would also implement checkpointing by saving the byte offset periodically, so that if the process fails, it can resume from where it left off — not from the beginning.”

And if the interviewer asked about distributed systems:

“If the file is spread across multiple machines, I would use a stream processing framework like Kafka + Flink to partition the data, process each partition independently, and use checkpointing for fault tolerance.”

What This Taught Me

• Don’t jump to big frameworks before understanding the core problem.
• Always ask about scale — is this a one‑time job on a single machine, or a continuous pipeline on a cluster?
• Fault tolerance and resumability are often more important than raw speed.
• Even a wrong answer can be valuable — it pushed me to learn the right one.

Your Turn: How Would You Answer?

Have you ever faced a “large file, small memory” question? What was your answer?

Drop a comment — I’d love to hear your approach.

If this article helped you, clap 👏 and follow for more real‑world interview stories.