Evaluating Technical Documentation
Korean version
How do we measure the quality and value of technical writing in the age of AI?
While working as a technical writer, there’s a question that keeps coming up without any solid answer: “How should we evaluate a technical writer’s work?”
It’s a question management asks during performance review season, and it always pops up when technical writers get together to debate.
We can break this problem down into two parts.
- First, how much did the technical writer’s output contribute to the company?
- Second, how much did the actual deliverables handled by the technical writer improve?
The first refers to how much product sales or business performance changed because of the technical documentation, while the second refers to how much the technical writing itself — the documentation — actually got better.
Evaluating Contribution
The first part is just as hard to measure for technical writers as it is for any engineer or support staff belonging to a non-product team. For example, a recruiting team can evaluate the candidates they hire (their work output), but it’s hard to measure how much those hires actually increased company revenue. A security engineer can see how much a product’s security vulnerabilities (work output) have decreased, but measuring how much fixed costs were cut because of it is tough. Just like this, there are many roles where you can evaluate the output itself, but not the indirect contribution to the company as a whole.
Still, scraping the bottom of the barrel for ideas, a few methods have been proposed:
One approach is to measure contribution by looking at the number of customer inquiries or complaints — and the support team’s time spent handling them — comparing periods with and without documentation, or when providing docs that haven’t been touched by a technical writer versus ones that have. But customer inquiries and complaints are driven by more than just documentation, so if you’re unlucky, the metrics might actually look worse after deployment. Besides, sometimes companies don’t track customer complaints in the first place so there’s no baseline, or the deliverable is a one-off document or blog post that doesn’t generate direct inquiries. Another idea is using page views (PV) on technical docs to measure impact. While PV can indicate customer interest, it’s hard to prove a direct link saying interest grew *because* the documentation was well-written. If people are interested in a product, PV might go up even if the docs are written poorly. So, realistically, there’s no good way to measure how much a technical writer contributes to company revenue.
Evaluating Deliverables
The second part is a bit different. This is about seeing whether technical docs written by a technical writer explain the product more easily and accurately than those written by other roles, which is practically a measure of the technical writer’s competence. Because of this, discussions among technical writers have sometimes drifted toward using certifications. Apparently, the US has technical communication degrees and certifications that industry insiders recognize. Getting certified means you have technical writing skills, and you can trust that what you write will be reasonably good. However, since all deliverables produced by a single technical writer would receive the same evaluation, it’s essentially evaluating the person rather than the output. Another proposed method is surveying developers or product managers who worked with them. Compared to certifications, this is a bit subjective, and if you write bad survey questions, it easily gets distorted into an evaluation of the person. An experimental survey used at a former company had questions like, “Did the technical documentation improve after the technical writer joined?” or “Were requests handled well?” Anyone who has done peer reviews knows that if you have to keep working with that person, you’re likely to give them a good rating anyway.
So, is there really no way to evaluate the actual output instead of the person?
Fortunately, a few years ago at an overseas conference, I heard about test-based documentation evaluation. It’s applicable to executable, tutorial-style technical documentation: you build a program that directly runs the code and commands shown in the doc, and verify whether it runs without issues and proceeds exactly as the text describes. Recently, methods using AI to do similar kinds of verification have also been introduced.
However, as mentioned earlier, the types of documentation this works for are limited, and command-execution accuracy isn’t the entirety of technical document quality.
So, to sum it up, there hasn’t been a proper way to evaluate deliverables either — until now.
The introduction got pretty long with the phrase “how to evaluate a technical writer’s work.” As you’ve probably guessed, this was all a build-up to talk about “the quality of technical documentation.”
If we had a quantitative way to measure that a technical document was well-written (the first part), we could eventually use it when a method arises to prove that those well-written docs actually contributed to company revenue (the second part).
Technical Documentation Quality in the AI Era
Quality evaluation is even more necessary in the age of AI. If we leave documentation writing to AI, we need to check if the AI wrote it properly. But if humans try to do that, we’ll run into the “human bottleneck” situation that’s trending everywhere lately. On the flip side, if we build a way to evaluate documentation quality and use it to score AI-written text, humans only need to review the substandard outputs. Also, to look at quality, tracking the trend of improvement over time matters more than a snapshot at a specific moment, which requires periodically evaluating quality under the same conditions. If humans had to do this, it would become a bottleneck, but with proper measurement criteria, we can hand off repetitive evaluation tasks to AI.
To sum it up, in the AI era, AI writes the docs, AI evaluates the quality to decide whether a human needs to look at them, and a quality dashboard lets us see if human intervention is actually improving the docs.
Evaluation Criteria & Metrics
To tell AI to evaluate documents, we need to set evaluation criteria. To do that, we have to figure out what matters most in technical documentation and how to measure it. I’ve already laid out what matters in the titles of the three sections in Part 2 of my book, <Who Says Developers Can’t Write? (Korean book)>:
Accuracy, Conciseness, and Completeness.
Accuracy means writing things without errors, conciseness means making things easy to understand, and completeness means covering everything that needs to be included. In fact, these three are the ultimate goals of technical writing.
So how do we judge whether something is written accurately, concisely, and completely?
Get Ragina Jeon’s stories in your inbox
Join Medium for free to get updates from this writer.
Our team divided each item into generic metrics (which can be measured regardless of the product) and product metrics (which require product knowledge). Generic metrics include freshness, consistency, readability, structure, and AI readiness. Product metrics include feature coverage, information architecture, and cross-doc consistency. Information architecture and cross-doc consistency are similar to generic metrics and haven’t been discussed in detail within the team yet, so I’ll leave them out for now. Connecting just the discussed metrics to the ultimate goals of technical writing looks like this:
As you can see, all metrics tie back to accuracy, conciseness, and completeness.
To elaborate, personally, I think “writing to be easily understood by humans” has lost its meaning in the AI era. Do we really need to consider human readability when humans are reading technical docs less and less? Humans have varying levels of knowledge and attention spans, so we used to need to explain things as simply as possible and structure explanations for the human brain (e.g., avoiding listing more than 5 things at once). But AI knows more than humans and doesn’t lose focus, so why bother putting effort into human readability? (Though people say AI forgets parts if the text is too long, unfortunately, improving readability in technical writing isn’t just about shortening the length.)
Taking it a step further, do we even need to consider AI readability? With advancing models, we often hear that “even if you write like trash, AI figures it out brilliantly.” I’ve heard stories where people built templates for AI to read easily, but when a new model came out, it read it just fine even without the templates. Between templates and what I covered a few months ago in my post <Writing for AI>, methods for improving AI readability focus on format or expression, but since many models now read fine regardless of format or expression, it might no longer be valid. However, the method of optimizing for RAG search — which I touched on partly in writing for AI — is still valid. It’s meant to boost discoverability, which is always cited as a problem in technical docs. In short, even if techniques for making things easy for AI to understand are no longer valid, techniques for making things easy to search via AI are still worth considering. Our team grouped AI readability and AI searchability together under “AI readiness,” and I feel like we could probably drop AI readability going forward.
You might wonder why discoverability isn’t in the metrics, but I don’t think ‘discoverability’ is a metric of document quality itself. Making documents easy to find requires tweaking company doc culture, policies, infrastructure, and engineering rather than the documents themselves. Therefore, while discoverability is a must-consider factor when planning technical documentation, it’s not appropriate as a document evaluation metric. However, I believe writing techniques to enhance AI searchability from a document creation standpoint are still worth researching.
After writing the blog post shared earlier, our team did some research on finding document quality metrics, and a presentation about document quality came up at an overseas conference recently as well:
The speaker is a technical writer at Booking.com, and she presented five metrics that, interestingly, look very similar to our team’s results.
The metrics that differ from ours are style and discoverability. I already explained why we didn’t include discoverability, so let’s talk about the style metric.
Booking.com made adherence to a style guide a key metric, which I think is necessary for publicly facing docs. In an era where AI reads and writes documentation, document style doesn’t matter as long as the content is accurate, but nevertheless, public docs affect company reputation and brand, so you still have to maintain standards. That’s probably why Booking.com included the style metric. However, since our team researched quality metrics to design an internal document management process for the AI era, whether style guidelines were strictly followed wasn’t included as a metric. That said, since the consistency metric includes items you’d typically find in a style guide, like “using consistent terminology,” we can’t say style scores are entirely absent.
Ultimately, while our metrics differ slightly depending on our respective goals, the items and meanings we want to measure are practically the same. If we get into the detailed measurement methods, it probably feels even more similar.
If you ask me which of these many metrics is the most important, it’s freshness, consistency, and feature coverage. In terms of technical writing techniques, it boils down to two things, excluding conciseness (=readability) as mentioned earlier: accuracy and completeness. As long as you
- Cover everything that needs to be addressed,
- write it without errors, and
- deliver it on time
it doesn’t matter if the document is in Korean or Arabic, formatted in Markdown or MS Word, or written in formal, informal, or rough prose. AI will read it just fine anyway.
How to Measure Metrics
How our team measures each metric is a story for another time since it takes a while to explain. Instead, let’s compare some of Booking.com’s measurement methods with ours. (You can check how Booking.com measures each metric in their video.)
Booking.com identifies product features and measures the coverage metric by checking whether conceptual, procedural, and reference documentation exists for each feature. I was surprised to find it almost identical to our team’s thinking. The only slight difference is that in our team, some argued that document types don’t necessarily equal feature explanations. Much like doubting the validity of readability, we questioned whether document types are still valid in the AI era. So we considered a method that looks at actual content mentions alongside document types.
Booking.com apparently didn’t use traditional readability scores because they felt they didn’t fit technical docs, instead defining their own “visual conciseness” (which you can find in my book) for the readability metric. They also evaluate page layout and overall information architecture (IA) as readability metrics, which matches our team’s structure and information architecture metrics.
Booking.com measures discoverability based on whether an AI agent can read the docs and accurately answer questions in an FAQ. Interestingly, our team uses a similar approach for our AI readability metric. In a RAG system built on internal tech docs, part of the AI readability score is based on how closely the AI’s technical answers match the ground truth written by the owner. We didn’t build this part, so we just use that value as-is, while our team calculates the final score by measuring whether it complies with the format/expression guidelines covered in the blog post mentioned earlier. In a way, what Booking.com calls discoverability, we call AI readiness/AI readability.
How to Use Metrics
You might be wondering where to use these metrics. As mentioned earlier, they can serve the following roles:
- A gate to evaluate AI-written documents and decide whether humans need to review them
- A dashboard showing the trend of internal technical documentation improvements
- (Bonus) A guide that evaluates documents and provides feedback on how to improve them
If automated, a quality evaluation agent could measure each metric, summarize how to route and improve the target docs, send substandard docs to humans, and pass the rest to the documentation writing agent for automatic improvement based on the guide. You can also get a snapshot at a glance of which documents improved by how much over time, which ones are up to date, and so on.
When you think about it, it’s nothing groundbreaking. It’s everything we already do in software development, just replacing code with documentation. Just as developers are great at writing code, so they orchestrate dev-focused AI agents and build harnesses, technical writers are great at writing docs, so it’s only natural for technical writers to orchestrate doc-focused AI agents and build harnesses. As I’ve said many times, this is the path technical writers should take in the AI era.
Next Steps
Personally, my team tried building this, but we hit pause due to organizational direction and headcount issues. But I’m not stopping here — I’m going to try building it at my new company.
Sometimes doing new things makes your heart feel full, and one of those moments is realizing that other people are doing the exact same work you’ve been pursuing. Knowing that gives me the comfort of knowing, “My direction wasn’t wrong.” Seeing Booking.com’s presentation gives me a little reassuring peace of mind once again.
