Why “We Support Every Vendor” Is Almost Always a Lie
Six years of multi-vendor TR-069, and what the spec does not tell you
At 02:40 a PON port fails in a city you have never visited. Four hundred and twelve subscribers lose service. Your monitoring platform raises four hundred and thirteen alarms, your phone starts buzzing, and somewhere in that noise is a single fibre that needs a technician.
You open the management console. You can see the Huawei terminals. You can see the ZTE ones. What you cannot do is ask one question that covers both.
So you open a second console. Then a spreadsheet.
This is the part of running an internet service provider that nobody puts on a datasheet, and it is the reason I have spent the last several years building an operating system for operators rather than another point tool. What follows is what we have actually learned — including the parts that make our own claims smaller.
The promise
TR-069 is a good standard. TR-181 — the data model that sits on top of it — is a genuinely thorough piece of specification work by the Broadband Forum. It defines where the WAN configuration lives, where the Wi-Fi settings live, where the device information lives. If every vendor implemented it completely and identically, multi-vendor management would be a solved problem and this article would not exist.
Every vendor implements it. Nobody implements it the same way.
The gap between those two sentences is where operators lose their nights.
Six things that break, in the order they will break for you
1. The same setting lives in different places
Two models from the same vendor, same firmware branch, and the Wi-Fi channel parameter sits at a different node on each. Not a different vendor — the same vendor, two product lines, two different engineering teams.
The lesson we took from this cost us a quarter: build your capability map per model, never per vendor. The moment you write code that says “if vendor == X then path = Y” you have built something that will be wrong within six months.
2. Silent success
A SetParameterValues returns 0. No fault, no error, clean response. The device never applies the value.
You find out three weeks later when a subscriber complains that the speed change they paid for never happened, and your platform has been reporting it as applied the entire time.
We now treat every write as unconfirmed until a GetParameterValues read-back proves it. That doubles the round trips and it is worth every one of them. "No error" is not the same as "it worked", and a management platform that conflates the two is lying to its operator by default.
3. Partial batch application
Send five parameters in one call. Three apply, two do not, and you receive one success response covering the whole batch.
Either send them individually and accept the latency, or verify each one afterwards. There is no third option that is honest.
4. A timeout and an unsupported capability look identical
This one matters more than it sounds.
Most auto-configuration servers return a timeout when a device does not answer, and also return a timeout when the device does not support what you asked for. At 02:40 those two outcomes look exactly the same on the screen — and one of them means “get in the van”.
We return a typed unsupported so the operator knows the difference between a capability gap and a dead device. It is a small design decision that changes what the night shift does.
5. Inform storms after a power event
Every subscriber unit in a zone loses power. Power returns. Every one of them informs at the same moment.
If your inform interval is 300 seconds and you have twenty thousand devices, your steady-state load is about sixty-six requests per second — comfortable. Your burst load is twenty thousand requests in a few seconds, and that is a different machine entirely.
Plan for the burst. Add jitter to the inform interval. And test it, because a platform that has only ever been load-tested against the average will fall over on the first area-wide outage.
6. Firmware push ordering
Some devices reboot into the new image before acknowledging the download. Your platform sees no acknowledgement, marks it failed, and retries.
Now you are pushing firmware to a device that is mid-flash.
Get Saim Abbas’s stories in your inbox
Join Medium for free to get updates from this writer.
We have seen this brick units. The fix is not clever — it is simply knowing that this behaviour exists on certain models and handling it as a known case rather than as a generic retry.
What actually works
Grade the support, do not claim it
This is the single most useful thing we did.
Instead of a binary “supported / not supported”, every device model is certified at one of four tiers:
Tier What it means Tier 0 — Reachable Identity, uptime, reachability. Works with anything that answers. Tier 1 — Monitored Interfaces, optical power, alarms, environment, configuration read. Tier 2 — Managed Provisioning and writes: authorise a unit, set a VLAN, push firmware, zero-touch activation. Tier 3 — Optimised Vendor-specific telemetry, bandwidth allocation, rebalancing.
The tier is published per model, before purchase.
This sounds like a marketing downgrade. It is the opposite. It lets you say yes truthfully. A customer with an unusual terminal is not told “unsupported” — they are told it is Tier 1 today, what Tier 2 would require, and roughly when it could arrive.
An operator told in advance that their terminal supports monitoring and authorisation but not remote de-authorisation will plan around it. The same operator discovering that during a migration will not forgive you.
Ship a capability manifest, then distrust it
Every model ships with a manifest declaring what it can and cannot do. Then — and this is the part that matters — the platform probes the device at onboarding and reconciles what it actually exposes against what the manifest claims.
The manifest is wrong more often than you would like. Firmware updates change behaviour. A vendor adds a parameter and does not tell anyone. A regional SKU differs from the global one.
Reconciling at onboarding turns a future outage into a line in a report.
Fail loudly, early, and specifically
Classify every outcome: success, unsupported, refused, malformed, timed out. Never “assume success because nothing came back.”
And when a multi-step operation fails halfway through, reverse it back to the state before it started. A half-applied service profile is worse than a failed one, because nobody knows it is there.
The part where I am honest about our own limits
I build Netxol, an operating system for internet service providers — one appliance in the operator’s own rack that runs the network, the subscribers, the billing and the business on a single connected data model. Multi-vendor device management is one layer of it.
So I have an obvious interest in telling you that multi-vendor management is hard and that we have solved it. Here is the more useful version:
We have not solved it. Nobody has. What we have done is stop pretending the problem is binary.
A vendor who will not expose an interface cannot be forced to. When a model cannot do something, we publish that at a lower tier and the customer is told before they buy, not during a migration. Our support matrix has gaps in it and we show the gaps.
EPON remains the hard case. GPON is tightly specified and vendors implement most of it. EPON is looser — terminals expose optical power at the terminal but not per unit, allow a unit to be authorised but not de-authorised, or report different identifiers depending on which command you use. We handle what we can and document what we cannot.
If you want the detail, the full architecture — including the threat model with its residual risks named — is written up and available. We maintain a register of 736 capabilities with an honest status against each one, and we will show it to anyone who asks, including the parts that are not built.
A company that only shows you what is finished is showing you a brochure.
If you take one thing from this
Capability is per model, not per vendor. Publish it. Probe it. Reconcile it.
Everything else in this article is downstream of that one decision.
I am Saim Abbas, CTO at BesTel Communications and the architect of Netxol — an AI-native operating system for internet service providers. We build the layer that sits above the access network, vendor-neutral and on the operator’s own hardware.
If you run an ISP and any of the above sounded like your last six months, I would genuinely like to hear which vendor caused it. I collect these.
Contact: saim.abbas@netxol.com
Tags: Telecommunications · Networking · ISP · Fiber Optics · Software Architecture CRM NMM NOS Billing RADIUS AAA Authentication OLT Management
