How We Architected OpsX (opsx.in): The Edge-First Attendance, Payroll & Workforce ERP by Opsx Apex Private Limited

JavaScript With Vinay

10 min read

·

9 hours ago

--

Listen

Share

At 09:00 a factory gate has forty people in a line and one tablet on the wall. The Wi-Fi drops twice a week. If every punch waits on a round trip to a server in another city, the line waits too, and the people at the back come in “late” for something that was never their fault.

That one scene shaped most of OpsX (opsx.in), the attendance, payroll and workforce ERP we build at Opsx Apex Private Limited. Inside our repo the project is still called ims-saas, because it started life as an Industrial Management System and grew outward from there. This post is the architecture as it is built today, including the parts we would do differently.

What does “edge-first” mean for an attendance system?

The edge is wherever a person punches: the employee’s phone, the kiosk tablet at the gate, the biometric terminal on the wall. Edge-first means those devices capture, verify and queue the punch on their own, so the line keeps moving when the network does not. It does not mean the device is trusted. The server still owns the time of the punch, the rules of the shift and every rupee that comes out of payroll.

That split is the whole design in one sentence: the device decides fast, the server decides finally.

What does the OpsX architecture look like?

One backend codebase, three Node entrypoints, two web apps and two mobile apps.

• apps/api: NestJS 11, REST under /api. Two instances behind nginx, restarted one at a time on deploy.
• apps/realtime: a Socket.IO gateway on a Redis adapter, for live notifications.
• apps/worker: BullMQ consumers and cron jobs. It has no HTTP listener at all.
• Control Center: an Angular 21 SPA for us, the platform owner. Tenants, plans, billing, form engine, feature flags.
• ERP Portal: an Angular 21 SPA for the customer. HR, attendance, payroll, inventory, procurement.
• OpsX Partner and OpsX Kiosk: two Expo / React Native apps, one for the employee’s own phone, one for a shared tablet at a site.

The three backend apps import the same modules from libs/backend. They differ only in which providers they boot. Shared DTOs, enums and event names live in @ims/contracts, which both the Angular apps and the NestJS apps import, so a status string cannot drift between the screen and the database.

Press enter or click to view image in full size

We also wrote down a GraphQL design at the start. It was never built. The API is REST only, and the old design pages are marked as such, because a blueprint that describes a system nobody built is worse than no blueprint.

How does a kiosk keep punching when the internet drops?

A face punch is verified on the tablet, signed with the tablet’s secret, written to a local queue, and the person is greeted at once. The network catches up later.

The pieces, in order:

1. The camera frame goes through two ONNX models on the device: YuNet finds the face, a face embedder turns it into a vector. A head-turn check runs before the match, which makes a photo held up to the camera much harder to pass off as a person.
2. The vector is matched against the site’s face templates. Those templates live in their own SQLCipher database file on the tablet, and its 256-bit key sits in the Android keystore through SecureStore. A wall tablet is a device somebody can pick up; biometric data on it is encrypted at rest.
3. The punch claim is signed with the terminal’s secret and inserted into punch_queue with INSERT OR IGNORE. Its punchId is derived from the claim itself, so a retry collides with the original instead of creating a second punch.
4. Every 20 seconds the kiosk drains the queue, oldest first, in batches. The server answers per punch. A duplicate counts as success. A punch that keeps failing is quarantined after a fixed number of attempts, not retried forever.
5. Every 30 minutes the tablet pulls a delta of face templates. The roster cache is considered stale after 12 hours.

Press enter or click to view image in full size

The interesting part is what is not queued. A PIN punch needs the network. The PIN is checked on the server, and the roster cached on the tablet deliberately leaves out every PIN hash. A tablet carrying every PIN hash of a site turns one stolen device into an offline cracking exercise against the whole workforce. So a PIN punch with no connection would be an identity nobody verified, and the kiosk does not pretend otherwise.

Face enrolment is also online only. The server runs a duplicate check across the whole tenant before it accepts a new face, and that check cannot run on one tablet.

Can you trust the time a phone says a punch happened?

No. OpsX treats the punch time as an age, not a wall clock.

A phone’s clock belongs to the employee. If the server took occurredAt at face value, changing the phone's time would turn a late arrival into an on-time one. So the app and the kiosk send two readings from the same device clock: occurredAt (when the punch happened) and sentAt (when the request left). The server computes sentAt − occurredAt, which is the punch's age, and subtracts it from its own clock. Any offset on the device cancels out, because both readings carry it.

The guards around that:

• An age above 12 hours is refused to server time. A day that old is a regularisation request with a reason, not a silent backfill.
• A negative age is refused.
• An old app build that sends no sentAt falls back to a 15-minute skew limit.
• Biometric terminals are the one exception. The ingest drain sets a server-only clockTrusted flag; the request DTO cannot carry it and the whitelist strips it if a client tries.

The kiosk’s face batch carries one sentAt for the whole request. Before we added that, a face punch that sat offline for more than 15 minutes was recorded at the time it reached the server, which after a long outage could land it after midnight, on the wrong day.

Why is “faceVerified” stored as evidence and not as proof?

On the employee’s own phone, the face match runs against an embedding cached on that phone. The server cannot check it. A modified client can send faceVerified: true and nothing on the server would catch it. That is the cost of matching on the device, and we chose it on purpose, because the alternative was sending daily face photos to a server.

So the punch row stores faceVerified and faceScore as evidence: a claim, a score, a gap between receivedAt and the punch time, a coordinate. Together they make a far better audit record than a bare timestamp, and a pattern of low scores shows up in aggregate. They never gate pay.

geofenceVerified goes the other way. The server holds the office coordinates and the radius, so it derives that field itself and never accepts it from the client. Taking the client's word for it would replace the check with a request to be trusted.

How do biometric terminals send punches without flooding the API?

The ingest endpoint returns 202 Accepted and does the work later. The tenant comes from a device registry, never from the request body, so a terminal cannot claim to belong to someone else.

The handler only authenticates the terminal, deduplicates, and parks each record as a pending event. It looks at no roster and no shift. A terminal that comes back online after a weekend replays its whole buffer at once, and if that burst had to wait on the attendance rules it would time out and replay again into the same congestion. A separate drain turns pending events into punches, inside the tenant's context. We learned the hard way that the drain needs a caller: the first version existed, was tested, and nothing ever invoked it, so a whole day of punches could sit at pending while every dashboard looked fine.

Records that match no employee are kept with a quarantined status and purged 30 days after the punch happened, counted from the punch and not from arrival, so a terminal replaying an old buffer cannot keep a stranger's record alive.

How does a punch become a payslip?

Every later stage reads a classified day, never a raw punch:

punch → day frame → classified day → payroll period → payslip → salary ledger
           ▲              ▲                  ▲
         shift       leave ledger      pay policy snapshot

The day classifier turns dates into frames and runs pure stages over the whole array in a fixed order. Each stage records why it decided what it did. Our first classifier decided each date on its own, and it was structurally unable to express one rule customers asked for: the sandwich rule, where a holiday between two absences counts as an absence. That rule asks a question about a day’s neighbours. The new pipeline sits behind an engineVersion, 21 fixtures pin the new engine to the old one's answers, and a shadow comparator runs both and reports every day where they disagree.

Two smaller decisions from this part saved real money:

• A punch outside the shift window is refused only when the shift was assigned. If the punch itself chose the roster (an inferred shift), measuring it against that roster asks whether the punch agrees with a guess made from itself.
• The idempotency check runs before the window check. A punch already recorded is a fact. A phone draining its offline queue at 22:00 must get “already recorded”, not see every punch of the day rejected.

How are tenants kept apart in one MongoDB?

The plan decides the physical layout. Starter and Professional tenants share one database, and every document carries a tenantId. Enterprise tenants get a dedicated database, opened lazily and pooled in an LRU of 50 connections with a 15-minute idle eviction, so the database's connection limit is not exhausted by tenants who are asleep.

A Mongoose plugin, tenantScopePlugin, injects the tenant predicate into every query. It also hard-blocks $lookup, $unionWith and $graphLookup inside aggregations, because a cross-collection stage escapes the scope without any error. The consequence for feature work is real: you cannot join inside a tenant aggregate. You run the aggregate, then enrich the page with a second scoped find({ _id: { $in: [...] } }).

The one path the plugin cannot see is the raw driver, db.collection(...). A repo guard, tools/check-raw-collection.mjs, fails the build on it. Our first version of that guard read code line by line and missed the second raw call in the same file. A guard that only catches the first occurrence teaches you to trust it too early.

Why does the general ledger live in PostgreSQL when everything else is in MongoDB?

Books need things MongoDB does not give you cheaply: NUMERIC, CHECK constraints the database enforces when the application is wrong, row-level security per tenant, and an append-only table where the delete privilege is revoked, not merely unused.

So all operational data (employees, attendance, stock, orders, documents) stays in MongoDB, where the form engine can write each tenant’s own fields. Only the double-entry general ledger is in Postgres.

The two are never joined by a distributed transaction. They are joined by a transactional outbox:

1. The business write and a financeOutbox row commit in the same Mongo transaction.
2. The worker drains the outbox every minute, with a concurrency of one across the whole worker.
3. A unique index on source_event_id in Postgres makes a double post impossible.
4. A failure is written on the outbox row and shown on the finance screen. Whatever cannot be delivered goes to a dead-letter trail.

The books are allowed to lag the business by up to a minute. That is the price of atomicity, and we pay it on purpose: a minute is short enough that nobody notices, and long enough that a Postgres restart passes between two drains without anyone knowing.

When should an event go on the bus, and when in the outbox?

We also have an in-process domain event bus. It is in memory and fire-and-forget: a handler that throws is logged and dropped, and a process that dies between commit and delivery loses the event without a trace.

The rule we wrote for it:

Publish on the bus only when losing the event would be a nuisance. When losing it would be wrong, it belongs in the outbox.

“Refresh a cached last purchase price” can be stale until the next receipt. “Add this receipt to the supplier’s running total” is permanently wrong if lost, and nothing would ever notice. A bus event fired in the middle of a transaction is less safe than the direct call it replaced, because the direct call at least rolls back with everything else.

What did we get wrong along the way?

• A green build does not mean the worker boots. NestJS resolves dependency injection at runtime. Registering a job processor without its whole service chain passes typecheck, tests and every guard, and then crash-loops on deploy. We boot the worker locally before shipping any new processor.
• Zero downtime is near, not absolute. Two API instances are taken out of rotation and restarted one at a time. That covers a deploy. It does not cover a schema change that the old instance cannot read, so rollback is planned per layer: nginx, process, schema and data do not roll back together.
• The 09:00 slowdown was a caching problem, not a message-broker problem. Each punch used to cost six or seven Mongo reads. A version-stamped cache key, rotated by the eleven routes that change punch state, took it to one read, and the punch write went from three operations to one upsert that treats a duplicate-key error as success.

Checklist: building attendance that works at the edge

• Decide which checks the device may run alone (face match) and which need the server (PIN, enrolment).
• Derive the punch id from the punch itself, so a retry is a duplicate, and treat a duplicate as success.
• Send sentAt with occurredAt and compute the age on the server. Never trust a device wall clock.
• Encrypt biometric templates at rest on shared devices, and keep secrets like PIN hashes off them entirely.
• Store on-device verification as evidence. Never let it gate pay.
• Return 202 for device ingest, and resolve the tenant from the device registry.
• Run payroll on classified days, not on raw punches, and record why each day was classified the way it was.
• Put the ledger where constraints are enforced, and feed it through an outbox, not a bus.

If you run a factory, a warehouse or a field team in India and want to see this in production, OpsX is at opsx.in. If you are building something similar, I am happy to compare notes in the comments, especially on the parts we are still unsure about.