The Open Finance collection queue, in production
context
Celcoin is a Brazilian Banking-as-a-Service fintech. I designed its Open Finance data cache, an internal product for the fintechs built on top of it.
The bank side has a regulatory call budget per month. So a fintech's screen never calls the bank. Only a scheduled collector does. It works from a worklist: one row per consent, resource and month, each with the time it is due.
In October the collector was in production. And it was falling behind.
problem
630 thousand rows were overdue. The oldest was about 100 hours late. Workers died every night with the Node heap out of memory.
The obvious move was more workers. I measured first. CPU sat at 1 to 3 percent. The pod was idle 70 percent of the time. Demand was 13 times capacity at the current cadence. The collector was not short of compute. It was waiting on the wrong things.
decisions
One bottleneck at a time. Each one measured before deciding.
1. Taking the next job took 5.8 seconds
To claim one row, the query scanned 532,185 index keys. Fifty claims per cycle. Months parked by the quota had a null timestamp, and they sat at the front of the index.
Chose: a dedicated nextAttemptAt field with its own index. Rolled out in expand/contract (two phases, both versions compatible), with a backfill of 1.55 million rows and an ADR.
Rejected: patching the existing query. It broke five ops scripts and only bought about two and a half days.
2. Workers died every night
The midnight purge loaded 1.66 million documents into memory at once. And a BullMQ job with a fixed id does not guarantee a single run when the job (6.5 h) outlasts its own interval (6 h).
Chose: purge in batches and a single runner. Review also stopped an S3 socket timeout that never armed, before merge.
3. Retrying products the bank does not offer
94 percent of failures were a 404 for a product the bank simply does not offer. Retrying them burned 45 percent of the claims per hour.
Chose: a "not offered" class that parks the row until the month turns. Three 404s in a row, not two, because a second retry can land inside a gateway hiccup.
4. One slow bank held everyone
A mass timeout at one institution dropped collection from 8,813 to 4,097 per hour. A call waiting for its timeout held the tick for every bank.
Chose: a continuous pool with 8 lanes instead of fixed ticks. Each bank gets at most 2 of 4 slots, and a circuit breaker of its own with a 5 minute window.
Rejected: a 60 second window. Five 30 second timeouts on two slots take 75 seconds, so it would never open on the real case. Cap and breaker shipped off by default, until the per-bank metric was read in production.
5. One thing I did not ship
A merge request would have blocked revoked consents before serving. It also blocked 116 live consents to catch 4. I cancelled it. Instead, the cache checks consent status before serving: an indexed lookup, measured at 3 ms.
mechanism
what changed
| before | after | |
|---|---|---|
| next-job lookup | 5,832 ms, 532,185 keys | 1 ms, 1 key |
| collection, concurrency 1 to 4 | 2,151 / h | ~7,970 / h (3.7x), zero 429s |
| out-of-memory restarts | every night | 0, three nights running |
| worker memory | 346 to 440 Mi | 147 to 252 Mi |
| overdue rows | 630k, oldest ~100 h | 286k draining ~48k / h, oldest 3.2 h |
| claims spent on 404s | 4,052 / h (45%) | ~32 / h (estimated) |
All measured in production, except the line marked estimated.
my part
The diagnosis, the decisions and the review were mine. Coding agents wrote most of the diffs, under my review, with mutation testing as the bar. That workflow is its own case: a multi-agent harness with rules in code.