luccas@portfolio: less
luccas@portfolio:~$ less cases/order-to-invoice.md

Salon-management multi-tenant SaaS · Cyrex Enterprise · backend and platform engineer, 2 years · Go, AWS, OpenTofu

Order to ERP invoice, without losing an event

+ webhook acked fast, processed from a queue
+ idempotency by the destination's state
+ disabled webhooks re-enabled every 30 min
+ rotating token survives deploys

context

A salon-management multi-tenant SaaS, with salons in Belgium and Luxembourg and real production.

Salons sell products online through a commerce platform. Every order has to become an invoice in the salon's accounting ERP. Each salon is a cost unit there, with its own VAT rules.

problem

Three things fail on their own schedule.

  • The ERP can be down when an order arrives.
  • The commerce platform disables a webhook without telling anyone. Orders just stop arriving.
  • The ERP's OAuth refresh token rotates on every refresh. Lose the new one and invoicing stops until someone re-authorises by hand.

decisions

Chose: eventual consistency. The invoice lands seconds after the order, not inside the webhook request. In exchange, no event is lost.

Rejected: invoicing inside the webhook. That makes us hostage to the provider's timeout and to the ERP being up at that exact moment.

Chose: idempotency by the destination's state. Before creating, the worker asks the ERP: is there already an invoice for this order?

Rejected: a "processed" flag in our own database. That is a dual write, two stores with no shared transaction. A crash between them and they disagree.

One honest limit. There is also a lock on the order's metadata. It is a read then a write, not atomic. Best effort, not exclusion. The ERP lookup is what makes it safe.

mechanism

Order to invoice: webhook acknowledged fast, queue, Go worker, ERP lookup before create, dead-letter queue, webhook monitor and token refresher running beside the flow commerce platform order webhook handler verify, enqueue, 200 before any work SQS queue dead-letter queue with an alarm order worker Go Lambda ERP: invoice for this order? idempotency by the destination yes skip no create the invoice webhook monitor every 30 min token refresher CAS on DynamoDB
The webhook only enqueues. The worker asks the ERP before it writes.
  • The handler verifies the webhook, publishes jobs to SQS and returns 200 before any processing.
  • A Go Lambda creates the invoice and credits or reverses loyalty points. A partial batch failure returns only the failed message. Every queue has a dead-letter queue with an alarm, so a poison message never disappears in silence.
  • A webhook monitor runs every 30 minutes. It lists the webhooks on the platform and re-enables the inactive ones, up to three attempts.

the rotating token

Rotation is a compare-and-swap on DynamoDB, conditional on the refresh token being the one we expect. Two refreshers cannot break the chain.

The refresher has a circuit breaker, confirmed across ticks before it opens. It separates a terminal refusal, where a person must re-authorise, from an ambiguous answer (a 5xx, or an HTML page). The ambiguous one gets a grace period and a re-read, because the ERP may have rotated on its side.

Then the cases only production found:

  • A deferred write used the caller's context. SIGTERM cancelled it, and a freshly rotated token was lost on shutdown.
  • An ECS rolling update left two processes writing the same token. The deploy became stop-then-start.
  • A DynamoDB TTL was deleting the token row.

what changed

  • An order becomes an invoice with no manual step.
  • An ERP outage delays invoices. It does not lose them.
  • A webhook the platform disables comes back within 30 minutes.
  • A deploy or a shutdown no longer loses the ERP token.

my part

The first version of the order flow was a teammate's. I evolved the webhook handler and rewrote most of the ERP integration. The token refresher, the webhook monitor and the alarms are mine end to end.

cases/order-to-invoice.md (3/4) :q home · ls cases/ · ask the chat