“We sync in real time with webhooks” is one of the most optimistic sentences in commerce. I have built Express webhook platforms, Lambda consumers, and long-lived cron suites that move orders, inventory, prescriptions-adjacent workflows, and back-office state between Shopify and internal systems. The honest version is: we reconcile continuously under uncertainty.
Webhooks are hints, not truth
A webhook says “something happened.” It does not guarantee:
- exactly-once delivery
- ordered delivery across topics
- that your handler finished
- that Shopify’s state still matches the payload by the time you process it
At-least-once delivery means duplicates. Network retries mean duplicates. Your bug that returns 500 after committing means duplicates you caused yourself.
Idempotency keys, upsert semantics, and “fetch current Shopify state before acting” are not optional polish. They are the product.
Cron is not failure. It is acknowledgment
Teams sometimes treat cron as legacy embarrassment. In reality, cron is how you handle:
- missed webhooks
- bulk reconciliation
- upstream systems that only expose batch interfaces
- business processes that are inherently periodic (reports, aging, cleanup)
I have maintained cron-driven back-office jobs alongside modern webhook platforms. The architecture works when each job has a clear purpose and clear ownership. It fails when cron silently rewrites the same tables webhooks also rewrite without coordination.
Poison messages and parking lots
A bad payload that crashes a consumer will either:
- block the queue (bad)
- get deleted (worse if unlogged)
- retry forever (expensive)
- land in a dead-letter queue with alerting (adult)
Dead-letter strategy is part of design. So is replay tooling. If only one engineer can replay from raw logs, you do not have an operations model.
Auth, verification, and pretending the internet is friendly
Webhook endpoints are public enough to be interesting. Verify signatures. Reject stale timestamps when appropriate. Rate-limit abuse. Do not assume “obscure URL” is security.
Internal cron endpoints are also attack surfaces if network policy is loose.
Observability must answer operational questions
When a merchant says “inventory is wrong,” you need to answer:
- Did we receive the webhook?
- Did we process it?
- Did we write our DB?
- Did we write Shopify?
- Did a cron later overwrite it?
Correlation ids across webhook receipt → handler → downstream API calls turn mysteries into timelines.
What I design for now
- Handlers that are safe to run twice
- Reconciliation jobs that assume drift exists
- Alerts on DLQ depth and processing lag
- Runbooks for replay
- Explicit ownership: which system wins when states disagree?
Real-time is a marketing word. Reliable reconciliation is an engineering practice.