Anatomy of a Salesforce–ERP sync

An architecture deep-dive: event-driven triggers, an idempotency trap measured in dollars, two Salesforce gotchas that swallow errors, and what the rebuild changed. The pattern is real; the specifics are not: the client is an international nonprofit, and identifiers, tools, dates, and amounts have been changed, rounded, or removed.

We maintain an integration that moves donation data from Salesforce (NPSP) into a cloud ERP for an international nonprofit. It was originally a vendor-built black box — the vendor declined to hand over source or documentation, so the team reverse-engineered it from data structures and system configuration alone and rebuilt it on a low-code automation engine. Not long after, it's being rebuilt again on a code-first orchestration platform — much faster this time — because the knowledge now lives in the repo as ADRs, tests, and runbooks rather than in one person's head.

Between those two rebuilds sits a production incident that doubled receipts in the general ledger while every system reported success. We told that story as a business case study; this post is the architecture tour — how the sync works, exactly where it broke, and what the redesign does differently.

The trigger chain: a field flip is the API

There is no scheduler and no Run button. The unit of work is a batch — all the payments in one bank payout. An operator (or an auto-run rule) sets a status picklist to Sync, and that field change is the entire control surface:

View full

Transport latency is milliseconds; semantics are batch. The workflow probes the ERP — "is there already a receipt for this payout number?" — creates if not, updates if so, then writes back: every payment gets stamped with the receipt's record number, and the batch is flipped to Synced last.

Two properties of this chain deserve attention, because together they caused the incident:

1. The pub/sub bus is at-least-once, and there was no queue, no dedup, and no dead-letter path anywhere. If the subscriber acknowledges slowly, the bus re-sends the same event tens of seconds later.
2. "Probe, then create-or-update" is only idempotent if the update is idempotent. This one's update path appended the full line set to the existing receipt instead of replacing it.

The failure, measured

A multi-week backlog (manual batch rules with no owner, calendar, or alert simply weren't run) was pushed through in one afternoon: several hundred batches in a few hours. Under that burst the engine's acknowledgements lagged past the redelivery threshold, and those batches produced roughly half again as many executions — about a quarter of the payouts were written two to four times.

The damage took two shapes, distinguished only by when the redelivered event's probe ran:

Net: well over a hundred damaged receipts, a mid-six-figure inflation on a mid-seven-figure run of legitimate volume — with zero API errors. Every create and every additive update returned success. The pipeline verified that calls didn't fail; nothing verified the result was right.

The team also measured the load envelope afterward, which we'd recommend to anyone running an event-driven sync — know your brackets before the backlog day:

View full

The counterintuitive headline: dollar value is irrelevant to risk. A later, larger wave processed without incident; the storm broke on a smaller run. Concurrent volume is the variable.

Salesforce gotcha #1: the composite API swallows failure

The write-back uses Salesforce's composite API with allOrNone=false. Its failure mode is nasty: the response is an HTTP 200 even when every row fails, with per-record errors in a positionally-aligned array inside the body. The transport layer can't raise; if your workflow step merely returns that body, the orchestrator renders a green step over a fully failed write.

{
  "compositeResponse": [
    {
      "httpStatusCode": 400,
      "body": [
        {
          "errorCode": "INSUFFICIENT_ACCESS_OR_READONLY",
          "message": "insufficient access rights on object id"
        }
      ],
      "referenceId": "payment_0"
    }
    // …one entry per record, positionally aligned with your request.
    // The outer response? 200 OK. Parse the body or fly blind.
  ]
}

Worse, the original flow stamped the batch Synced — a terminal state — before stamping the payments. One partial failure left an unrecoverable zombie: receipt posted in the ERP, payments unstamped in Salesforce, and no retry able to reach them because the batch already claimed completion.

Two rules fell out of this, and we'd carve both into any integration:

• Parse the body, raise on row errors. An HTTP status is not a result.
• Terminal state lands last. A mid-run crash must leave a re-runnable batch, never a falsely finished one.

Salesforce gotcha #2: the sharing wall

The integration identity had object-level Edit on the payment object — and still couldn't write. Payments are ControlledByParent under Opportunities with a Private org-wide default, and without Modify All Records the Edit permission is unreachable on records the identity doesn't own. Production had many distinct Opportunity owners, including an external vendor's staff.

The diagnostic that settles it: query UserRecordAccess — HasReadAccess: true / HasEditAccess: false is the signature. But you must run that query as the integration identity; an admin's query always says true. And resist the temptation to "fix" a failing test by reassigning record ownership to the integration user — that hides the production defect instead of surfacing it.

What the rebuild changes

The migration isn't a lift-and-shift; it's a point-by-point answer to the failure analysis:

The verification strategy is the part worth stealing. Three tiers: unit tests (CI-enforced), parity replay — run captured production executions through the new code touching no systems, and diff the payload it would send against what the legacy system actually sent — and a live sandbox tier that materializes captured production rows as real records, fires the deployed flow as the least-privilege identity, and diffs the resulting ERP record against the same production golden. The first live-tier run caught bugs a green parity run had shipped.

Capacity got proven the same way: a stress ladder run well beyond the production ceiling, with every write-back landing over dozens of composite chunks, zero errors, and the batch stamped Synced last.

The takeaway

Every one of these defects was individually survivable. At-least-once delivery is fine if writes are idempotent. An additive update is fine if events never duplicate. Manual batch rules are fine if someone owns the calendar. Silent composite errors are fine if something downstream reconciles amounts. The incident happened where the ifs intersected — and the fix, in every single case, was moving an assumption out of habit and into an artifact: a queue policy, a concurrency key, a hard refusal in code, a test against a production golden.

Prefer the story to the schematics? Read the case study. Want the pocket version? The seven hard lessons are in dgtl Bytes. And if you're staring at your own tangle of point-to-point syncs, this is precisely the architecture posture the Integration Spine™ formalizes.

We write about Salesforce, finance systems, and the unglamorous plumbing between them. If you're designing or rescuing a CRM-to-ERP sync, our inbox is open.