Mainline

Five services were already in production — form capture, signed delivery, routing, dedup, variant exposure. Each shipped on its own, each with its own docs and its own test suite. Building a console to trace a lead across all five turned up something more interesting than the console: they could not talk to each other.

One service's webhook destinations could not send an Authorization header the next one required, and could not emit the envelope it demanded. One hop later, two services signed and verified mutually incompatible HMAC schemes. No correlation column existed anywhere, so "which delivery came from this submission?" had no answer at any layer. Each service had independently reinvented idempotency, signing and error shapes — five implementations of three problems, none interoperable.

Nothing was broken. Every service passed its own tests and served its own users. They had simply never been asked to work together, which is a different kind of failure than an outage: it does not page anyone and it does not fix itself.

The order was the design decision.

The console shipped first, against the broken version, with two explicitly named bridge classes standing in for the impossible handoffs and a comment on each saying it was to be deleted. Unlisted, noindex, no portfolio link.

That was not caution. It was so the conformance probe could record a red baseline in production before the contract existed — all five services failing trace.http.echo, measured against deployed code, with a timestamp. Without that baseline there is nothing to move from and the migration is a claim. With it, the dashboard went 5 failing → 4 → 3 → 2 → 1 → 0, one service at a time.

The contract — Interchange, published to Packagist.

W3C Trace Context for tracing, Standard Webhooks for signing, RFC 9457 problem+json for errors, IETF Idempotency-Key semantics for idempotency. Adopted standards rather than an invented one, so the signature verifier that came out of it works against Svix, Clerk and Resend too.

Verifying those specs against the published documents rather than from memory was made a gate of its own, and it caught four errors before any code shipped. The costly one: Standard Webhooks secrets are base64 with a whsec_ prefix and the HMAC key is the decoded bytes. Getting that wrong produces signatures no compliant verifier accepts, and it looks like a configuration problem for as long as you let it.

Three rules shaped the package more than the code did. Additive, never replacement — every service keeps its original scheme documented and working, and the contract is an additional negotiated mode selected per source and per destination, so there is no flag day anywhere in the program. Exact version pins, one service at a time — a shared package across six services is a single point of failure, so no caret, no tilde, and a release is never tagged and adopted in the same change. Prove it at the queue boundary first — every hop here crosses a queue, so HTTP-only propagation would have produced five disconnected root spans. The gate was a throwaway app proving context survives HTTP → dispatch → worker → outbound HTTP as one trace, run on the database queue driver with the ambient scope cleared between dispatch and execution. On the sync driver the same test passes vacuously.

Zero downtime by construction, not by care.

Every default preserves current behaviour, so each adoption deploy is a no-op on the wire and all of the risk lives in a separate per-source cutover on a different day. New columns nullable, so a rollback can leave them. New scheme fields defaulting to the service's existing scheme.

The risk that actually matters in a signing cutover is not that a bad signature gets accepted. It is that a good lead is silently rejected because sender and receiver disagreed for ten minutes — and a dropped enquiry is indistinguishable from no enquiry. Nobody files a ticket. So every inbound surface gained a dual-accept window, tested in both directions including the rollback direction, with a persistent per-source, per-scheme counter in the database because log retention on the host is 30 days with a size cap that can truncate sooner. Inferring "no legacy traffic" from logs that may have rotated is not evidence.

The rollback that breaks what it was meant to save.

Code rolls back with a symlink swap and a php-fpm restart, and migrations only add nullable columns. That covers code and schema, and it is the part everyone writes down. It does not cover configuration — and configuration is where the risk was deliberately put. A destination switched to the new scheme, with code reverted to a release that does not implement it, means the rollback is clean, the deploy is green, and the data plane is down. So every scheme change records its previous value at the moment of change and the rollback reverts both as one step.

The before/after is structural, not cosmetic.

Same scenario, same five services, same six hops — trace depth 1 versus 5. Before, every hop was a child of the root span because the console made every call, so the waterfall renders flat. After, each hop parents to the one that called it, so it nests. Depth is computed by walking stored parent/child links, not read off a label: a flat fan-out is depth 1 whatever it is called. Both bridges are deleted; the console now makes exactly one call, the form submission, and discovers the rest from the services' own records.

What the migrations found.

Adopting a contract meant exercising code paths that had never been exercised, which turned up more than a dozen production defects in services that were reporting healthy. A semantic dedup path that had never once worked behind four stacked defects, each converting a failure into a success. A service with no queue worker configured at all, ever, whose own infrastructure directory held another service's supervisor config copied verbatim — zero deliveries across every submission it had ever received, while ingest ran synchronously and every submission returned 200. An API key pepper correctly set in .env and resolving to an empty string in production because config:cache runs on every deploy. CI that ran a single test suite while an entire feature directory had never executed. And a load harness that drove six copies of one payload into a service that deduplicates identical submissions — measuring the idempotency rather than the throughput.

The common shape is not carelessness. Every one of them produced a success signal — a 200, a green check, a restarted service, an "n/a" — and a success signal is where nobody looks.

What it proves.

Same person read five production codebases and found the interoperability failure, designed and published the contract, migrated all five services onto it in production without a flag day or a lost lead, built the console that makes the difference legible, and wrote the case study that ends on what is still unfinished rather than on the good numbers. The case for hiring me to unify services that were never designed to be unified — on live systems, with real traffic, one service at a time.