Rotating the root of trust — break-glass key rotation for a Joomla licensing system

If you sign your extension’s subscription-validation responses with an RSA key, that private key is the root of trust for every copy you’ve ever shipped. So what do you do when it might have been exfiltrated? You can’t just mint a new one and swap it in — the instant you do, every customer’s extension rejects the new signature and in our case drops to the free tier limiting functionality they paid to subscribe to.

TL:DR – This is how I rotated a possibly-compromised signing key across a live Joomla subscription system without breaking the field, plus the one design idea that made it safe.

What actually rotates

The instinct is to panic about the customer keys. Don’t. In a well-built subscription system the subscription-key string a customer holds is not derived from the signing key — it’s just an identifier the server looks up. The signing key only signs the validation response the server sends back. So a rotation changes exactly one thing: the signature on that response. Customer keys are untouched; nobody re-keys, nobody re-purchases.

That reframes the whole problem. A rotation is two moves, and the order is everything:

1. Ship the new public key to every extension that is a client — it’s baked into the extension build and used to verify responses.
2. Swap the private key on the server — it signs responses.

Two kinds of rotation

Chained rotation: trust without a rebuild

For a routine roll you never want to rebuild and re-ship every extension. The solution is an endorsement chain. Each new key is signed by the key it replaces, and that signature (the “cert”) travels in every response alongside the new public key:

// On the server: endorse a freshly generated key with the current one.
openssl_sign($newPublicPem, $sig, $currentPrivateKey, OPENSSL_ALGO_SHA256);
$cert = base64_encode($sig);

// Every validation response now carries the chain, e.g.
// "signing_chain": [ { "pub": "<newPublicPem>", "cert": "<base64 sig>" }, ... ]

The client still pins only the original public key from its build. On each response it walks the chain outward from that anchor, adding any link whose cert verifies against a key it already trusts, to a fixed point:

$trusted = [$pinnedPublicKey];          // the key baked into this build
do {
    $added = false;
    foreach ($chain as $link) {
        if (in_array($link['pub'], $trusted, true)) {
            continue;
        }
        foreach ($trusted as $t) {
            $ok = openssl_verify(
                $link['pub'],
                base64_decode($link['cert']),
                openssl_pkey_get_public($t),
                OPENSSL_ALGO_SHA256
            );
            if ($ok === 1) { $trusted[] = $link['pub']; $added = true; break; }
        }
    }
} while ($added);

// A response signed by the rotated key now verifies — with no rebuild,
// no re-key, and no action from the customer.

Because the full chain is in every response (and cached with it), an offline client verifies a rotated key the next time it checks in. There’s no “transition window” to manage. Rotate as often as you like; the chain just grows K1 → K2 → K3, anchored at the build-pinned key.

Break-glass: when the key might be stolen

Chained rotation has one fatal assumption: that the current key can be trusted to endorse its successor. If the current key is compromised, that’s exactly what you can’t rely on — an attacker holding the stolen key could endorse a key of their own. So break-glass deliberately breaks the chain: you generate a brand-new anchor (no endorsement, trusted only because it’s pinned in a fresh build), ship that build, and only then swap the server’s private key. The old key isn’t retired — it’s repudiated.

The hard question: how does the server know it’s the authority?

Here’s the part that turned out to be the real design problem. The subscription server also runs the same software a customer could self-host. So when an install asks “am I the genuine authority — the one allowed to issue keys?”, what answers it?

The original answer was a hostname check: if the domain is ours, you’re the authority. That’s spoofable (the host header is attacker-controlled), and it leaked onto a test installation once. The fix ties authority to the one secret nobody else can hold — the private signing key itself:

// Am I the genuine authority? Derive the public key FROM my active
// private signing key, and check it's one the released extensions trust
// (the pinned anchor, or a key that chains back to it).
$details   = openssl_pkey_get_details(openssl_pkey_get_private($activePrivatePem));
$myPublic  = $details['key'];
$amAuthority = isTrusted($myPublic, $trustedSet);   // same walk the clients do

The subtlety: the public key is public — it ships in every build — so matching a stored public key would be forgeable. That’s why the check derives the public half from the private key. RSA guarantees only the true private key derives that public key; you can’t fabricate one without breaking RSA, and you can’t forge a chain endorsement without the predecessor’s private key. The grant is computed fresh every time — there’s no flag or row in a database to copy. A third party can edit their database, self-issue keys, and point validation at themselves all day; none of it produces the private key, so none of it makes them the authority. And it rides through rotation for free, because “the keys the clients trust” is exactly the chain they already walk.

Executing the break-glass through CI

With the design settled, the execution was mechanical. Builds inject the public key from a CI secret, chosen by the dev/prod flag:

# 1. Generate the new keypair on a trusted machine. Private stays there.
openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out signing.key
openssl rsa -in signing.key -pubout -out signing.pub

# 2. Set the new PUBLIC key as the build secret; clear the rotation-tolerance
#    secret (a missing secret = empty = trust only the new key — hard cut).
#    3. Build + release every paid extension (each pins the new public key).
#    4. THEN adopt the new private key on the server, so the abused key dies.

For a genuinely-abused key you don’t wait for full adoption — you hard-cut: build trusts only the new key, swap the private key as soon as the builds publish, and accept that not-yet-updated installs sit on the free tier until they auto-update. That’s the price of killing a compromised key fast, and it’s the right trade.

Gating it with tests so it can’t regress

A trust model is only as good as the next commit that quietly weakens it. So the security properties became tests that gate every release in CI — not just local hooks a developer can skip:

• A standalone proof that the chain/possession logic holds: rotation survives, an unendorsed key is rejected, and a copied public key with no matching private is rejected.
• A static guard that the shipped code still derives authority from the private key and walks the trusted chain — and that the old hostname shortcut stays gone.
• A per-extension static security scan and the official Joomla Extension Directory Checker, both failing the build on any finding.

Those last two double as honest, always-current trust signals on the public listing: every release passes them by construction, so the page can say so without anyone updating it by hand.

Lessons learned

• Separate identity from secrets. Customer keys are identifiers; the signing key is the secret. Rotate the secret without touching identities and the blast radius collapses.
• Tie privilege to possession, not to a name. A hostname, an IP, a flag in a table — all forgeable or copyable. “Can you derive the public key the world trusts?” is not.
• Put the transition in the response, not in a runbook. Carrying the endorsement chain in every response made routine rotation a non-event with no window to babysit.
• Order is the whole game in break-glass. Public key out first, private key swapped last. Reverse it and you take the field down.
• Make the invariant a CI gate. If a security property only lives in your head, the next refactor will quietly break it.

The rotation is done and verified on live traffic: the authority recognises itself by key possession, clients verify the new key through the chain, and the old key is repudiated for good. The thing I’d underline is that none of the safety came from secrecy about how it works — it came from anchoring every decision to a key only the real authority can hold.

See also Joomla front-end starts returning 500 errors - The issue that kicked off this journey.