Note to Self-Hosters: Private Key Rotation in 6.68.0

Developer help Self-hosting
1

(See also: 🚨 Heads up! If you have integrations that validate members, the jwks has changed!)

Summary

Tomorrow’s weekly release (v6.68.0) contains an automatic private key rotation mechanism for ghost_private_key (used to sign staff jwt tokens for external services i.e. ActivityPub) and members_private_key (used to sign members jwt tokens for external services).

Admin/Content API Keys, staff/member sessions, and magic links are not affected by this change. If you aren’t validating member identities as part of an external service and are using the Ghost-hosted ActivityPub service (ap.ghost.org) - then this change does not impact you and you can upgrade as normal.

If you are running self-hosted ActivityPub and/or using member identity tokens for connection to external services, then read the rest of the post to make sure you’re prepared for the key rotation.

TL;DR

  • Ensure you’ve updated self-hosted ActivityPub installs to at least v1.2.13 (though you should regularly update to ensure you pick up all bugfixes and security patches)
  • Ensure that any external code reading the jwks.json endpoints is configured to verify JWTs using the kid token header to match the right JWKS key (most spec-compliant libraries do this by default)

More Info

Ghost serves the ‘jwks’ information for the ghost and members keys at /ghost/.well-known/jwks.json and /members/.well-known/jwks.json, respectively. Following the JWKS specification, each route returns an array of keys, identifiable via kid. JWTs are signed by one of the available keys, and the kid is added to the JWT header. This is designed such that systems providing a public JWKS list can perform token rotation as needed, and external systems following the spec correctly will continue to validate JWTs successfully before and after the rotation is complete.

Because Ghost’s JWKS endpoints are served with a 24h Cache-Control header, the built-in rotation mechanism operates on a 48h cycle, roughly:

  1. Generate new key
  2. Wait 24h
  3. Start signing new JWTs with the new key
  4. Wait another 24h
  5. Remove the old key from jwks.json

If your code is not matching tokens by kid header (most JWT/JWS standard libraries do this by default), or if your implementation is making the assumption that the jwks.json list never changes, then you will need to make fixes to those systems before upgrading to 6.68.0. Also, if you have a caching layer in front of Ghost that caches the jwks.json data for longer than the 24h set by Cache-Control, you should clear that cache after you update to 6.68.0.

4 Likes