Skip to content

Managing the key-encryption key (KEK)

The server encrypts the secrets it keeps in the database with a key-encryption key (KEK) that it never stores there. This page covers how to generate, store, back up and rotate the KEK, how to keep it in a KMS instead, and what to do if it is lost or leaks. The configuration reference lists every setting.

The key hierarchy: the key-encryption key, kept where key_manager says and never in the database, wraps a data-encryption key made for each secret, which encrypts the secret; the database holds the key ID, the wrapped data-encryption key, and the encrypted secret in seven sealed columns. The list below describes each layer.

  • Secrets stored in the database, such as actors’ private keys, are encrypted with AES-256-GCM using a data-encryption key (DEK) generated for each secret. The DEK is wrapped with a key-encryption key (KEK) and stored next to the ciphertext (envelope encryption).
  • The KEK is not stored in the database, so a leaked database dump alone does not expose the secrets.
  • Each ciphertext carries the key ID of the KEK that wrapped its DEK. With key_manager = local, the key ID is local: followed by 16 hexadecimal digits derived from the KEK, so the same KEK always gives the same ID. You never assign IDs.
  • A ciphertext is bound to the row ID and column name where it is stored. Copying it to another row or column makes it undecryptable.
  • The sealed columns, named as my kek status prints them, are local_account_private_keys.private_key_enc (actors’ private keys), totp_enrollments.secret_enc (two-factor authentication secrets), login_provider_secrets.sealed_secret (client secrets of sign-in providers), webhook_secrets.sealed_secret (webhook signing secrets), instance_secrets.sealed_secret (the media proxy signing key and the key of the idempotency fingerprints), tls_acme_accounts.sealed_private_key (the keys of the ACME accounts when the server gets its own certificates), and tls_certificates.sealed_private_key (the keys of those certificates).
  • The state of a passkey sign-in or a sign-in through an external provider is sealed the same way but not stored: the browser holds it for at most 30 minutes.
  • The signing key (HMAC-SHA256) for media proxy URLs is a random key that the server generates on its first start and stores in the database, encrypted like the other secrets. It does not change when you rotate the KEK, so media proxy URLs keep working across rotations, and my kek rewrap re-wraps it with the other secrets.
  • key_manager is local (the default: it uses the kek setting), gcpkms (the KEK stays in Google Cloud KMS; see Google Cloud KMS), awskms (the KEK stays in AWS KMS; see AWS KMS), or plugin (the KEK stays with a program of yours; see A plugin).

Without the KEK, restoring the database does not let you decrypt actors’ private keys. Once a private key is lost, other servers can no longer verify that actor’s signatures, and the actor’s identity in the fediverse is broken (followers’ relationships cannot be carried over). Treat the KEK with the same care as the database.

What is lost with the KEK: after a restore without it, everything that is not sealed is still readable, every sealed value fails to open, and the actors lose their identity. The paragraph below says the same.

What is not sealed stays readable: posts, accounts, follows, and the rest of the database, and the media files in storage. Every sealed value fails with an unknown key-encryption key error: the actors’ private keys, two-factor authentication secrets, client secrets of sign-in providers, webhook signing secrets, the media proxy signing key, and the keys of the certificates the server got itself. The same happens when another KEK is set in place of the one that wrapped the secrets, or when the KMS key or the plugin’s key that wrapped them is disabled or destroyed.

Encode 32 random bytes with standard base64.

Terminal window
openssl rand -base64 32

Where openssl is not available, head -c 32 /dev/urandom | base64 works too. Do not leave the generated value in your terminal history or in tickets.

The KEK is a secret setting, so it can be passed only through the environment variable MY_KEK, or through MY_KEK_FILE, the path of a file holding the value. Putting it in the configuration file (TOML) prevents startup.

Target Example of how to deliver it
Self-hosted (binary / systemd) Put it in a file only the owner can read (chmod 600) and point MY_KEK_FILE at it
Docker / Compose Mount a Docker secret at /run/secrets/my_kek and set MY_KEK_FILE=/run/secrets/my_kek. The files of deploy/docker do this from secrets/kek (Docker Compose files): to rotate, replace that file and set MY_KEK_PREVIOUS in .env
Kubernetes / microk8s Mount a Secret as a file and point MY_KEK_FILE at it
Cloud Run Expose a Secret Manager secret as a file or an environment variable
AWS ECS Pass it from Secrets Manager or SSM Parameter Store (SecureString)
  • Give every process, such as web and worker, the same KEK.
  • The KEK never appears in logs or error messages (only the key ID does).
  • my config check validates the KEK’s format (base64 of 32 bytes) and prints the value as (redacted). In the production profile it also fails on a KEK that is not random, such as all zeros or a repeated byte, and my serve logs a warning about one (see the production checklist).

Back up the database and the KEK, separately.

  • Keep the KEK somewhere other than where the database backups live (a password manager, offline media, a secret store in another account, and so on). If both are in one place, a leak of that place makes the database encryption pointless.
  • During a rotation, include the kek_previous keys in the backup for as long as ciphertext wrapped with them remains in the database.
  • Include the KEK in your restore procedure, and test restores regularly (Backup and restore). After restoring a database, set the KEK (and kek_previous) that was in use at backup time to the same values. With a different KEK, anything that reads a secret fails with an unknown key-encryption key error that names the key ID.

A KEK in kek_previous is used only for decryption, and new secrets are always encrypted with kek. So that a rolling deploy, where old and new processes run side by side, never produces ciphertext that some process cannot read, switch in two steps.

A rotation of the local KEK in five steps, with the values of kek and kek_previous after each: the new KEK goes into kek_previous first, then swaps places with the old one, my kek rewrap moves the stored secrets, and the old KEK leaves the configuration. The steps below say the same.

  1. Generate a new KEK and back it up.
  2. Restart every process with MY_KEK=<old KEK> and MY_KEK_PREVIOUS=<new KEK>. Now every process can decrypt with the new KEK.
  3. Restart every process with MY_KEK=<new KEK> and MY_KEK_PREVIOUS=<old KEK>. From now on secrets are wrapped with the new KEK, and ciphertext of the old KEK is still readable.
  4. Re-wrap the stored secrets with the new KEK: run my kek rewrap once, with the same configuration as step 3 (see Re-wrap the stored secrets). When it exits with 0, no secret in the database needs the old KEK any more.
  5. Wait at least 30 minutes after step 3, so that the short-lived sign-in tokens issued with the old KEK have expired, then restart every process with MY_KEK=<new KEK> and no kek_previous. Keep the backup of the old KEK for as long as you keep database backups taken before step 4: restoring one of them needs it again.

kek_previous accepts several comma-separated keys. Listing the same key twice, or listing the same key as kek, prevents startup.

my kek rewrap replaces only the wrapped data-encryption key of each secret and keeps its ciphertext; it never writes or prints a secret.

Terminal window
my kek status # count the stored secrets by the key ID of the KEK that wraps them; decrypts nothing
my kek rewrap # re-wrap every secret that a key of kek_previous wraps with kek
  • my kek status prints one line per column and key ID, marks the key ID of kek with (current), and ends by telling whether any secret still needs a key of kek_previous. A key ID that is neither kek nor in kek_previous means a KEK is missing from the configuration.
  • my kek rewrap reads the secrets in batches of 100 and writes each batch in one short transaction. It prints, per column, how many secrets it re-wrapped and how many were already wrapped with kek. It can run while the server is serving, and it is safe to stop or to run again at any time: it skips the secrets already wrapped with kek, and a secret that the server rewrote in the meantime is left as the server wrote it.
  • A secret that cannot be re-wrapped is left unchanged and reported with its row ID, and the command exits with 1. Either its KEK is not configured (the message names the key ID: add that KEK to kek_previous and run it again), or the stored value is damaged or was copied from another row. Do not remove a KEK from kek_previous while my kek status still counts secrets under it.

With key_manager = gcpkms, a symmetric Cloud KMS key (a CryptoKey with the purpose ENCRYPT_DECRYPT) wraps the data-encryption keys, and the KEK never leaves Cloud KMS. A leaked database dump and a leaked configuration together still do not expose the secrets: decrypting them needs the permission to use the key.

  • Set kms_key to the resource name of the key, without a version: projects/{project}/locations/{location}/keyRings/{ring}/cryptoKeys/{key}. Names longer than 248 bytes are rejected at start. The key ID of the stored secrets is gcpkms: followed by this name.
  • The server authenticates with Application Default Credentials: on Cloud Run, the service account of the service; elsewhere, GOOGLE_APPLICATION_CREDENTIALS.
  • Grant that service account roles/cloudkms.cryptoKeyEncrypterDecrypter on the key (not on the project), for every process, such as web and worker, and for the commands my kek status and my kek rewrap. Without it, reading a secret fails with an error that names the role; a missing or disabled key version fails likewise, and those errors are not retried until the configuration is fixed. A quota error or an unavailable Cloud KMS is temporary, and the work that needed the secret is retried.
  • The server keeps a data-encryption key that Cloud KMS decrypted for kms_dek_cache_ttl (default 10m), so most reads of a secret do not call Cloud KMS.

A rotation in Cloud KMS (by hand, or with a rotation period on the key) adds a new primary version and keeps the old versions enabled. New secrets are wrapped with the new version, the server decrypts each secret with the version that wrapped it, and the key ID does not change, so a rotation needs no my kek rewrap.

Do not disable or destroy an old version while secrets still need it: a secret wrapped with a disabled or destroyed version can no longer be decrypted. Cloud KMS does not tell which secrets a version wrapped, so to retire key material, move the secrets to a new key first:

  1. Create a new key (it may be in the same key ring) and grant the service account the role on it.
  2. Restart every process with kms_key set to the new key and kms_key_previous set to the old one. The old key now only decrypts.
  3. Run my kek rewrap, with the same configuration. When my kek status no longer counts secrets under the old key’s ID, remove kms_key_previous and restart.
  4. Only then disable the old key’s versions, and keep them restorable for as long as you keep database backups taken before step 3: restoring one of them needs them again.

With key_manager = awskms, a symmetric AWS KMS key (key spec SYMMETRIC_DEFAULT, key usage ENCRYPT_DECRYPT, customer managed) wraps the data-encryption keys, and the KEK never leaves AWS KMS. A leaked database dump and a leaked configuration together still do not expose the secrets: decrypting them needs the permission to use the key.

  • Set kms_key to the key’s ARN (arn:aws:kms:{region}:{account}:key/{key-id}), an alias (alias/{name} or its ARN), or the key ID. At start the server resolves the name to the key’s ARN, and the key ID of the stored secrets is awskms: followed by that ARN. Pointing an alias at another key therefore changes nothing until the next start, and then only for new secrets: the secrets already stored keep naming the old key, so set it in kms_key_previous (see Retire an AWS KMS key). AWS managed keys (alias/aws/...) are rejected; only the AWS service that owns one can use it.

  • The server authenticates with the AWS SDK’s default credential chain: on ECS, the role of the task; on EC2, the instance profile; elsewhere, the environment (AWS_ACCESS_KEY_ID and the like) or the shared configuration files. The region is that of the key’s ARN; for an alias or a key ID, set AWS_REGION.

  • Allow kms:Encrypt, kms:Decrypt, and kms:DescribeKey on the key, for every process, such as web and worker, and for the commands my kek status and my kek rewrap. A policy of the IAM role:

    {
    "Effect": "Allow",
    "Action": ["kms:Encrypt", "kms:Decrypt", "kms:DescribeKey"],
    "Resource": "arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
    }

    The resource is the key’s ARN even when kms_key is an alias. The key policy must also let the account’s IAM policies grant access (the default key policy does).

  • The server fails to start when the key does not exist, is disabled or pending deletion, or is not a symmetric encryption key. Later, a missing permission or a disabled key fails the reading of a secret with an error that names the cause, and those errors are not retried until the configuration is fixed. Throttling (ThrottlingException) or an unavailable AWS KMS is temporary, and the work that needed the secret is retried.

  • The encryption context of each call holds the row ID and the column name of the secret ({"my-aad": ...}, in base64url). CloudTrail logs it in plain text; it contains no secret.

  • Cost: AWS charges a monthly fee for each customer managed key (USD 1 at the time of writing, plus the same for each of the first two rotations of its key material) and a fee per request beyond a monthly free tier (see AWS KMS pricing). The server keeps a data-encryption key that AWS KMS decrypted for kms_dek_cache_ttl (default 10m), so a server with ten actors that deliver all the time makes about 43,000 requests a month per process, a few cents beyond the free tier.

Automatic key rotation in AWS KMS keeps the key’s ARN and its earlier key material, so the key ID does not change and a rotation needs no my kek rewrap.

To stop using a key, or to point an alias at another key, move the secrets to the new key first. Do not disable the old key or schedule its deletion while secrets still need it: a secret wrapped with it can no longer be decrypted.

  1. Create the new key and allow the role to use it.
  2. Restart every process with kms_key set to the new key and kms_key_previous set to the old one (its ARN, not the alias). The old key now only decrypts.
  3. Run my kek rewrap, with the same configuration. When my kek status no longer counts secrets under the old key’s ID, remove kms_key_previous and restart.
  4. Only then disable the old key, and keep it for as long as you keep database backups taken before step 3: restoring one of them needs it again.

With key_manager = plugin, a plugin of yours keeps the KEK, in Vault, in an HSM, or wherever it chooses, and wraps the data-encryption keys for the server.

The plugin sits in the same trust boundary as the KEK. It sees every data-encryption key in plaintext, so whoever controls it, or the host it runs on, can read every secret of the server, including the actors’ private keys. Give it the protection you give the KEK. That is also why no setting but the configuration file and the environment can change its address.

Terminal window
MY_KEY_MANAGER=plugin
MY_PLUGIN_KEY_MANAGER_ADDRESS=unix:///run/my/key_manager.sock
  • The address is unix:// or https://. The server refuses http://, on loopback too, since the keys would cross in plaintext. The other settings of the destination are in Plugins.
  • Run my plugin test --kind key_manager <address> before you point the server at a plugin (Test a plugin). It wraps only keys that it makes up and throws away.
  • The key ID of the stored secrets is plugin: followed by the plugin’s ID for its KEK, which is 1 to 248 bytes of visible ASCII.
  • Keep every KEK that a stored secret still names. A rotation in the plugin shows as a new ID in the next wrap. The secrets wrapped before keep naming the old ID, and the plugin must still open them. my kek rewrap moves them to the current ID, and my kek status counts the ones that remain.
  • A plugin and a KMS never serve together: kms_key and kms_key_previous are refused with a plugin, and so is plugin_key_manager_address with key_manager = gcpkms or awskms.

Every process, and every my command that opens the database, such as my kek status, does this before anything else. It is the one kind of plugin whose failure stops a start.

  1. It asks the plugin for the ID of its current KEK.
  2. It runs a self-test with a random key that it throws away: a wrap and an unwrap must give the key back, an unwrap with different authenticated data must answer OUTCOME_DECRYPT, two wraps of the key must differ, and the wrapped value must not hold the key.

A plugin that does not answer, or answers 503, is tried again for up to 30 seconds, for a sidecar that comes up after the server. After that the process exits with an error. A wrong answer, and any other failure such as a refused token, stops it at once: the self-test exists to stop a plugin that ignores the authenticated data or passes keys through, and waiting does not mend one.

The server fails closed. Whatever goes wrong, a refused or timed-out call, or an answer that is empty, unknown, or malformed, the operation that needed the key fails with an error. It never stores a secret in plaintext, never switches to the local KEK or another key, and never treats a failure as a result. A job that fails this way retries later, and a request fails for the person who made it. Each failed call counts in my.plugin.failures by kind and Connect code, and the log holds no key, authenticated data, or wrapped value, nor anything the plugin said.

Wrapping is never cached, and a value sealed a moment ago has nothing in the cache, so these stop at once:

  • Creating an account, which generates the actor’s keys.
  • Setting up two-factor authentication, and registering a passkey or signing in with one, whose ceremonies keep their state sealed.
  • Signing in through an external provider, and saving a provider’s client secret.
  • Registering a webhook, and my kek rewrap.

Reading a stored secret works only while the process holds its data-encryption key from an earlier read, for at most kms_dek_cache_ttl (default 10m). After that, or for a secret it has not read yet, these stop too:

  • Federation delivery and signed fetches, which need the actors’ private keys. Delivery catches up through the retries of its jobs, once the plugin is back.
  • Checking a two-factor code at sign-in.
  • Webhook delivery.

A restart stops as described above, so do not restart the server to mend an outage of the plugin: a server that is up keeps serving what its cache covers, and one that restarts serves nothing until the plugin answers. What needs no secret, such as reading timelines and posting, goes on. The media proxy’s signing key is read at start and stays in memory. A renewed client certificate or CA file needs no restart either: the server reads the files again once they change (Plugins).

The RPCs are those of my.plugin.key_manager.v1.KeyManagerService, with a default deadline of 5 seconds and messages of at most 128 KiB each way.

POST /my.plugin.key_manager.v1.KeyManagerService/Describe
{}
{"outcome":"OUTCOME_OK","keyId":"transit/my-kek-3"}
POST /my.plugin.key_manager.v1.KeyManagerService/WrapKey
{"dek":"<base64>","aad":"<base64>"}
{"outcome":"OUTCOME_OK","keyId":"transit/my-kek-3","wrapped":"<base64>"}
POST /my.plugin.key_manager.v1.KeyManagerService/UnwrapKey
{"keyId":"transit/my-kek-3","wrapped":"<base64>","aad":"<base64>"}
{"outcome":"OUTCOME_OK","dek":"<base64>"}
  • keyId is the plugin’s own ID, without plugin:. UnwrapKey may name an ID older than the current one.
  • wrapped is 1 to 65,535 bytes and must not hold the key. Two wraps of a key differ.
  • aad is authenticated data that holds row IDs and column names, and UnwrapKey must fail unless it is what WrapKey got. That is what stops a secret from being copied to another row.
  • Never log dek, aad, or wrapped.
outcome Meaning
OUTCOME_OK The call succeeded
OUTCOME_UNKNOWN_KEY UnwrapKey: the plugin does not hold the KEK of keyId
OUTCOME_DECRYPT UnwrapKey: wrapped or aad does not authenticate

Answer an HTTP or Connect error when you cannot answer, such as when the key store is out of reach. Do not answer OUTCOME_DECRYPT or OUTCOME_UNKNOWN_KEY then: to the server, they say that the value itself is bad. The server does not retry a call itself, so a plugin can get the same call again from a job.

A key ID starts with the kind of key manager that wrapped it (local:, gcpkms:, awskms:, plugin:). The server wraps new secrets with the current key_manager and unwraps each stored secret with the key manager its key ID names, so a move between the local KEK and a KMS is a rotation across kinds.

  • With a KMS, kek and kek_previous only decrypt: they open the secrets still wrapped with the local KEK, and every new secret is wrapped with kms_key. kms_key_previous lists earlier keys of the same KMS, which only decrypt too.
  • With key_manager = local, kms_key and kms_key_previous, when set, only decrypt: they open the secrets still wrapped with the KMS.
  • From the local KEK to a KMS:
    1. Create the key and grant the permissions as in Google Cloud KMS or AWS KMS.
    2. Restart every process with key_manager = gcpkms or awskms, kms_key set to the key, and the current KEK moved from kek to kek_previous. Secrets are now wrapped with the KMS, and the ones wrapped with the local KEK still open.
    3. Run my kek rewrap with the same configuration.
    4. When my kek status no longer counts secrets under a local: key ID, remove kek_previous and restart. Keep the backup of the KEK for as long as you keep database backups taken before step 3.
  • From a KMS back to the local KEK, do the same the other way round: set key_manager = local and a new kek, and keep kms_key set, which now only decrypts, until my kek rewrap has moved every secret and my kek status no longer counts secrets under a gcpkms: or awskms: key ID. The keys in kms_key and kms_key_previous must then all belong to the same KMS.
  • To move from one cloud’s KMS to another’s, move to the local KEK first and then to the other KMS; there is no direct move.
  • A plugin moves the same way, through the local KEK alone:
    • To it, restart every process with key_manager = plugin, plugin_key_manager_address, and the current KEK moved from kek to kek_previous, then run my kek rewrap. When my kek status no longer counts secrets under a local: key ID, remove kek_previous and restart.
    • From it, restart every process with key_manager = local, a new kek, and plugin_key_manager_address kept, which now only decrypts, then run my kek rewrap. When my kek status no longer counts secrets under a plugin: key ID, remove the address and restart. Until then the plugin must still pass the checks at start, which include a wrap.
  • The server keeps a data-encryption key that the KMS or plugin decrypted for kms_dek_cache_ttl (default 10m, at most kms_dek_cache_entries, default 1024), so a secret in use reaches it about once per period and process. 0 in either setting disables the cache.

If only the KEK leaked, rotate it as above. If both the KEK and the database may have leaked, rotation does not protect you: the actors’ private keys in the database are already readable, so the actor keys themselves must be regenerated.

Web Push is signed with the server’s VAPID key (P-256, RFC 8292). It is a secret separate from the KEK and is not stored in the database.

  • Create a key pair with my config vapid-key, and give every process the private key (the value of vapid_private_key = ...) through MY_VAPID_PRIVATE_KEY or MY_VAPID_PRIVATE_KEY_FILE. The server derives the public key from the private key and hands it to clients, so you do not configure it.
  • If you do not set it, Web Push is disabled and clients do not offer it. In the production profile, my config check and the start of my serve warn about that.
  • Browsers bind a subscription to the public key, so changing the key stops delivery to existing subscriptions. When a user turns push notifications on again in settings, the client subscribes again with the new key. Do not change the key unless it leaked; if it leaks, switch to a new key and ask users to turn notifications on again.

For LLMs and coding agents:this page as Markdown, llms.txt,llms-full.txt, llms-small.txt

Analytics on this site