SEQN Auth Key Rotation Runbook

This runbook covers rotation for SEQN Auth credentials, including SMTP deployment credentials. Payment-provider flows remain out of scope.

Credential inventory

CredentialPrefix or envOwnerRotation impact
Publishable project keypk_live_App/admin consoleBrowser-safe; rotate only by creating or recreating projects.
Project secret keysk_live_App backendBackend integrations fail until consumers receive the new key.
Webhook signing secretwhsec_Webhook receiverReceivers fail signature validation until they accept the new secret.
OIDC client secretOIDC_CLIENT_SECRETSilver Auth API/AuthentikNew login callbacks can fail if API and Authentik disagree.
Authentik API tokenAUTHENTIK_API_TOKENSilver Auth API/AuthentikProject provisioning is blocked while invalid.
Session secretSESSION_SECRETSilver Auth APIExisting app sessions and pending-login cookies are invalidated.
Application secret hash secretAPPLICATION_SECRET_HASH_SECRETSilver Auth APIExisting sk_live_ verification fails unless the value is preserved or project secrets are reissued.
Invitation token hash secretINVITATION_TOKEN_HASH_SECRETSilver Auth APIOutstanding invite links fail unless the value is preserved or links are reissued.
Webhook secret hash/encryption secretsWEBHOOK_SECRET_HASH_SECRET, WEBHOOK_SECRET_ENCRYPTION_SECRETSilver Auth APIExisting webhook secret metadata cannot be verified/decrypted unless values are preserved or webhook secrets are reissued.
Google OAuth secret hash/encryption secretsGOOGLE_OAUTH_SECRET_HASH_SECRET, GOOGLE_OAUTH_SECRET_ENCRYPTION_SECRETSilver Auth APIStored customer-owned Google OAuth secret metadata cannot be verified/decrypted unless values are preserved or credentials are re-saved.
SMTP credentialsSMTP_USERNAME, SMTP_PASSWORD, SMTP_FROM_EMAILVPS ops / email providerInvitation email delivery fails until the new credentials are saved and the API is recreated.
Database passwordsAUTHENTIK_PG_PASS, SILVER_POSTGRES_PASSWORDVPS opsRequires coordinated database and service restart.
Backup encryption keyOFFSITE_BACKUP_AGE_RECIPIENT or OFFSITE_BACKUP_GPG_RECIPIENTVPS opsOld backups require the old private key until retention expires.

Standard safety rules

  • Take a fresh local backup before high-blast-radius rotations.
  • Keep a dated incident or maintenance note with actor, reason, scope, and rollback plan.
  • Never paste raw secrets into docs, alerts, screenshots, support tickets, or chat.
  • Prefer secret stores and one-time terminals over local files.
  • Rotate canaries first when possible.

Project secret key rotation

Use for exposed sk_live_ keys, environment moves, or scheduled hygiene.

  1. Identify the project and backend consumers.
  2. Prepare the receiving app secret store for the new value.
  3. In the hosted console, rotate the project secret.
  4. Copy the one-time sk_live_ value directly into the backend secret store.
  5. Restart or redeploy the backend consumers.
  6. Run GET /v1/backend/application from the backend environment.
  7. Confirm no new invalid_secret_key errors.

Current rotation replaces the valid secret immediately, so schedule production rotations when consumers can be updated at the same time.

Webhook signing secret rotation

Use for exposed whsec_ values or scheduled receiver hygiene.

  1. Confirm the receiver can accept a new secret.
  2. If the receiver supports dual secrets, add a pending new secret slot.
  3. Rotate the webhook endpoint secret in the hosted console.
  4. Copy the one-time whsec_ value into the receiver secret store.
  5. Send a test webhook event.
  6. Confirm delivery logs show successful signatures.
  7. Remove the old receiver secret after the grace window.

If the receiver cannot accept dual secrets, schedule a short maintenance window because outgoing signatures switch immediately.

OIDC client secret rotation

  1. Take a fresh local backup.
  2. Create or rotate the client secret on the Authentik OAuth2 provider for silver-auth.
  3. Update the VPS .env value for OIDC_CLIENT_SECRET.
  4. Recreate only the silver-auth-api container.
  5. Start a fresh login from an incognito browser.
  6. Confirm /auth/callback completes and /v1/me returns the signed-in user.

Active sessions should continue unless SESSION_SECRET also changes.

Authentik API token rotation

  1. Create a replacement Authentik API token with the same minimum provisioning permissions.
  2. Update AUTHENTIK_API_TOKEN in the VPS .env.
  3. Recreate only the silver-auth-api container.
  4. Create or update a canary project.
  5. Confirm provisioning status is synced.
  6. Revoke the old Authentik token.

Session secret rotation

SESSION_SECRET protects opaque session token hashes/cookies and signed pending-login cookies. Rotation signs users out and invalidates in-progress login handoffs, but it should not invalidate project secret keys, invitation links, webhook secret metadata, or stored Google OAuth credential metadata when the dedicated secret-domain env vars are preserved.

  1. Announce a maintenance window.
  2. Take a fresh local backup.
  3. Replace SESSION_SECRET with at least 32 random characters.
  4. Recreate silver-auth-api.
  5. Confirm anonymous users redirect to login and new sessions work.
  6. Monitor support channels for forced sign-in reports.

Secret-domain env rotation

Use this when rotating the dedicated hash/encryption env vars. Prefer preserving the old value until you can reissue the affected customer-facing secret; rotating these values without reissue is intentionally disruptive.

  1. Take a fresh local backup.
  2. Identify the affected domain: project secrets, invitation links, webhooks, or Google OAuth credentials.
  3. Replace only the matching *_SECRET env var.
  4. Recreate silver-auth-api.
  5. Reissue the affected project secret, invitation link, webhook signing secret, or Google OAuth credential.
  6. Run the matching smoke: backend key health, invitation acceptance, webhook test delivery, or Google OAuth credential status.

Database password rotation

  1. Take a fresh local and offsite backup.
  2. Stop write traffic if possible.
  3. Rotate the Postgres role password inside the target database.
  4. Update the matching .env variable.
  5. Recreate dependent containers.
  6. Confirm /healthz, /v1/config, login, and admin console access.

SMTP credential rotation

  1. Create the replacement SMTP credential or app password in the email provider.
  2. Update SMTP_USERNAME, SMTP_PASSWORD, and sender values in the VPS .env.
  3. Recreate only the silver-auth-api container.
  4. Create a canary invitation and confirm the email delivery status is sent.
  5. Revoke the old SMTP credential.

Backup encryption key rotation

  1. Add the new public recipient to .env.
  2. Run backup-offsite.sh.
  3. Confirm the new archive can be decrypted by the new private key.
  4. Keep the old private key until all old encrypted backups pass retention.
  5. Remove the old private key only after restore coverage is no longer needed.