Operate it

One service and one database. This page is what to know before it is carrying real traffic.

Configuration

Everything is configuration, and every setting has an environment variable: Onbe:Section:Key becomes Onbe__Section__Key.

Setting	Required	Default	What it is
Onbe:Issuer	yes	—	This control plane’s own issuer URL
Onbe:Database:Provider	no	postgres	postgres, or sqlite for the embedded database
Onbe:Database:ConnectionString	postgres	—	Postgres. A secret
Onbe:Database:MigrationConnectionString	no	the above	Postgres only: a role allowed to change the schema. A secret
Onbe:Database:Path	sqlite	—	Where the embedded database file lives
Onbe:Signing:Keys:0:Path or :Pem	yes	—	The token signing key. A secret
Onbe:Signing:Keys:N:Kid	no	a thumbprint	An explicit key id; omit for the RFC 7638 thumbprint
Onbe:Signing:ActiveKid	no	the only key	Which key signs new tokens. Required once more than one is configured
Onbe:Admin:ApiKey	no	—	Enables the admin API. At least 32 characters. A secret
Onbe:UpstreamIdp:Issuer	yes*	—	Your identity provider’s realm URL; discovery is derived from it
Onbe:UpstreamIdp:MetadataUrl	yes*	—	The discovery document instead, for a provider shaped differently
Onbe:UpstreamIdp:Audience	yes	—	The audience subject tokens must carry
Onbe:UpstreamIdp:SponsorCheck:UsersUrl	yes	—	Where to ask whether a user is still active
Onbe:UpstreamIdp:SponsorCheck:TokenUrl	yes	—	Where the control plane gets its own token to ask
Onbe:UpstreamIdp:SponsorCheck:ClientId	yes	—	The client it asks as
Onbe:UpstreamIdp:SponsorCheck:CacheTtl	no	00:00:30	Never longer than DefaultTokenTtl
Onbe:Tokens:DefaultTaskTtl	no	00:30:00	Task lifetime when a registration does not override it
Onbe:Tokens:DefaultTokenTtl	no	00:05:00	Token lifetime, likewise. Never exceeds the task default
Onbe:Agents:MinTaskTtl	no	PT1M	Shortest max_task_ttl a registration may set
Onbe:Agents:MaxTaskTtl	no	P1D	Longest max_task_ttl a registration may set
Onbe:Agents:MinTokenTtl	no	PT30S	Shortest max_token_ttl a registration may set
Onbe:Agents:MaxTokenTtl	no	PT1H	Longest max_token_ttl a registration may set
Onbe:Tasks:SweepInterval	no	1 minute	How often expired tasks are marked terminal
Onbe:Tasks:SweepBatchSize	no	20	Tasks expired per transaction
Onbe:Audit:Chains	no	16	How many hash chains the ledger is. 1 to 256
Onbe:Audit:Aggregation:Enabled	no	true	Summarise denials that name nobody, per reason per window
Onbe:Audit:Aggregation:Window	no	PT1M	The window. PT1S to PT15M
Onbe:Audit:Sink:Url	no	—	Where to copy audit records
Onbe:Audit:Sink:BearerToken	no	—	Sent to the sink. Requires an HTTPS sink. A secret
Onbe:Audit:DrainInterval	no	5 seconds	How often queued records are posted
Onbe:Audit:DrainBatchSize	no	100	Records per request to the sink
Onbe:RateLimit:Enabled	no	true	The per-source token bucket
Onbe:RateLimit:PermitsPerMinute	no	600	Sustained requests from one source, per instance
Onbe:RateLimit:Burst	no	120	Most permits the bucket holds. Never below one second of the rate
Onbe:RateLimit:TrustedProxies	no	—	Comma-separated CIDR networks whose X-Forwarded-For is believed

* Exactly one of Onbe:UpstreamIdp:Issuer and Onbe:UpstreamIdp:MetadataUrl. Setting both is a startup failure rather than a quiet precedence rule.

A setting the chosen database provider does not use is also a startup failure, not a value silently ignored: a leftover connection string usually means somebody believes a different database is in use.

Durations in configuration take either form — Onbe:Tokens:DefaultTokenTtl accepts 00:05:00 and PT5M alike. An agent registration’s durations are ISO 8601, because that is what the API contract says. A duration the loader cannot parse is a startup failure naming the key, and years and months are not durations a TTL can use.

Misconfiguration is a startup failure with the key named, not a warning. A token TTL longer than the task TTL, a sponsor cache longer than a token’s life, a bearer token for a plaintext sink — all refuse to start. Configuration has every setting with its bounds.

Signing keys

The key signs every task token. Tool servers verify against the JWKS at /.well-known/jwks.json, so rotating it is visible to everyone who validates locally.

Onbe.Server keys generate --out /run/secrets/onbe/active.pem

That reads no configuration at all — no database, no identity provider — so it works before any deployment exists. It writes an owner-readable PKCS#8 PEM, refuses to overwrite a file that is already there, prints the key id and the setting that configures it, and prints nothing private. The key id defaults to the RFC 7638 thumbprint; --kid <name> chooses it instead, and the command then also prints the Onbe__Signing__Keys__0__Kid setting that makes the server use it.

A key made some other way works too: a P-256 private key in PKCS#8 (BEGIN PRIVATE KEY) or SEC 1 (BEGIN EC PRIVATE KEY) form, given as a path to a mounted secret or inline as a PEM. Tokens are signed ES256.

Outside the Development environment, a missing key is a startup failure. A key is never generated silently: a signing key that appeared on its own is a key nobody is minding, and the first anyone would know is when it changed again on the next restart. In Development an ephemeral key is generated and logged loudly.

To see what is loaded:

Onbe.Server keys

Output

{"keys":[{"kty":"EC","crv":"P-256","use":"sig","alg":"ES256","kid":"9Fy-qRMxKnbHaNBF9nwF68GQVxzhgz3miPA15FHHv38","x":"…","y":"…"}]}
Active kid: 9Fy-qRMxKnbHaNBF9nwF68GQVxzhgz3miPA15FHHv38

Rotation is three deploys, and the command prints them:

Onbe.Server keys rotate --out /run/secrets/onbe/next.pem

It loads the currently configured keys exactly as the server would, writes a new key beside them, and prints the rollout with the settings for each step. It changes nothing else.

1. Publish the new key without signing with it. Add its entry and set Onbe:Signing:ActiveKid to the key that should still be signing — the old one. With more than one key configured ActiveKid is not optional, and leaving it unset is a startup failure rather than a guess about which key signs.
2. Start signing with the new key. Move ActiveKid, and deploy. Give verifiers time to refresh their cached JWKS between the two; one that has not seen the new key will reject tokens signed by it.
3. Stop publishing the retired key, not before the longest token lifetime has passed. rotate computes that wait from the largest max_token_ttl of any registered agent, disabled ones included, and says so.

Waiting longer is always safe. Removing the key early is not: every token it signed fails verification the moment it stops being published.

A key cannot be recovered. If the file is lost, every token it signed is unverifiable: run keys generate, restart, and anything holding a token from the lost key exchanges again. If the key is disclosed, treat every token it signed as compromised: generate a new key, make it active at once, remove the old one without waiting out step 3, and revoke the affected tasks. Keys backed by a KMS, where the private key never exists on disk, are not implemented; the signing key is a file today.

Migrations

Onbe.Server migrate

Output

Applied 9 migration(s):
  - 20260910060829_InitialSchema
  - 20260910162631_AddAssertionReplays
  - 20260911081817_AddRevocationExpiry
  - 20260911124255_AddOutboxNextAttempt
  - 20260911132757_AddAuditQueryIndexes
  - 20260912131056_AddAgentJwks
  - 20260912132211_WidenAuditReason
  - 20260913223129_AddAuditEventCount
  - 20260915165924_AddAuditEventChain
Schema version: 20260915165924_AddAuditEventChain

Migrations are never applied at startup. Applying them is an explicit command, run by somebody who decided to run it. A rolling deploy that migrates on boot is a race between instances and a schema change nobody approved.

Give the command a MigrationConnectionString for a role that may change the schema, and let the running service connect as a role that may not. Each provider keeps its own migrations, because the two schemas are not identical underneath — Postgres stores scope lists as arrays and the embedded provider stores them as JSON text — while the control plane above them is one implementation.

The embedded database

Onbe:Database:Provider=sqlite and a Path is the same control plane with the same rules, and it gives up four operational things: one instance writes at a time, so a second node is correct but useless; the file must be on local disk, never a network filesystem; there is no failover, so losing the host loses everything not backed up; and a backup is SQLite’s own backup or VACUUM INTO, not a copy of the file from under a running server. Moving to Postgres is a fresh database and a re-registration of agents — there is no migration of one provider’s data into the other, and the ledger does not travel.

The expiry sweeper

A background sweeper marks expired tasks terminal and revokes their grants, writing one task.expired record per task. It runs every SweepInterval and handles SweepBatchSize tasks per transaction, repeating until a pass comes back short.

The batch is small on purpose: each one holds a chain’s append lock for the length of one append, and a large batch would hold it long enough to matter.

Expiry is not revocation and is never recorded as one.

Verifying the ledger

Onbe.Server audit-verify

Output

Audit ledger intact: 621 record(s) verified across 3 chain(s), heads 609:2b8d0e4c…,617:9c1f6a30…,621:484fa8a9….

Exit codes: 0 every chain intact, 3 a chain broken, 1 the ledger could not be read, 2 a malformed head argument.

Run it on a schedule, and keep the heads somewhere the database cannot reach. Pass the comma-separated list of heads from the previous run back — one per chain — and it proves each chain still reaches its head unchanged, which is the only way to detect a truncated tail:

Onbe.Server audit-verify 609:2b8d0e4c…,617:9c1f6a30…,621:484fa8a9…

The audit ledger explains why that is the part that matters.

Secrets

Five things are secrets and none of them may ever be logged, returned in an error response, or committed: the database connection string, the migration connection string, the signing key, the admin API key and the audit sink’s bearer token. The options types that hold them print a fixed placeholder instead of their contents, so they cannot leak through a log line that formatted an object.

The same applies at runtime to task tokens, task grants and subject tokens.

Running more than one

Instances are interchangeable. Audit appends take one advisory lock per chain, so any number of instances write the same fixed set of chains without queueing on each other; the sink outbox is drained with a skip-locked read, so no record is delivered twice by design. Agent registrations are read from storage per request and never cached, so a change through the admin API takes effect on the next request everywhere, not after a restart.

Two things are per instance: the rate limiter’s bucket (n replicas admit up to n times the configured rate) and denial counts (n instances write up to n summaries per reason per window).

Keycloak

Two things about a realm fail late and read badly. The realm’s advertised issuer must equal the URL the control plane fetches it from — Keycloak takes it from its frontend URL, not the address it was reached on, so fix the frontend URL rather than pointing Onbe:UpstreamIdp:Issuer somewhere else. And a Keycloak access token carries the client itself and account in aud by default, so the human-facing client needs an audience mapper emitting Onbe:UpstreamIdp:Audience or every exchange fails. Onbe.Server doctor names both in one line each.

Kubernetes

A tagged release publishes a signed container image and a Helm chart, both signed keylessly with cosign and the image with its SBOM attached, so an install verifies the signature and pulls the chart rather than building its own. The chart requires four values — the issuer, the identity provider’s realm URL, and the names of the Secrets holding the connection string and the signing key — and creates no Secret and no key. It runs migrate as a pre-install and pre-upgrade hook Job, refuses /admin and /audit on the default ingress, and helm test runs doctor. v0.1 is not tagged yet, so neither the image nor the chart is published.

Sizing

Measured on one modest rig running the control plane, Postgres and Keycloak together, so read it as a floor rather than a certification: idle is about 8 millicores and 110 MB; an exchange costs 4–6 CPU-ms and about 2 KB of storage, 1.37 KB of it ledger and permanent; introspection is a third of that and no ledger row; and 200m of CPU with 512 MiB covers ten exchanges a second, database included. Memory does not scale with rate — what moves it is being pushed past the ceiling with the limiter off, which is why the limiter stays on. Re-run the numbers on your own hardware before planning around them.

A checklist before real traffic

• Onbe:Admin:ApiKey set, at least 32 characters, and not the one from the quickstart.
• A signing key from a secret store, not a file next to the binary.
• ASPNETCORE_ENVIRONMENT not Development, so a missing key fails instead of inventing one.
• Migrations run by a separate role, deliberately.
• high_risk_audiences filled in for everything you would not want reached five minutes after a kill switch. Tokens for those audiences then carry introspect_required and both server SDKs act on it; a tool server written against the raw HTTP contract has to read the claim itself.
• Onbe.Server doctor run from where the control plane runs. It checks the configuration, the signing key, the identity provider, the database and the admin API, and exits non-zero if any of them is wrong. A realm whose issuer is not the URL you reach it on, and a default audience that is not yours, are the two failures that otherwise surface late and read badly.
• Onbe:RateLimit:PermitsPerMinute raised to your fleet — 600 a minute is ten a second per source, and a fleet behind one egress is one source. Onbe:RateLimit:TrustedProxies set if anything fronts the service, or every caller shares one bucket. Do not switch the limiter off.
• Onbe:Audit:Aggregation:Enabled left on.
• audit-verify on a schedule, with the heads stored off this database.
• Log shipping that does not swallow decision: "deny".