Attaching the first cell
loom cell attach turns a prepared first cell (loom cell prepare, mode recording,
directory row attaching) into a serving one: it places every org the cell holds in the
router’s directory, raises the cell’s mode to enforced and creates its org triggers. This
page is the operator’s procedure; the design is
cells.
Before attach
Run attach at low load, as one run at a time, and not during a rollout, a provider
transition, a logical backup, a long erasure or a migration: each holds a transaction or a
snapshot long enough for step 4’s wait to run out of --wait. Steps 2 and 4 run index skip
scans on the Postgres primary (each statement holds vacuum’s horizon while it runs) and
read the org column of every ClickHouse table, which competes with serving load.
Every item below is a precondition attach checks, or one it cannot check and relies on:
- Every router replica runs this release; every process and every CLI and CI image of the cell runs it too (upgrades, steps 16 and 17).
LOOM_CELLis set on every process of the cell. Every CLI and CI configuration that writes for an org hasLOOM_CELLset withoutLOOM_CELL_ROUTER_TOKEN: inrecordingit then refuses only new orgs, so nothing has to change at the mode commit. With the token removed, revoke theINSERTonloom.tenancyand theUPDATE (since)onloom.tenancy_modethe CI role held for recording new orgs.- The ClickHouse writer credential is rotated after that rollout (upgrades, step 18).
- No fingerprinted setting has changed since prepare, and none changes until attach ends:
a change gates every process against
loom.cell_self. LOOM_SERVES_TRAFFIC=falseis set only on pods no load balancer routes to: attach exempts them from the router check.- The cell’s processes and the attach CLI share one
LOOM_INSTANCE_HEARTBEAT_SECONDS: attach judges which processes are live by its own value. LOOM_PG_DSNpoints straight at Postgres, with no pooler of any mode, for the servers and for attach: attach refuses a session that moves between backends or loses the tenancy marker, and refuses a server whose heartbeat reportsmarkedfalse.- With SCIM on, its tokens belong to this cell and every active account, current
refresh session and personal service-account creator graph satisfies the
provisioning precondition.
Provision missing identities, or explicitly cut off their existing tokens with
--revoke-unprovisioned. - On a ClickHouse cluster of several hosts, attach’s user exists on every host, and the
cluster has an interserver
<secret>or aremote_serversuser equal to attach’s. One replica that cannot sync blocks attach (SYSTEM SYNC REPLICA … ON CLUSTER): an availability cost, not a safety one. - Only Loom’s own roles and superusers hold the rights attach’s tenancy-privilege check
counts (inserting a tenancy row, changing the mode, creating in a schema, triggers on
loomtables, owning them or their trigger functions,session_replication_role,ALTER SYSTEMofloom.tenancy_checked,pg_write_server_files,pg_execute_server_program), and no role that holds one, or owns what it counts, owns aSECURITY DEFINERfunction another role can execute that does any of that. Attach checks the catalog; it cannot read a function’s body for intent, or a role that runs code as the server’s OS user. postgresql.conf,postgresql.auto.confand any pendingALTER SYSTEMentry, on the primary and on every standby, set neitherloom.tenancy_checkednorsession_replication_role. Attach sees a live entry on the primary (its unmarked session reads it), never a standby’s files.- Postgres has a synchronous standby attached (
synchronous_standby_namesset) andmax_prepared_transactions = 0; the database encoding is UTF8. - No org longer than 2048 bytes is in the cell’s stores (below).
--dry-runlists any.
Attach’s Postgres role needs the privileges of pg_read_all_stats
without SET ROLE (grant its membership with INHERIT TRUE), USAGE on schema
loom, SELECT on loom.cell_first, loom.tenancy_mode, loom.cell_self,
loom.authority and every org table, UPDATE on loom.tenancy_mode, INSERT on
loom.tenancy, SELECT, INSERT, UPDATE and DELETE on loom.cell_probe, TEMPORARY
on the database, EXECUTE on pg_logical_emit_message, and effective privileges of the owner
role of loom.tenancy and every guarded table without SET ROLE (grant their memberships
with INHERIT TRUE). Its ClickHouse user needs the grants the writer
check names (SELECT on system.* and on the configured databases, SHOW USERS,
SHOW ROLES, SHOW ROW POLICIES, SHOW TABLES, SHOW COLUMNS,
SYSTEM FLUSH ASYNC INSERT QUEUE, and on a cluster SYSTEM SYNC REPLICA, CLUSTER and
READ ON REMOTE), no settings profile or quota capping the rows or bytes it reads, and the
insert into loom._migration_lock that writes the probe nonce. The CLI needs write access
to the bucket’s probe object and LOOM_ROUTER_OPERATOR_TOKEN.
Run it
loom cell attach --cell <cell> --dry-run \
--ch-allow-user <servers' ClickHouse user> --pg-allow-role <servers' Postgres role>
loom cell attach --cell <cell> \
--ch-allow-user <servers' ClickHouse user> --pg-allow-role <servers' Postgres role>
--dry-run runs the refusals and step 2’s scan, prints the count and a sample of the orgs
per source table, the ones it would skip, and whether the scan fit in half of
--scan-budget. Cell state, tenancy triggers and placements stay unchanged; normal
participant enrollment and retirement are still recorded. The router’s own checks at
attach/begin (the probe, the hosts, the upstream) do not run. Name every ClickHouse user,
user directory, object and replica that may write the configured databases with --ch-allow-user,
--ch-allow-user-directory, --ch-allow-object and --ch-allow-replica, and every
Postgres role besides attach’s own that may make an org held with --pg-allow-role; each
list is audited by the router. --wait (default 600 s) is the wall time the waits share;
every lock try runs at least once, with a lock wait of at least 100 ms, so --wait 0 runs
one try of each wait with no retry. The final heartbeat checks get at least two heartbeats
plus the router timeout, with one extension for a process first seen booting; this floor
is not charged to --wait. --scan-budget (default 7200 s) bounds each full scan, and
step 2 must take at most half of it, since step 4 reads the same tables again.
Each Postgres mode or trigger try reserves cleanup time inside its client deadline.
With SCIM on, --scan-budget also bounds each complete provisioning census. It checks
active accounts and their contacts, current refresh families with their stored verified
email, and live personal service accounts with every creator and ancestor. It names an
unprovisioned subject and refuses before serving. --revoke-unprovisioned cuts off that
subject’s existing tokens under SCIM’s revocation prefix; it does not provision a user.
Already accepted cutoffs remain if a later read, audit or subject fails. Rerun the same
command and flag to finish. An outright revocation or a newer cutoff is never narrowed.
--dry-run --revoke-unprovisioned never revokes; command participation is still recorded.
A serving rerun only verifies.
On interruption its owner closes the session before cancelling transaction work.
A close that cannot complete uses transport abort and reports failure; that abort
does not establish that server locks or an unacknowledged commit have disappeared.
Retry the command to resolve an unknown commit through its durable state checks.
A refusal names what to fix. Rerun attach after fixing it: every step is idempotent, and a
rerun resumes by the directory row and the cell’s mode. On a serving row it verifies
that the connected authority, configured cell and bucket match the directory registration,
then requires enforced mode before printing that it is done.
The point of no return
Step 3’s mode commit is one-way. Before it, a refused attach leaves the cell recording,
its tenancy rows and its triggers in place (they pass every row in recording), and the
cell can be backed out (the first-cell branch of
troubleshooting):
loom cell remove serves only a first cell that never left recording. After it, scoped
principals place their orgs live within seconds, every refusal leaves the cell enforced
and attaching, and the cell goes forward: fix the cause and rerun attach. Unsetting
LOOM_CELL then gates every process, an outage, not a rollback. A row with placed > 0
always goes forward.
From enforced on, native minting requires the org to be held by this cell, with or
without --platform-admin. Admin account routes cannot bring in a new org (WRONG_CELL).
An already issued ordinary token scoped to its org places that org on its first request.
A direct call of the router’s operator place route, or a live
placement with the cell’s credential, cannot read the cell’s Postgres and is accepted once
the directory mode is enforced; the cell’s credential is the servers’ alone.
Orgs attach does not place
- An org longer than 2048 bytes refuses step 2, naming it, before any tenancy row is
written. One first found in step 4 is skipped and named. Remove such an org by hand, in
this order: its bucket objects, from the
keys of its rows in
loom.documents,loom.document_blobsandloom.blob_uploads, so no memory content is left outside erasure; then its rows in every table with anorgcolumn of both ClickHouse databases and of Postgres schemaloom, the erasure ledger tables included; then itsloom.tenancyrow, on a session withloom.tenancy_checkedset. - A name no request can carry (not UTF-8, a NUL) found in a data table is skipped and named; its rows stay outside placement.
- An org found only in a log table (
agent_eventsand the tables derived from it, the code search trace) is never placed: a log row keeps the org an unauthenticated caller typed. Once that org is placed in another cell, its log rows here are outside its erasure there.
loom doctor --stray-orgs names the orgs the cell’s stores hold without a tenancy row,
smallest tables first within --stray-orgs-budget (default 60 s), every table the budget
left unchecked and every table it could not read. Remove a stray org’s rows by the same
procedure as an over-long org’s.
After attach
loom doctor runs attach’s point-in-time checks again on every run: it warns on a Postgres
role outside the owner roles, the superusers and its own --pg-allow-role that can make an
org held or lower the mode (cell_tenancy_privilege). Attach exempts the role it runs as,
the operator’s; doctor does not exempt the role it runs as, so a CI or monitoring role
holding such a right is named. It warns too on a role or database setting or a subscription
that passes an unchecked writer (cell_unchecked_writers). A grant or a writer added after
serving passes attach’s checks, which ran once; doctor is how it is found.