# Architecture Guide Operating contract for agents working on FairyStack-managed servers. The system model diagram on the Architecture page is the human overview; this file is the rules. ## Topology - Each customer has one control box (runs FairyStack) and one app box (runs all of that customer's applications). Administrators can prepare unassigned spare pairs within the same provisioning allowance, with a target of two. New invitations reserve an existing pair and prepare its running services, dedicated AuthReturn app, HTTPS and release enrollment before a claim code is issued. Claiming atomically binds the signed account to this ready runtime; it never adds machines or runs an installer. The durable runtime owner receipt supplies the current owner for authorization and allocation views. Public trial invitations skip mailbox confirmation. Old issued codes retain their recovery path and fixed assignment. - Each application is one Git repository under `/home/ubuntu/apps/` on the app box, described by its `fairystack.json`. - Agent sessions execute on the app box. Nothing runs against the control box filesystem. - App hostnames are `..`; the label is the first label of the control box's public origin. ## Request path ```text browser → DNS / TLS → nginx virtual host → frontend/* served as static files → /api/* and explicitly proxied routes → systemd service on PORT → /home/app_/data and read-only secret mounts → shared services over bounded network APIs ``` Nginx is the source of truth for public reachability. A 200 on the app port proves nothing about the public origin. ## Ownership | Concern | Owner | Verify by | |---|---|---| | App identity, hostname, port | `fairystack.json` and FairyStack inventory | manifest matches the deployed app | | Public pages and assets | nginx, serving `frontend/` | request the public origin | | API routes | app process; nginx proxies `/api/*` and listed routes only | exercise the public path | | Process and port | `/etc/systemd/system/app_.service` | read `PORT`; service active | | Routing and TLS | `/etc/nginx/sites-enabled/.conf` | `nginx -t`, reload, public smoke test | | Persistent data | `/home/app_/data/` (hyphens become underscores) | files owned by `app_` | | Secrets | SOPS store → app-owned `0600` copy → read-only bind mount | never hand-edit `/opt/secrets/` | | Shared services | the provider's public API and live agent guide | timeout, validate, test the boundary | ## Routing rules - A public static route needs a file under `frontend/`; a Flask route alone is private unless nginx lists it. - Route, navigation, nginx ownership, and public verification ship as one change. - Prefer static pages over bespoke nginx locations. If a route must be server rendered, add and test the explicit proxy. ## Before changing an app 1. Read `AGENTS.md`, `fairystack.json`, git status and remotes, and the service unit. Pull when a remote exists. 2. Read the nginx virtual host before deciding which layer owns a URL. 3. Search for existing routes, state models, and shared-service clients before adding another. 4. Fetch the live guide of every shared service the change touches. 5. Preserve unrelated working-tree changes and data ownership. ## Bound app-box execution A control box has a private execution-target registry containing its paired app box. Every visible conversation is bound to exactly one durable execution on that target. The control conversation creates or resumes that execution through an authenticated internal API; the binding carries a target ID, mutation ID, and self-contained objective. Later tasks stay in that conversation. Three separate concepts: - A service directory discovers a capability and may identify its hosting box. - The private target registry says whether this controller is authorized to execute work on that box. - The first execution creates the auditable target binding; later work resumes it. Jessald's control box is paired with Jessald App Box. Customers use the same fixed two-box route. ## Lifecycle operations The control box owns the create, rename, delete, restore, and restart recipes. It executes them on its paired app box over authenticated SSH. The app box runs a read-only observer (loopback port 9443, reached through the control box's tunnel) for inventory, health, source, deployment, and per-commit line-count history evidence; the control box installs it from its own release on every deploy, and it exposes no FairyStack mutation API. The console's current-app line-count sparkline reads that history for every app, FairyStack included. Use FairyStack's API for lifecycle operations. Never hand-build partial app infrastructure. ## Deploying an application Deploy runs on the app box because the unit, nginx host, and certificate must be activated where they are served. 1. Run `fairystack-app-deploy `. It is the only sudo entry point and execs root-owned `/usr/local/sbin/fairystack-deploy-app`. The rename and restore recipes call the same tool. 2. The tool reads `fairystack.json`, pins the committed Git revision as a release, reconciles secrets and mounts, installs `app_.service`, writes the nginx host for `public_host`, obtains TLS with an HTTP-01 challenge on port 80, and verifies health and the public origin. Receipt step names: `manifest`, `secrets`, `runtime`, `service_activation`, `tls_activation`, `health`, `public_verification`. 3. Failure restores the previous unit and nginx snapshot and names the failed step. Data is never rolled back. 4. Receipts live under `/var/lib/fairystack-app-deploy//` and are served by the observer to the control box. Both tools ship with every release (`upgrade_deploy_tools`) and at provisioning. App boxes admit public 80 and 443; control boxes do not. ## Verification checklist 1. Syntax checks and relevant regression tests. 2. Bump `VERSION`, commit, push when a remote exists. 3. `fairystack-app-deploy `. 4. Local port answers within an explicit timeout. 5. Every touched URL answers on the public origin with the expected status, content type, and body. 6. When local and public disagree, read the service and nginx logs. Local 200 plus public 404 is a routing defect. ## Agent runtime Work outside the fixed lifecycle runs through the agent runtime on the app box (Codex App Server, or Claude Code through the FairyStack bridge). A control-box service reuses the visible session API: it may add a provenance-marked system input to a session or create a target-bound session. There is no separate remote-shell catalog. ## Standing rules - Services talk only over network APIs; never import another service's code or read its database. - Read ports from configuration and `PORT`; never guess. - Run data-writing scripts as the app user. - Every network call has a timeout; every asynchronous workflow has a terminal deadline and a visible outcome. - Fix shared infrastructure defects at their owning layer, not with a consumer workaround. App lifecycle APIs and the full pair contract: . ## FairyStack release timings Open **Timings** (also linked from Architecture) for measured publisher, local control-box, and marketing activation timings, grouped by commit. Refresh reads the latest durable receipts; it does not start or retry a deploy. The newest publisher attempt for a selected commit is shown. Failures, unavailable publisher evidence, and receipts without detailed instrumentation stay visible. The publisher measures frontend tests, backend tests, preparation checks, queue wait, signing and delivery, receiver waits, public verification, and marketing activation. Its total starts after the isolated checkout is created; initial Git fetch and checkout allocation are outside the recorded interval. Customer activation precedes marketing activation. Receiver steps include preparation, Python dependencies, app-box runtime updates, integrations, executor capacity, cache warming, transport configuration, and web verification. Receiver timings are nested inside publisher waits, so their totals must not be added together. Marketing activation has its own step breakdown from the app box’s local receiver receipt. These are deployment elapsed times, **not bridge outage measurements**. - Read API: `GET /api/admin/release-profiles` (platform administrator). - Publisher receipts: `/home/ubuntu/.local/state/fairystack-release-profiles/`, one directory per attempt, read over the paired observer's `GET /v1/releases/profiles`; no control-box filesystem access to app-box data. - Receiver receipts: `/var/lib/fairystack-deploy/.json`. - `timings` records step name, start/end timestamps, monotonic elapsed milliseconds, and outcome. Terminal writes close the current step. Historical receipts without `timing_schema: 1` expose totals only. A missing terminal receipt beyond the observation deadline is shown as stalled with an unverified outcome; a timeout of the viewer's read does not cancel a release. ## Browser page readiness Timings also shows the latest 20 navigations recorded in this browser's local storage. The elapsed clock starts at Navigation Timing's time origin, before DNS, connection, and HTML delivery. Milestones cover configuration, authentication, app-box resolution, composer reveal, session discovery, snapshot rendering, selected-session refresh, and the existing selected-session paint boundary. Network timings overlap these milestones. This is browser-observed elapsed time, not a server-only latency measurement or a reconstruction of earlier visits. A sign-in/setup requirement, failed restoration, navigation away, or 60-second observation deadline is recorded as a distinct terminal outcome; none counts as ready. The observation deadline does not cancel authentication or session work. After verified sign-in, the latest local history is backfilled to owner-scoped `/api/browser-telemetry`; subsequent terminal records synchronize under a five-second deadline. Failed uploads remain local for bounded retry. The server retains the latest 200 records per owner for 30 days, with only allowlisted routes, milestones, network timings and release revision. Timings' Page readiness tab shows upload status and saved records; `&navigation=` opens one immutable observation. Incomplete same-tab predecessors have no fabricated completion duration. Pre-authentication failures require a later successful login to synchronize. ## Connection-preserving app deployments An existing single-service app can opt in with `"deployment":{"rolling":{"port":9341,"drain_seconds":1230}}`; `port` must be its already allocated upstream port, not a new allocation. Its entrypoint must honor `FAIRYSTACK_APP_SOCKET` by binding that Unix socket and allowing nginx to connect, and `/api/version` must return the exact `FAIRYSTACK_APP_REVISION`. Use the installed `fairystack-app-deploy ` and allow it to finish its bounded drain; short synchronous HTTP restart callers are not suitable for long drains. The deployer clones the installed service contract into a revision-pinned process, warms and verifies it, then reloads nginx with the new socket. Existing WebSocket and HTTP connections stay on their original process. New connections use the new release. The receipt and active pointer retain the current socket, service, previous generations and drain deadline for interrupted-deployment recovery. Only after all old upstream connections have closed does it retire the old process and update the established service alias. The candidate is enabled before routing changes so the route remains bootable if the deploy owner disappears. A never-ready candidate cannot replace the current route. A drain timeout fails visibly and restores previous routing without killing active streams on either process. If retirement cleanup fails after draining, the verified new route stays active. Data is never rolled back. Old and new versions overlap on the app's own store, so migrations must remain compatible throughout the drain. Do not enable this mode for destructive schema changes or entrypoints that ignore the socket. Validation includes isolated real nginx/WebSocket cutovers, exact-revision and Unix-entrypoint probes, readiness failure, stuck drains, rollback, and recovery of an interrupted deployment with multiple surviving generations. ## Agent browser inspection The paired app box may load its control box's pages, assets, and guides using its existing owner-bound `X-FairyStack-Agent-Token`. The nginx ingress exception applies to the whole host, retaining the IP deny for requests without a valid capability. API authorization remains in place. Browser automation must attach this credential only to the exact control origin, never to external resources. ## Frequent-release availability Release frequency must not control serving-process lifetime. Prepare immutable candidates while the current route remains active, bound concurrent warming to available CPU and memory, verify readiness, then switch new traffic and drain old owners. Managed executor imports admit at most two candidates before loading their store. Standby workers validate the local lifecycle credential and database; they load only the selected harness after claiming a turn. Their existing supervisor deadline still applies. An unavailable import slot or failed candidate leaves the previous owner running. The release queue keeps one activation owner and one newest pending tested commit. Superseded requests retain an explicit terminal receipt. Long-running turns keep their existing worker until completion; standby generations must not accumulate behind them. Old and new code must agree on stored data and API contracts during this overlap. Destructive migrations require a separate compatibility transition; reverting code never implies reverting operator data. Serving needs resource headroom for both generations and deployment work. A warm socket switch alone cannot protect a server whose CPU, memory, disk, or database is saturated by candidate preparation. Keep builds and tests on the release source box, constrain candidate startup on the serving box, and measure request continuity during preparation as well as cutover. An eventually healthy new revision is not evidence that the previous revision remained responsive.