Operations
What running Trellaris looks like day to day: how a run behaves and fails, what you can see afterwards, where the money goes, and what an administrator actually operates.
Runs
A run is one execution of one agent. It is the unit of status, streaming, artifacts, cost, and audit — and it is governed by a strict state machine. Only declared transitions are permitted, and every status write goes through that gate, so a run cannot end up in an impossible state.
| Status | Meaning |
|---|---|
queued | Created, waiting for a worker. |
running | Held by a worker under a heartbeat lease. |
awaiting_approval | Parked on a write. Loop state is saved; a decision resumes it. |
awaiting_input | Parked on a question to the user, or on missing connector consent. |
paused | A background run between checkpoints — resumable. |
succeeded | Finished with an answer. |
failed | Finished with a reason: a tool failure, a budget stop, a lost worker. |
canceled | Stopped deliberately — a kill switch, a suspension, an approval timeout. |
Failure is explicit
- No orphans. A worker that dies stops renewing its lease; a reaper moves the run to a terminal state with a reason. Background runs resume from their last checkpoint first, and only fail after the resume budget is spent.
- Tools don't crash runs. A failing tool returns a structured error the model can read, recover from, or explain.
- Approvals expire rather than parking forever.
- Version mismatches fail loudly. Resuming a checkpointed graph onto incompatible code is a controlled failure, never a silent behaviour change.
Seeing what happened
| Surface | Gives you |
|---|---|
| Event stream | Live tokens, tool calls, status changes, and artifacts for a run — and the same sequence replayed afterwards. |
| Run replay | A normalized step-by-step timeline: what was called, in what order, with what outcome. Content is shown post-redaction and annotated with which rules were applied. |
| Artifacts | Every file a run produced, downloadable, with retention applied. |
| Usage ledger | One row per run: model, tokens, and computed cost — including remote and orchestrated runs, so a delegated tree accounts fully. |
| Audit log | Who changed what, when — with a metadata-only SIEM export. |
Telemetry
The platform emits OpenTelemetry spans following the GenAI semantic conventions — agent invocations, model calls with provider, request and response models and token usage, and tool executions. Any OTLP-capable backend can ingest them; the operations runbook carries ready-made Datadog and Grafana configurations.
Models and cost
All model traffic goes through one gateway, which is where cost, routing, and resilience are handled once rather than per agent.
| Capability | Detail |
|---|---|
| Tiers | Agents declare default, cheap, or strong, and the organization maps tiers to concrete models. Re-point a tier and every agent follows, with no edits. |
| Internal tiering | Planning, critique, reranking, and grading run on the cheap tier; synthesis runs on the strong tier. |
| Bring your own keys | Per-organization provider credentials, encrypted at rest. |
| Auto-routing and failover | A tier resolves to an ordered candidate list. A provider outage fails over mid-conversation, with unhealthy providers put in cooldown and skipped until they recover — the run does not fail. |
| Cost accounting | Per-model pricing is applied at the gateway, so every run carries a real cost figure rather than a token count someone has to translate. |
Analytics
| View | Shows |
|---|---|
| Builder dashboard | Per-agent success rate, average and p50/p95 latency, cost, top failure reasons, and the evaluation-score trend. |
| Organization dashboard | Weekly active users, run and cost totals, automation volume, and top agents. |
| Showback | A monthly CSV of spend for chargeback, for administrators. |
| Cost SLOs | Per-agent cost objectives, swept on a schedule, alerting once per window. |
| ROI | A per-agent baseline of minutes saved per run turns run volume into time saved, on both dashboards. |
| Opportunities | Ranked suggestions for what to automate next and what to harden — mined from usage metadata only. |
Rollups are recomputed idempotently per organization-day, so a re-run repairs
rather than double-counts. Spend limits are enforced by the
budget policy, not by these
dashboards.
The admin console
| Tab | Purpose |
|---|---|
| Policies | Create, scope, enable, disable, and delete governance policies. |
| Kill switches | Pause or kill an agent, suspend a user, trigger the organization-wide emergency stop. |
| Audit | Browse the audit log and download the SIEM export. |
| Legal holds | Request, dual-approve, and release holds. |
| Privacy | Set the organization's content-access tier. |
| Risk | The agent risk register: recompute, attest, and dual-authorize sign-off. |
| External agents | The control tower — every remote agent registered in the organization. |
| Compliance | Tier-1 run metadata for review. |
| Teams & domains | Team membership and verified email domains. |
| Web search | Choose and configure the search provider used by web_search. |
Other operator surfaces sit outside the console: Build → Capabilities toggles the built-in meta-agents and the built-in skill pack, Settings → Channels links Slack and Teams workspaces, Settings → Access manages API keys and secrets, and Tools covers the library, gateways, and virtual servers.
Isolation and limits
- Sandboxing — code runs in an ephemeral container per run: no network, non-root, capped memory, CPU, process count and wall clock, dropped capabilities, no package managers. Files move in and out as archives, never through host mounts. Stronger kernel isolation is a runtime-class change in production, not a code change.
- Rate limiting — inbound webhooks and gateway calls shed with
429under load; capacity is respected rather than degrading into server errors. - Concurrency — per-user semaphores bound sandbox execution and automation dispatch.
- Retention — a scheduled sweep applies per-organization retention to audit records and conversations, with active legal holds pinning what must be kept.
Deploying
One Helm chart, three value sets, so the shape of the deployment matches the size of the organization:
| Tier | Shape |
|---|---|
| Starter | Everything in-cluster — database, cache, and object storage included. Good for a pilot. |
| Team | External managed services and TLS ingress. |
| Enterprise | Autoscaling, stronger sandbox isolation, OpenTelemetry export, and external secret management. |
Database migrations run as a release hook. The gateway and this documentation site are separate deployments, so each scales and ships on its own cadence. Ingress is configured for long-lived streaming responses, which is the one detail worth checking first when streaming misbehaves behind a proxy.
The docs/runbooks/ directory in the repository carries the
operational detail: deployment, day-two operations, observability, sandbox
configuration, backup and restore, and channel setup.