The pilot is signed. A Fortune 500 legal team wants your contract-review agent running on real matters within six weeks, and the founders are still celebrating in the group chat. Then the security questionnaire arrives: 240 rows in a spreadsheet, most of them routine. Row 87 is not.
“Can you provide a tamper-evident log of every action the agent took?”
Row 88: “Can the solution run inside our VPC?”
You trace what happens when an agent run starts. A Celery task picks it up. State lives in Redis. A runs table in Postgres has a status column, with a few rows stuck in running since last Tuesday. Retries come from a copied decorator. The only record of what the agent did is that table plus whatever reached the logs before the worker restarted.
The honest answer to both questions is no. And the runs table is not the product. It is plumbing that grew around the product.
Why this happens to almost every agent startup
Hand-rolled orchestration is the right call on day one. You need a demo, not a platform. A queue, a cache, and a status table get an agent loop running in an afternoon, and nothing about the first ten customers forces you to change it.
The trouble is that agent workloads have a shape that simple job queues were never designed for:
- Runs are long. An agent that waits for a paralegal to approve a redline may run for days. Queue workers expect seconds.
- Steps have side effects. The agent sends emails, updates CRMs, posts journal entries. A naive retry after a crash can repeat those actions.
- The path is non-linear. Tool calls, LLM turns, and human waits branch and loop. One status column cannot describe where a run is.
- Enterprise buyers ask about the whole system. Identity, encryption in transit, data residency, and audit are not features you bolt onto a queue. They are properties of the runtime.
So the demo system fails predictably once real customers arrive: runs stuck after a deploy, duplicate actions after a retry, no answer to “what did the agent do and why,” and a security review you cannot pass.
Why the usual patches fall short
Most teams respond with incremental fixes. Each is reasonable alone; together they quietly become a second product.
More columns on the runs table. You add current_step and last_heartbeat, then a reaper job for stale runs and logic to decide whether resuming is safe. You have started writing a workflow engine without a design.
Idempotency keys everywhere. These help only if every external call supports them and every engineer remembers. One miss means a customer gets two invoices.
Logging as an audit trail. Logs are mutable, often incomplete, and rotated. A reviewer asking for a tamper-evident record is not asking for a log search.
A heavier on-call rotation. Someone spends part of every week unsticking runs by hand. At an eight-engineer startup, that is a real share of the roadmap.
None of these address the underlying gap: you have no durable, inspectable record of execution that the system itself uses to recover.
What a production agent runtime actually needs
Before deciding whether to build or adopt, write down what the runtime must provide. For most agentic SaaS products selling upmarket, the list looks like this:
- Durable execution. Completed steps are persisted; after a crash or deploy, the run resumes without re-running work that succeeded.
- Retries with policy. Per-step backoff and limits, defined once.
- Human approvals and timers. A run can pause for days on a person, with a timeout and escalation, without holding a worker.
- Workload identity and mTLS. Each service and agent has a verifiable identity, and traffic between them is encrypted.
- An audit trail the system depends on. The execution history should be the same record used for recovery, so it cannot drift from reality. Ideally it can be signed so a third party can verify it.
- Multi-tenant isolation. One customer’s runaway agent should not starve another’s.
- Deployment flexibility. The same runtime should run in your cloud, a customer’s cloud account, or on-premises.
- Observability. Traces and metrics across every step.
Each item has established open-source or commercial solutions. None of them is why a customer picks your agent over a competitor’s.
The hidden cost of building it yourself
A credible version of this list is not a sprint. It means an event-sourced state model, a replay engine, a timer service, a certificate authority, signing infrastructure, tenancy controls, and packaging for several deployment targets. Then you maintain, upgrade, and staff on-call for all of it.
The larger cost is opportunity. The senior engineers who could build it would otherwise be improving prompts, tool design, evals, and domain workflows, the parts that actually differentiate you. Infrastructure every agent company needs is, by definition, not a moat.
There is also a sales cost: every month spent building audit and BYOC support is a month enterprise deals sit in security review.
How to adopt instead: open source first, managed where it helps
The vendor-neutral principle is simple: put your agent loop on top of a durable execution engine, and keep your agent logic, prompts, tools, and evals as your own code.
Dapr Workflow is one open-source option worth evaluating. Dapr is a CNCF graduated, Apache 2.0 project that runs as a sidecar next to your app on Kubernetes, VMs, or self-hosted machines. Its workflow engine is workflow-as-code, with SDKs for Python, .NET, Java, Go, and JavaScript. History is event-sourced into a state store such as PostgreSQL; after a crash, the workflow replays and completed activities are not re-executed. It supports chaining, fan-out/fan-in, child workflows, durable timers, external events for human approval, and compensation.
Other checklist items come from the broader runtime. Dapr’s Sentry service issues SPIFFE-based workload identities, mTLS between sidecars is on by default, and access-control policies restrict which app IDs can call which operations. OpenTelemetry tracing and Prometheus metrics come from every sidecar.
For Python teams, Dapr Agents is an open-source framework in the Dapr project, now at v1.0, that builds agents on Dapr Workflow. A global optics manufacturer built a document extraction pipeline on Dapr Workflow and Dapr Agents and reached production in about two months with no labeled training data.
Open-source Dapr already covers workload identity with mTLS and a durable workflow history, but it still leaves you operating the control plane. Diagrid, founded by the creators of Dapr, offers Catalyst as one managed option, with durable execution for popular agent frameworks, deployment options that include bring-your-own-cloud and self-hosted, and opt-in signed execution history. As with any vendor, verify these claims in your own environment, ideally one that resembles your largest customer’s.
A build-vs-adopt checklist
Use this before your next planning cycle. If you check more than three boxes, building your own orchestration layer is probably costing you more than it saves.
- ☐ At least one engineer spends recurring time each week fixing stuck or duplicated runs.
- ☐ You cannot reconstruct, step by step, what an agent did in a run from last month.
- ☐ A security questionnaire has asked about tamper-evident logs, workload identity, or mTLS, and you answered “roadmap.”
- ☐ An enterprise prospect has asked to run your product in their own cloud or on-premises.
- ☐ Your agents wait on humans, and those waits are implemented with polling or cron jobs.
- ☐ Retries are configured in more than one place, with different semantics.
- ☐ A deploy during business hours makes the team nervous because of in-flight runs.
- ☐ Your roadmap includes “build audit log” or “add workload identity” as engineering projects.
If you decide to adopt, a practical sequence:
- Inventory side effects. Every tool call that changes external state becomes an activity with an explicit retry policy.
- Separate orchestration from logic. Prompts, tools, and evals stay in your modules. The runtime calls them; it does not own them.
- Migrate one agent end to end. Pick the one with the worst failure modes and compare incident volume before and after.
- Model approvals as external events with timers. Delete the polling jobs.
- Turn on identity and tracing early, not under a deal deadline.
- Answer the questionnaire from evidence. Walk your security contact through a real execution history.
- Test the deployment target your biggest prospect needs. If that is their VPC, run a proof of concept there, whether you use open-source Dapr or a managed option for durable execution for AI agents.
Conclusion
The runs table got you to your first enterprise pilot. It will not get you through the security review or the second year of on-call. The orchestration layer is hard infrastructure, and every competitor needs the same thing.
Your advantage lives in the agent itself: the domain knowledge in your prompts, the quality of your tools, and the rigor of your evals. Adopt a durable runtime for everything underneath, and spend your engineering hours where customers can actually see the difference.
