Container Choice For Agent Workloads: Firecracker, Kata, gVisor, And The Tradeoffs

TL;DR

Three sandbox technologies have emerged as the credible options for agent workloads at scale: Firecracker, the AWS-developed microVM that powers Lambda and Fargate; Kata Containers, the OpenStack Foundation project that runs OCI containers inside lightweight VMs; and gVisor, the Google-developed user-space kernel that provides container-level isolation through syscall interception. Each technology occupies a different point on the tradeoff curve between cold-start latency, memory overhead, escape risk, and observability. Firecracker offers the strongest isolation at fifty to one hundred millisecond cold start. Kata offers near-VM isolation with a heavier memory footprint and two hundred millisecond cold start. gVisor offers container-friendly observability at one hundred millisecond cold start with weaker isolation against kernel-bypass attacks. Choosing among them requires matching the workload's actual needs to the technology's actual properties. This essay walks through each technology's architecture, the tradeoff dimensions, and the decision framework. The reader artifact is a Container Choice Decision Matrix that maps workload properties to technology recommendations.

Introduction: The Wrong Container For The Workload

A platform that hosted code-execution sandboxes for an autonomous coding agent had selected gVisor as its isolation technology. The selection was reasonable on paper. The workload involved running short-lived code snippets that the agent generated. gVisor offered fast cold starts because it did not require booting a guest kernel. The team had read the architecture documents and concluded that gVisor's user-space kernel provided adequate isolation for untrusted code.

The problem appeared six months in, when the agent was generating workloads that exercised parts of the syscall surface that gVisor did not fully implement. Some syscalls failed unexpectedly. Some succeeded but produced incorrect results. The agent's code was correct against a standard Linux environment but produced confusing failures inside the sandbox. The team spent weeks debugging issues that traced back to gVisor's syscall coverage gaps. The debugging consumed engineering capacity that should have been spent on the agent itself.

The team eventually migrated to Firecracker. The migration took a quarter. Cold starts became fifty milliseconds instead of one hundred. Memory overhead per sandbox grew by about fifty megabytes. Most importantly, the syscall surface became a full Linux kernel inside each microVM, which eliminated the coverage gaps. The agent's code worked predictably. The debugging time dropped sharply.

The lesson was that the team had selected gVisor based on a partial understanding of the workload. The cold-start property had dominated their thinking because the platform's marketing emphasized fast iteration. The syscall coverage property had been invisible because the team did not have a clear view of which syscalls the workload would exercise. The selection was wrong in a way that only became visible after the workload ran.

The broader pattern is that container choice for agent workloads is a real engineering decision with real consequences. The three credible technologies have meaningfully different properties. The right choice depends on the workload's actual needs, which are not always obvious in advance. This essay walks through the technologies and the decision framework that produces the right match.

Firecracker: The Minimalist MicroVM

Firecracker is a virtual machine monitor optimized for serverless workloads. AWS released it in 2018 to power Lambda and later Fargate. The project's design choices are aggressive: it strips the VM monitor to the minimum necessary to run a Linux guest, eliminates emulated hardware that serverless workloads do not need, and exposes a small attack surface to the host.

The Firecracker codebase is around fifty thousand lines of Rust. The small size matters because every line is a potential bug, and bugs in the VM monitor are catastrophic. Comparable hypervisors, like the older QEMU, run into millions of lines of C and have a much larger attack surface. The minimalism is not a marketing claim; it is a measurable property that produces fewer CVEs over time.

The architecture is a Type 2 hypervisor that runs as an unprivileged Linux process. Each microVM has its own kernel, its own root filesystem, its own memory, and its own virtual devices. The devices are minimal: a virtual block device for storage, a virtual network device for networking, a serial console for debugging, and a small number of paravirtualized helpers. There is no PCI bus emulation, no graphics, no USB. The reduced device set means less code in the path between the guest and the host.

The cold-start performance is around fifty milliseconds for a stripped-down kernel. The number assumes the VM image is in the host's page cache and that the kernel is built for fast boot. Production deployments often achieve sub-fifty millisecond starts through additional optimizations like snapshot-based boot, where a previously booted VM's state is restored rather than re-booting from scratch. The snapshot pattern is what makes Firecracker viable for use cases where every millisecond matters.

The memory overhead per microVM is typically one hundred to two hundred megabytes for a useful guest plus the kernel. The overhead is non-trivial but manageable for workloads where each VM serves a substantial computation. The overhead is prohibitive for workloads where each VM serves a single short call, in which case the per-call overhead would dominate.

The isolation properties are strong. The guest runs its own kernel inside a hardware-virtualized address space. A guest kernel exploit gives the attacker root in the guest but does not reach the host. The attacker would have to escape the hypervisor to reach the host, and the hypervisor's small surface makes such escapes rare. Side-channel attacks against the host are possible but are mitigated by the hypervisor's separation of guest memory from host memory.

The observability properties are mixed. The guest is a full Linux kernel, which means standard Linux observability tools work inside it: ps, top, perf, strace. The guest is opaque to the host beyond the device interfaces, which means host-side observability tools cannot see inside the guest. The pattern is appropriate for workloads where the guest's internal state should be invisible to the host operator, like multi-tenant code execution.

Firecracker is the right choice for workloads that prioritize isolation over observability and that can absorb the per-VM memory overhead. The cold-start latency is competitive with the alternatives. The isolation is the strongest of the three options. The memory overhead is the highest of the three options. The decision depends on the workload's tolerance for each property.

Kata Containers: The Compatibility Bridge

Kata Containers is the OpenStack Foundation project that emerged from the merger of Hyper.sh's runV and Intel's Clear Containers in 2017. The project's goal is to provide VM-level isolation while preserving OCI container compatibility. A Kata container looks and behaves like a standard Docker container from the developer's perspective, but each container runs inside its own lightweight VM.

The architecture has three layers. The OCI runtime, called containerd-shim-kata-v2, accepts container creation requests in the standard OCI format. The runtime hands off to a VM monitor, which is typically QEMU but can be Firecracker or Cloud Hypervisor. The VM boots a minimal Linux kernel and a small init system, then runs the container's workload inside the VM. From the container's perspective, the VM is invisible: the container sees its standard Linux environment.

The layering produces a heavier resource footprint than pure Firecracker. Each Kata container includes the VM monitor overhead, the guest kernel, and the container runtime inside the VM. The total per-container memory overhead is typically two hundred to three hundred megabytes, depending on the configuration. The overhead is the cost of the compatibility layer.

The cold-start performance is around two hundred milliseconds for a typical container. The number includes the VM boot time and the container runtime startup inside the VM. Optimizations like VM templating, where a pre-booted VM is forked for each container, can reduce the cold start to under one hundred milliseconds at the cost of some isolation. Most production Kata deployments accept the longer cold start in exchange for the standard configuration.

The isolation properties are similar to Firecracker. The guest kernel runs in its own hardware-virtualized address space. A container compromise that achieves kernel exploit reaches root in the guest but not the host. The hypervisor surface is the same as the underlying VM monitor, which is QEMU by default. QEMU's larger codebase means a larger attack surface than Firecracker, although QEMU has had years of hardening for this exact use case.

The compatibility properties are the strongest of the three options. Any OCI-compliant container image runs inside Kata without modification. Standard container orchestration tools, including Kubernetes, work with Kata as a container runtime. Standard observability tools that target containers work because the workload is a container from the orchestrator's perspective. Migrations between standard containers and Kata containers are straightforward.

The observability is better than Firecracker in the standard ways. The container's process list, its network activity, and its filesystem are all observable through the container runtime's interfaces. The VM is a transparent layer; the container abstraction is what the operator interacts with. The transparency makes Kata easier to operate at scale than raw Firecracker.

Kata is the right choice for workloads that need VM-level isolation but want to preserve container compatibility and tooling. The memory overhead is the highest of the three options. The cold-start latency is the slowest. The isolation is comparable to Firecracker. The compatibility advantage is decisive when the team has existing container infrastructure or when the workload is best expressed as a standard container.

gVisor: The User-Space Kernel

gVisor is the Google-developed sandbox runtime that provides container-level isolation through syscall interception in user space. Google developed gVisor for App Engine and Cloud Run, where the workload mix involves many short-lived requests across many tenants. The project was open-sourced in 2018.

The architecture is structurally different from the VM-based options. gVisor implements a user-space kernel called Sentry that intercepts syscalls from the sandboxed application and either handles them itself or delegates to the host kernel through a restricted gateway. The Sentry runs as an unprivileged process. The application sees what looks like a Linux kernel but is actually the Sentry's reimplementation.

The approach has interesting tradeoffs. The Sentry has full visibility into every syscall the application makes, which enables fine-grained policy enforcement and detailed observability. The Sentry's reimplementation is partial; it covers the syscalls that typical workloads use but does not cover everything Linux supports. Workloads that exercise unusual syscalls or rely on specific kernel behaviors may encounter coverage gaps.

The coverage gaps were the issue in the introduction's failure case. Most common syscalls work. Less common syscalls, especially those used by lower-level system tools, may not. The gVisor project publishes a syscall compatibility matrix that documents which syscalls are implemented and to what extent. Workloads that depend on syscalls outside the well-supported set should test carefully before committing to gVisor.

The cold-start performance is around one hundred milliseconds for a typical container. The Sentry starts faster than a guest kernel because it does not have to perform hardware-style initialization, but the overall startup includes the container runtime overhead. The cold start is competitive with the VM-based options for most workloads.

The memory overhead per container is around fifty megabytes for the Sentry plus the application's own memory. The overhead is the lightest of the three options. The lightness comes from not having a guest kernel; the Sentry is a smaller process than a Linux kernel.

The isolation properties are the most subtle of the three. The Sentry handles many syscalls itself, which means many attack vectors against the host kernel are blocked because the host kernel never sees the syscall. Attacks that depend on host kernel vulnerabilities have a harder time succeeding. However, the Sentry itself is a complex piece of code that has had its own vulnerabilities, and the gateway between the Sentry and the host kernel is an additional surface that has been targeted by escape research. The isolation is strong against most attacks but is generally considered weaker than the VM-based options against sophisticated attackers.

The observability properties are the best of the three. The Sentry sees every syscall and can produce detailed traces of the application's behavior. Standard tools for observing containers work because gVisor presents itself as a container runtime. The combination of detailed syscall observability and container-friendly tooling is unique to gVisor.

gVisor is the right choice for workloads that prioritize observability and lightweight resource use over the strongest possible isolation. The memory overhead is the lowest of the three. The cold-start latency is competitive. The isolation is adequate for most threat models. The observability is unmatched. The syscall coverage gaps require workload testing before commitment.

Tradeoff Dimensions In Detail

The choice among the three technologies depends on how the workload weights four tradeoff dimensions: cold-start latency, memory overhead, escape risk, and observability. Each dimension has multiple sub-considerations that affect the decision.

Cold-start latency matters when the workload involves many short-lived sandboxes. An agent that spawns a new sandbox per request, with requests arriving every few seconds, sees the cold-start latency on the request critical path. A fifty-millisecond cold start adds five percent to a one-second request. A two-hundred-millisecond cold start adds twenty percent. The difference is significant for user-perceptible latency. For workloads where the sandbox lifetime is measured in minutes or hours, the cold-start latency disappears into the noise.

Memory overhead matters when the workload involves many concurrent sandboxes. A platform running ten thousand concurrent sandboxes pays the memory overhead per sandbox. Fifty megabytes per sandbox is half a terabyte across ten thousand. Two hundred megabytes per sandbox is two terabytes. The hardware bill differs by a factor of four. For platforms with smaller concurrency, the memory overhead is a smaller fraction of the operating cost.

Escape risk matters when the workload's compromise has high consequences. A workload that handles untrusted input from anonymous users on the internet has a high probability of attempted compromise. A workload that runs trusted code from internal teams has a low probability. The escape risk should be weighted by the consequences of a successful escape. For workloads where escape consequences are high, the strongest isolation is worth significant cost in other dimensions.

Observability matters when the workload's operation requires understanding what happened inside the sandbox. A debugging workflow needs to see the application's behavior. A compliance workflow needs to produce evidence of what was done. A security investigation needs to reconstruct the sequence of events that led to a suspicious outcome. Workloads with strong observability needs benefit from the technology that exposes the most internal detail.

The weights on these dimensions are workload-specific. A code-execution platform for an autonomous coding agent might weight syscall coverage and observability heavily because the agent's debugging cycle depends on both. A multi-tenant function platform might weight cold start and memory overhead heavily because the per-call economics depend on both. A high-trust internal workload might weight observability over isolation because the internal threat model is mild.

Named Artifact: The Container Choice Decision Matrix

The artifact below is a decision matrix that maps workload properties to technology recommendations. The matrix has rows for workload archetypes and columns for the four tradeoff dimensions. Each cell contains the recommended technology for the archetype-dimension combination, with brief reasoning.

For multi-tenant code execution from anonymous users, the recommendation is Firecracker. The cold-start latency is competitive at fifty milliseconds. The memory overhead is acceptable at two hundred megabytes per VM. The isolation is the strongest available, which matches the high-consequence threat model. The observability is adequate for the platform's needs because the workload's internal state should be opaque to the platform anyway.

For autonomous agent code execution where the agent debugs its own code, the recommendation is gVisor with a fallback to Firecracker. The cold-start latency at one hundred milliseconds is acceptable. The memory overhead at fifty megabytes supports high concurrency. The observability advantage is significant for the debugging cycle. The fallback to Firecracker handles the syscall coverage cases that gVisor does not implement. The architecture supports both technologies through a runtime selector.

For function-as-a-service workloads with high request rates and small per-request costs, the recommendation is Firecracker with snapshot boot. The cold-start latency drops to twenty milliseconds with snapshots. The memory overhead is manageable at high concurrency because the snapshot footprint is smaller than a full boot. The isolation is the strongest available, which matches the multi-tenant threat model.

For existing container workloads that need stronger isolation than standard containers provide, the recommendation is Kata. The compatibility advantage is decisive because no workload code changes. The cold-start latency at two hundred milliseconds is acceptable for workloads that run for minutes or hours. The memory overhead is acceptable at the typical concurrency. The isolation is similar to Firecracker.

For internal workloads with low threat profiles, the recommendation is the cheapest option that meets the operational needs, typically standard Docker containers without any of the three technologies. The trust differential between the workload and the platform is low enough that the additional isolation does not justify the cost. The right answer for internal workloads is often that the question of sandbox technology does not apply.

For research and experimentation workloads where the team wants maximum visibility into application behavior, the recommendation is gVisor. The syscall trace data is uniquely valuable for understanding what the application does. The isolation is adequate for the research threat model. The cold-start and memory properties are favorable.

The matrix is a starting point. Real decisions depend on specifics that the matrix cannot capture: the team's operational maturity with each technology, the existing infrastructure investments, the customer expectations, the regulatory environment. The matrix narrows the choice from three options to one or two, and the team's specifics determine the final selection.

Operational Considerations

Beyond the technology choice itself, several operational considerations affect how the technology performs in production. These considerations apply across all three options but may favor one over another in specific scenarios.

The first consideration is the runtime substrate. All three technologies need a host operating system, a kernel version that supports the underlying primitives, and an orchestration layer that can dispatch workloads to the runtime. The choice of orchestrator may favor one technology over another. Kubernetes works with all three through the Container Runtime Interface, but the integration maturity differs: Kata has been a first-class Kubernetes runtime for years, Firecracker integration through Kubernetes is improving but not as mature, and gVisor through runsc is well-supported.

The second consideration is the storage subsystem. The sandbox needs storage for its root filesystem and any persistent state. The storage performance and configuration affect cold-start latency and runtime performance. Firecracker uses simple block devices, which are easy to configure but require image preparation. Kata supports the standard container image stack, which is more complex but well-tooled. gVisor uses container images directly through its own filesystem layer, which simplifies the image management.

The third consideration is the networking subsystem. The sandbox needs network access for any outbound calls and for inbound serving if applicable. The networking model affects both performance and security. Firecracker uses tap devices that the host configures, which provides full control but requires host-side setup. Kata uses standard container networking, which integrates with existing tooling. gVisor implements its own network stack in the Sentry, which provides observability but may have compatibility gaps for unusual protocols.

The fourth consideration is the operational tooling. The team will need monitoring, logging, and incident response tooling that works with the chosen technology. Mature tooling exists for all three options, but the specific integrations differ. The team should evaluate their existing tooling against each option before committing.

The fifth consideration is the upgrade cadence. Each technology releases updates regularly. The team needs to upgrade to maintain security patches and performance improvements. The upgrade process differs across the three options and may affect the operational burden. Firecracker upgrades are straightforward because the surface is small. Kata upgrades involve coordinating across the runtime, the VM monitor, and the guest kernel. gVisor upgrades are similar to standard container runtime upgrades.

Performance Benchmarking Methodology

The published numbers for cold-start latency, memory overhead, and throughput across the three technologies vary widely depending on the benchmark methodology. Numbers from one source may not be comparable to numbers from another. Teams that rely on published benchmarks without running their own often find the production behavior differs from expectations. The right approach is to run a benchmark that matches the team's actual workload and measure the relevant properties directly.

The cold-start latency benchmark should measure the time from sandbox creation request to the first useful operation inside the sandbox. The first useful operation is workload-specific. For a code-execution workload, it is the moment the user's code begins executing. For a function-as-a-service workload, it is the moment the function handler is invoked. The difference between sandbox creation and first useful operation includes the runtime initialization, which can be substantial for some technologies.

The benchmark should run repeatedly to capture the latency distribution. The mean is less interesting than the tail. A technology with a fifty-millisecond mean cold start might have a five-hundred-millisecond ninety-ninth percentile cold start. The tail dominates user experience for high-volume workloads. The benchmark should report the mean, the median, the ninety-fifth percentile, and the ninety-ninth percentile. The numbers should be collected over a sufficient sample to be statistically meaningful, typically at least ten thousand cold starts.

The memory overhead benchmark should measure the resident set size of each sandbox at steady state. The measurement should distinguish the runtime overhead from the application's own memory use. The runtime overhead is the part that depends on the technology choice. The application memory use is the part that depends on the workload. The benchmark should report the runtime overhead independently so the team can predict the overhead for their actual application.

The throughput benchmark should measure how many sandboxes the host can run concurrently while maintaining acceptable latency. The number depends on the host's CPU and memory, the workload's resource use, and the technology's overhead. The benchmark should drive concurrency upward until latency degrades, then report the concurrency at which the degradation begins. The number is the practical capacity for that host configuration.

The benchmark should include the network and storage paths because they affect real performance. A sandbox technology that has fast cold starts on a benchmark with no I/O may have slow cold starts when storage operations are involved. The benchmark should include the typical I/O pattern of the workload.

The benchmark should be repeated periodically because the technologies evolve. New versions of Firecracker, Kata, and gVisor are released regularly with performance improvements. The team's measured numbers should be refreshed annually at least, more often if the team is making significant infrastructure changes.

The benchmark methodology and results should be documented and shared internally. The documentation becomes the reference for capacity planning, cost modeling, and architectural decisions. The shared knowledge prevents teams from making decisions based on published numbers that do not match the team's actual experience.

Counter-Argument: The Choice Matters Less Than The Operational Maturity

The natural objection is that the choice of technology matters less than how well the team operates whatever they choose. A poorly operated Firecracker deployment is less secure than a well-operated gVisor deployment. The marginal isolation differences between the technologies are smaller than the differences between mature and immature operational practices. The team should choose whatever technology they can operate well rather than optimizing for properties that operational deficiencies would erase.

The objection is correct in the limit and misleading in practice. The technology choice and the operational maturity are not independent. Choosing Firecracker without operational maturity for it is a problem. Choosing gVisor without operational maturity for it is also a problem. The right framing is that the team should choose the technology whose operational requirements match the team's capacity and preferences.

That framing leads to different choices than the pure-property comparison. A team with strong Linux kernel expertise and existing VM operations can run Firecracker effectively. A team with strong container orchestration expertise and limited VM experience may have an easier time with Kata or gVisor. The right answer depends on the team, not just on the technology.

The operational-maturity argument also points to the value of consolidation. A team that runs three different sandbox technologies for three different workload classes pays operational overhead for each. A team that runs one technology with workload-specific configurations pays less overhead. The matrix in this essay shows three options not because every team should run all three but because different teams in different situations should choose different ones. Most teams should pick one and use it for everything that fits its profile.

What Armalo Does

Armalo supports all three sandbox technologies as runtime modes that agent operators can choose from. The choice is exposed through the agent registration interface as process isolation, container isolation, and microVM isolation, where container isolation can be either Kata or gVisor depending on the operator's preference. The choice influences the agent's runtime characteristics and is reflected in the agent's pact and composite score.

The runtime selector evaluates the workload's properties against the decision matrix and recommends a default mode. Operators can override the recommendation. The selection is recorded in the agent's pact and influences the security dimension of the composite score. Agents that operate in stronger isolation modes earn higher security scores, which feeds the trust oracle and influences counterparty decisions.

Armalo's egress controls work uniformly across all three technologies. The controlled proxy enforces destination allowlists regardless of the underlying isolation. The DNS resolver is internal regardless of the technology. The capability-based authorization for outbound calls is enforced at the runtime layer. The technology choice affects the isolation around the agent, but the policy enforcement around the agent's outbound activity is the same.

The quarterly stress test described in a companion essay covers all three technologies. The side-channel, kernel exploit, egress bypass, TOCTOU, and callback escalation tests run against each technology in Armalo's infrastructure. The results inform Armalo's recommendations for which technology to use for which workload class. The recommendations evolve as the test results evolve.

FAQ

Can a single platform run multiple sandbox technologies simultaneously? Yes, with operational complexity. Each technology requires its own runtime, its own monitoring, and its own incident response procedures. The complexity grows with the number of technologies. Most platforms benefit from picking one technology that fits the majority of their workloads and accepting the suboptimality for the minority.

How does the technology choice interact with serverless platforms like Lambda or Cloud Run? Lambda uses Firecracker internally. Cloud Run uses gVisor internally. The choice is made by the platform, not by the user. Users who care about the underlying isolation should evaluate the platform's choice against their threat model. Users who want to choose the technology themselves should run their own infrastructure.

What about WebAssembly as a sandbox technology? WebAssembly is an emerging option that offers strong isolation through its bytecode model and very fast cold starts measured in microseconds. The tradeoff is that WebAssembly does not support arbitrary Linux workloads; the application must be compiled to WebAssembly bytecode. For workloads that fit, WebAssembly is competitive. For workloads that need a full Linux environment, the three technologies in this essay are the right choices.

How does the choice affect the cost of running the platform? Memory overhead translates directly to cost. A platform running ten thousand concurrent sandboxes pays for the memory overhead of each sandbox. The cost difference between fifty megabytes and two hundred megabytes per sandbox is substantial at scale. The cold-start latency affects the cost indirectly through user experience and throughput. The isolation properties affect the cost indirectly through risk-adjusted operations.

What if the workload needs GPU access? GPU access in sandboxes is an evolving area. Firecracker supports GPU passthrough through specific configurations. Kata supports it through standard container GPU patterns. gVisor's support is limited because GPU access requires syscalls and ioctls that the Sentry does not fully implement. Workloads that need GPUs should evaluate the specific GPU integration story for each option.

How do you migrate between technologies if the initial choice turns out to be wrong? The migration involves rebuilding the workload deployment for the new technology. Container-based workloads migrating from one container technology to another are straightforward. Migrations to or from Firecracker involve more work because the deployment model differs. The migration cost should be factored into the initial choice; a wrong choice is recoverable but expensive.

Are there other sandbox technologies worth considering? Cloud Hypervisor, Cloud-Hypervisor's older cousin Crosvm, and various proprietary sandboxes from cloud providers exist. The three technologies in this essay are the most widely deployed and the most mature for general-purpose agent workloads. The others may be appropriate for specific situations but require deeper evaluation.

Bottom Line

The three sandbox technologies for agent workloads are not interchangeable. Each has a clear profile across cold-start latency, memory overhead, escape risk, and observability. Firecracker is the strongest isolation at moderate cost. Kata is the compatibility bridge for existing container infrastructure. gVisor is the lightweight observability-rich option with syscall coverage caveats. The right choice depends on the workload's actual properties and the team's operational capacity.

The decision matrix in this essay narrows the choice from three options to one or two for any given workload archetype. The team's specifics determine the final selection. The wrong choice is expensive in the form of operational friction, debugging time, and potentially security exposure. The right choice is invisible because the platform just works.

The quarterly testing described in the companion essay applies to all three technologies. The technology choice does not eliminate the need for ongoing security validation. The choice positions the platform on a particular point of the tradeoff curve. The validation confirms the platform stays at that point as the underlying technology and threat space evolve.

The right time to make the choice is at the start of the platform, before the first production workload commits the team to a technology. Migrating later is possible but expensive. The matrix is a tool for making the choice deliberately. The choice itself is the team's, informed by the workload, the threat model, and the operational realities of the team that will run the platform.