Scaling an autonomous agent from one team to fifty: A serverless-to-container decision framework

QA teams shipping complex products generate test failures around the clock. Every failure needs a bug ticket with steps, screenshots, repro rates, and regression labels. The manual process takes 10-15 minutes per failure. Multiply that by 30+ failures per week across a team, and you have burned an entire engineering day on documentation that should be automatic. I built a system to eliminate this entirely. Not a dashboard. Not a notification bot. An autonomous agent that observes test results, makes judgment calls – is this a new bug, a duplicate, or a regression? And takes action without human intervention. It files structured bugs, attaches screenshots, calculates repro rates, detects regressions build-over-build, and notifies the right people. The team just marks FAIL and moves on. The system runs on Lambda today. It works perfectly for one team. But what happens when other teams want it? What happens when you need to serve 10, 50, or 100 teams? That’s where the architectural decisions get interesting – not because Lambda fails, but because the workload characteristics shift into territory where a different tool becomes the better fit. What I built (and why it works) The architecture is elegant: an EventBridge rule triggers a Lambda function every 15 minutes. The function polls the test management API, processes failures through a decision engine (dedup check against DynamoDB, regression detection against previous build results), and files bugs via the JIRA API. Total infrastructure cost: under $5/month. Lambda was the correct choice here for three reasons: Zero operational overhead: No servers, no scaling config, no patching. Deploy and forget. Cost proportional to usage: One team generates ~200 invocations/month. At Lambda pricing, this is effectively free. Fast iteration: The decision engine went through 20+ revisions in the first month. Lambda’s deployment model let me ship changes in seconds. Within weeks of deploying, the system filed more complete and consistent bugs than the team had ever produced manually. Zero missed failures. Regressions caught in minutes instead of days. Status interruptions to QA dropped to zero because anyone could query the system directly via Slack. 8-10 hours per week saved. Other teams have noticed. “Can we have that?” is a recurring question. The answer seems simple – deploy another instance per team. But at scale, the workload characteristics change in ways that shift the architectural decision. The scaling decision Lambda handles single-tenant agentic workloads beautifully. The challenge at multi-tenant scale isn’t that Lambda can’t do it; it’s that the workload characteristics shift: From stateless events to long-running context: A single team’s poll cycle finishes in seconds. But cross-project correlation – “is this the same failure pattern we’re seeing across three teams?” requires analyzing hundreds of results and their relationships in a single pass. That’s a fundamentally different compute profile. From isolated invocations to shared resource contention: When 50 functions trigger on the same EventBridge schedule, they all hit TestRail and JIRA APIs simultaneously. You need centralized rate limiting, connection pooling, and backpressure – patterns that benefit from persistent, shared infrastructure. From single-team to multi-tenant isolation: Each team needs guaranteed resources, network boundaries, and scoped credentials. The isolation model becomes as important as the compute model. Lambda’s recent launches address many of these individually – Managed Instances solve connection persistence, Durable Functions break the 15-minute ceiling for long-running workflows, and Tenant Isolation provides native tenant boundaries. For many multi-tenant workloads, these capabilities narrow the gap significantly. But for this specific workload – long-running agents that need persistent connections, cross-tenant analysis, GPU inference, and namespace-level isolation simultaneously- the decision framework points toward a container-native execution model with serverless-like operations. That’s where EKS Auto Mode fits. The decision framework This led me to build a simple decision rubric – not Lambda vs. containers, but which characteristics does your workload actually exhibit? Workload characteristicLambda (+ Managed Instances, Durable Functions)Auto Mode Event-driven, short-livedOptimalOverengineered Single-tenant, low complexityOptimalUnnecessary Multi-tenant with isolation needsPossible (Tenant Isolation)Native (namespaces, RBAC, NetworkPolicies) Long-running with persistent statePossible (Durable Functions)Native (pods, persistent connections) Cross-tenant analysis requiring shared memoryComplex (custom orchestration)Natural (shared services namespace) GPU-burst inferencePossible (Lambda GPU – if available)Native (GPU NodePools, scale-to-zero) Centralized rate limiting + connection poolingCustom build (SQS + Step Functions)Shared platform service “The deciding factor for my workload: I need all of the right-column characteristics simultaneously. Together, they compose more naturally in a container-native environment.” The deciding factor for my workload: I need all of the right-column characteristics simultaneously. Cross-team pattern detection requires a shared analysis context. ML-based bug classification requires burst GPU. Per-team isolation requires namespace-level boundaries. Centralized rate limiting requires a persistent shared service. Each of these is individually solvable in Lambda. Together, they compose more naturally in a container-native environment. What unlocks at scale: cross-team intelligence The most interesting architectural capability that emerges at multi-tenant scale isn’t per-team automation – it’s cross-team pattern detection. This is what makes the scaling exercise worthwhile. Cross-team failure correlation When 50 teams run their test suites independently, failures appear isolated. But patterns emerge across teams that no single team can see: Three teams report navigation failures on the same firmware build → systemic regression, not three independent bugs Test failures correlate with deployment timing across unrelated services → infrastructure issue, not product bugs Identical error signatures across teams with different test suites → shared dependency failure This requires an analysis engine that can hold context from all teams simultaneously – comparing fingerprints, correlating timestamps, and identifying shared root causes. That’s a persistent, long-running workload with access to cross-tenant state. It runs naturally as a shared service in a platform namespace, consuming events from all tenant agents. ML-based bug classification Once you have enough historical data (thousands of bugs filed across teams), patterns become classifiable: Auto-assign bugs to the correct component owner based on failure signature Predict severity based on failure characteristics (crash vs. degraded behavior vs. cosmetic) Suggest related bugs for investigation (“this looks like the same root cause as PROJ-4521”) Detect anomalous failure rates before they become incident-scale This inference workload is bursty – you need GPU when bugs arrive, not 24/7. A GPU NodePool that scales to zero when idle and provisions g5 instances only during inference keeps costs proportional to actual usage. The same pod that files a bug can synchronously call the classification service, get a component assignment, and include it in the ticket – all within the same request path. Queryable knowledge base via Slack The agent isn’t just an automation – it becomes the organization’s queryable QA state. Via Slack, anyone can ask: “What’s the failure rate for navigation this week across all teams?” “Show me regressions introduced in build 2.4.1” “Which teams are seeing the most P0 bugs right now?” This transforms the agent from a task automator into an intelligence layer. The answers require cross-team state aggregation, trend computation, and natural language interpretation – all served from persistent, indexed state that accumulates over time. What EKS Auto Mode brings to this architecture EKS Auto Mode launched at re:Invent 2024 with a specific promise: full Kubernetes without managing nodes. For a multi-tenant agent platform, this means: Nodes appear when pods need them. Deploy a tenant namespace with an auto-filer pod. The Karpenter-based scheduler provisions a node within seconds. When the pod is deleted (tenant offboarded), the node is consolidated or terminated automatically. You never think about EC2 instances. Security is not your configuration burden. Nodes run immutable Bottlerocket AMIs – read-only root filesystem, SELinux enforcing, no SSH access. Nodes rotate every 21 days automatically. For long-running agent pods, this means a graceful drain roughly every three weeks. At most one missed poll cycle. Scaling is reactive to demand. During off-hours when agents are idle between polls, Auto Mode consolidates workloads onto fewer nodes. You pay for what you use. GPU workloads slot in without architectural changes. Define a GPU NodePool for the ML classification service. EKS Auto Mode provisions g5 instances only when inference pods are scheduled and terminates them when done. The multi-tenant architecture At scale, the system would have three layers: Shared platform services: a rate limiter that prevents any single tenant from exhausting external API quotas, a connection pool manager that maintains persistent sessions to TestRail and JIRA, and a Prometheus instance collecting metrics across all tenants. Per-tenant agent namespaces: each team gets an isolated namespace containing their auto-filer pod, reporter pod, and Slack bot pod. ResourceQuotas guarantee resources. NetworkPolicies enforce that Team A’s pods cannot talk to Team B