What AWS Lambda was hiding

Earlier this year I finished six months of moving seven Python Lambdas into services. Each was closer to a small application than a thin handler.

Moving the code into services changed how long state survived and which deliveries shared process resources. The services also had to manage long-lived connections and credential renewal.

Those changes exposed problems in code that had run safely under Lambda for years. We found some before rollout and others in production.

If you're here for the technical content and not the migration context, skip to the cases.

Why the migration

The Lambdas were the platform's ingestion path for external time-series data. The decision to migrate had nothing to do with the runtime semantics. Three operational pressures pushed for it:

• Multi-environment support. Compared to our established Kubernetes platform serving the rest of the cloud, maintaining environment parity across Lambda deployments wasn't manageable.
• Global scaling. Scaling Lambda functions across regions required separate deployment pipelines per region, each with its own configuration and monitoring.
• Cognitive load. Seven Lambdas and a slow buildup of glue code had made the system harder to reason about than the work it was doing.

The old architecture, briefly

The Lambdas ingested external time-series data and synced it to different target databases, with the entire pipeline written in Python.

Each sync Lambda did the same two things: extract and transform data into a shape suitable for its target database(s), then write it. The extraction logic was shared across the four sync paths and lived in a common library, but each Lambda still ran it independently as part of its invocation. The full pipeline included three additional Lambdas that performed slightly different extractions for adjacent purposes.

Seven Lambdas total. Four sync paths repeating the same extraction work, plus three more doing variations of it.

The new architecture

The new architecture had one deployable unit responsible for extraction and normalization, and a set of consumers each responsible for one or more target databases. Consumers would receive events over a shared contract, fetch the normalized payload from S3, and write it to their respective stores.

The extraction service was rewritten in Go. Two reasons: we wanted static typing on the critical path (this service is the producer every consumer depends on), and the team already ran Go across the rest of the platform, so it was a known quantity to build and operate.

The Go service was designed for the runtime it runs in. None of the cases that follow happened there.

I kept scope to one language change at a time. The downstream consumers stayed in Python. Their pandas and broader Python ecosystem dependencies would have made a simultaneous language migration too risky.

The eight cases

The rollout used a custom canary configuration: test and anonymous traffic first, then a small set of real customers, then wider. The migration had one hard constraint: new-service output had to match the Lambda's exactly, on the same input. That constraint ruled out cleaning up every suspicious code path while moving it. The data layouts are complex enough that even small code changes can produce subtle differences customers would notice, so we deliberately kept changes minimal. Anticipated fixes that risked behavioral drift were held until the canary could confirm them against real traffic.

1. /tmp collisions and a duplicate-delivery discovery

On Lambda, concurrent invocations run in separate execution environments, each with its own /tmp. Two deliveries in flight at once never share a filesystem, so writing intermediate files to /tmp is safe from collisions by construction. Sequential warm invocations could share /tmp, but they did not write concurrently, so they could not collide. In the new service, all in-flight deliveries share one process and one /tmp – two deliveries with the same filename collide.

Anticipated before the migration. The question wasn't whether to fix it but how. We had two options for naming the intermediate files: generate a fresh UUID per event at the consumer, or use the delivery UUID from the source upstream. The time-series processing is idempotent at the database insertion level, so either approach would have worked for collision avoidance.

I used the delivery UUID for the destination directory. An atomic mkdir let the first delivery claim the path, so an overlapping duplicate on the same instance failed loudly. Cleanup preserved sequential replay.

None of this is novel – reusing the source's delivery UUID for downstream state is a standard pattern (Kleppmann covers it in Designing Data-Intensive Applications), and the temp path is just one more place to apply it. This applies only to the Python consumers. The Go extraction service holds intermediate state in memory and doesn't touch /tmp.

After the rollout, the choice paid off. We observed duplicate deliveries arriving from the upstream source – something the Lambda had been silently reprocessing. We added a guardrail at the producer level once we had evidence of the pattern.

A careful reader might ask why the guardrail went to the producer rather than the consumers, where idempotency is usually enforced. The answer is that the two layers guard against different things. At the producer, the same delivery should not be extracted twice at the same time – that's waste. At the consumers, we deliberately did not want to reject duplicates in general. Replaying from an older offset is how we recover from bugs, and a consumer that refuses anything it's seen before can't be replayed. The /tmp-path collision only catches concurrent duplicates – the same delivery in flight twice at once – which is exactly the scope we wanted. Sequential redelivery stays possible, the database-level idempotency makes it safe.

2. SQLAlchemy engine cache growth across requests

The first sign was premature restarts. The cause was OOM, but not from a spike – memory had grown gradually over hours.

Memory climbing toward the 4 GiB per-pod request, then OOM-killed and restarted. The sawtooth is the cycle repeating. Each line is a separate pod replica.

The first Lambda migrated had used a SQLAlchemy DB pool created per request. We anticipated this and switched to a persistent pool at service startup. What we didn't anticipate was that another code path implicitly created six DB pools per request, each from a different call site – a misuse of internal library code that Lambda had been concealing. The engine was being disposed correctly, which made the cause non-obvious.

Diagnosing the growth was a matter of adding tracemalloc and simulating long-running traffic. The traces pointed back to SQLAlchemy, but indirectly.

The accumulator was the SQLAlchemy compiled-statement cache. Each SQLAlchemy engine keeps its own cache of compiled SQL statements, keyed by the statement text. pandas.to_sql() populates that cache aggressively, because pandas generates a different SQL statement every time the column list or chunk size changes – a different column count means a different INSERT INTO ... (cols) VALUES (?, ?, ...), and each variant gets its own cache entry. With six engines per request instead of one, the problem multiplied. Six caches accumulated where there should have been one. The cache lives on the engine itself, not the connection pool, so dispose() released the pool resources but left the cache attached to the engine. At our request rate, those short-lived engines and their caches added up faster than Python reclaimed them.

This is memory bloat, not a memory leak in the strict sense – the cache has a bound. But the bound is high enough that, at our scale, it produces the same observable behavior as a leak. There's a discussion thread in the SQLAlchemy issue tracker where others have hit the same pattern.

For the services that talk to a single database, the fix was to make the engine global and let the cache stabilize at a steady-state size. Memory immediately flattened.

Same workload, same pod, after making the SQLAlchemy engine global – memory stabilized.

This bug mattered much less on Lambda. Concurrent invocations ran in separate execution environments, and each environment handled one invocation at a time. In the service, many deliveries created short-lived engines inside the same long-running process, so their caches accumulated together.

3. RabbitMQ consumers that don't reconnect

Lambda doesn't need to consume from RabbitMQ. The services do.

The weekend after rollout, the first broker hiccup hit, and the consumer died. It never reconnected.

The fix sat across two layers: our code and aio-pika. Our code created each queue inside a small helper function and let the reference go the moment the function returned, so Python garbage-collected the queue. aio-pika's RobustChannel tracked exchanges and queues in a WeakSet, on the assumption that user code would hold strong references. When connections dropped and our queues had no live references, they were garbage-collected and aio-pika had nothing to restore on reconnect. The contract of robust=True (re-declare on reconnect) silently broke when the user code happened to drop its reference.

The upstream fix changed the tracking from WeakSet to set:

Replace defaultdict(WeakSet) with defaultdict(set) for both exchanges and queues. Users who don't want exchanges/queues to be restored must now explicitly set robust=False. – commit

Once we picked up the new version of aio-pika, the reconnect path behaved as expected.

This was not a latent Lambda bug. It was a new responsibility introduced by leaving Lambda's push-based delivery model. SNS pushed events straight to the function, so there was no long-lived connection to drop and no reconnection to get wrong. The long-lived consumer connection, and the entire reconnection contract that comes with it, exist only in the long-running services.

The other lesson here is more boring: keep your dependencies current. The fix already existed upstream by the time we hit the bug – we just hadn't picked it up.

4. HTTP requests without retries

The Lambda made outbound HTTP calls to upstream services. Some of those calls didn’t have retry logic around them.

On Lambda, the runtime handled this for us. SNS invokes Lambda asynchronously, and Lambda retries a failed async invocation on its own. A network blip failed a call. Lambda ran the invocation again. A later attempt went through, and the application code never had to think about it.

The service has no equivalent. A failed HTTP call raised, the consumer logged the error, and the message went to the dead-letter queue or back to the broker depending on how that consumer was configured. Both outcomes are worse than what Lambda had been doing. Dead-letter needed manual replay. And requeuing to the broker redelivers on the broker's timing – not the quick, in-process retry-with-backoff a transient HTTP blip actually needs.

The fix was the obvious one: wrap the HTTP call in retry-with-backoff, give up after a few attempts, let the queue handle anything that exceeds that. The calls were GET requests, so aggressive in-process retry was safe.

Lambda’s async retries had been acting as invisible retry middleware. The retry policy had not disappeared. It had moved from the runtime into our codebase.

Technically, having retries on HTTP calls should have been done even on Lambda, but the runtime simply saved the code from having to care about it. Maybe it's one of the examples of why Lambda was a good choice for velocity.

5. Hammering upstream with HTTP requests

Some time after rollout, an upstream service started returning connection-refused errors. The Lambda had called the same upstream at higher total throughput without issues.

The reason was simple. The original code opened a new aiohttp.ClientSession for every outbound call, and closed it right after – no reuse:

async with aiohttp.ClientSession() as session:
    async with session.post(url, json=payload) as resp:
        ...

For a given number of in-flight deliveries, the migration did not create more connections. What changed was how many IPs those connections came from. On Lambda, traffic went out through three egress IPs. The service used only two. The upstream limits connections per source IP rather than in total, so the same load split across fewer IPs meant more connections per IP. At migration traffic we were still under the limit. As more customers moved over, the per-IP count crossed it and the upstream started refusing connections.

The fix was to create one aiohttp.ClientSession at service startup with a connection pool limit, and reuse it for every call. With a bounded pool, the connection count stayed below the upstream's per-source limit.

# at startup, once
session = aiohttp.ClientSession(
    connector=aiohttp.TCPConnector(limit=MAX_CONNECTIONS)
)

Lambda happened to spread these connections across enough IPs to stay under the upstream's per-source limit. Nobody planned that – it was just how many IPs the subnets had. The service used fewer IPs, and suddenly the limit mattered.

6. Async code that wasn't really async

The cases so far came from the first of the four consumers. The second consumer was more data-intensive than the first – larger upserts, more DB pools, more work per delivery. The symptom was Kubernetes pod restarts from failing liveness probes. The process was alive, it just wasn't responding.

The cause was sync I/O inside async code. One DB driver the service depended on had no async alternative, and the upserts it performed were large enough that blocking the event loop for their duration was measurable. While an upsert was running, nothing else got scheduled – including the probe.

Lambda's execution model had been making it invisible. Each environment handled one invocation at a time, and there was no Kubernetes liveness probe competing for the event loop. The sync call wasn't blocking anything because there was nothing else to block.

asyncio.to_thread() moved the blocking calls to Python's default thread pool.

When all workers were busy, additional upserts waited in the queue while the event loop remained responsive. Async code is only as async as its slowest sync call.

7. DLQ replay that took the pod down

Once the second consumer was stable under normal traffic, we triggered a DLQ replay to reprocess a batch of failed messages. The pod went OOM and Kubernetes restarted it. On restart, the unacked messages re-delivered and the pod went down again.

DLQ replay cascading into OOM pod restarts.

All Python consumers were sized the same way – same pod count, same processes per pod, same prefetch per process. The total concurrency was set to roughly match Lambda's per-function concurrency cap. The shape mirrored Lambda intentionally.

What didn't transfer was the isolation. On Lambda, each concurrent invocation had its own execution environment with its own memory and CPU budget. On the service, all those concurrent messages share one process's memory. As long as per-message work was light, the math held. The second consumer did significantly more work per message than the first – larger pandas dataframes, more sync DB calls, more bytes in flight. Under normal traffic the queue depth stayed low and the prefetch buffer never filled. The DLQ replay filled it instantly, all those heavy upserts ran concurrently in one process, and the pod ran out of memory before any of them finished.

The fix was to lower the second consumer's prefetch proportionally to its heavier per-message cost. Fewer messages in flight, smaller memory footprint, pod stayed up.

Lambda had been absorbing this too, but in a different way than the earlier cases. It wasn't hiding latent code, it was giving every message its own memory budget. Copying Lambda's concurrency math without Lambda's memory isolation turned the second consumer's heavier messages into a memory bomb.

8. Credentials that expired mid-process

A consumer talking to a single database started failing intermittently. Some deliveries synced cleanly. Others hit auth errors. Same code path, same delivery shape, different outcome. The pattern looked random, which made it harder to reason about than a clean failure would have been.

The cause was a mismatch between two lifetimes. The process lived indefinitely, while the credentials lived one month. The service used Vault-issued credentials and fetched them once at startup. Nothing refreshed them. The intermittent pattern was the connection pool – deliveries reusing connections opened before expiry succeeded, deliveries opening new connections failed. The bug had a deterministic cause but presented as noise.

The mental model was that the credentials were static. On Lambda this was effectively true. Nobody designed refresh logic because nobody framed the credential as something that needed refreshing.

The migration didn't change the credential. It changed the lifetime ratio. "Static" didn't mean permanent – it meant a longer rotation interval: long enough that a Lambda never met expiry, short enough that an always-on process does.

The immediate fix was a pod restart. The longer-term fix was a background refresh task with atomic update to the in-memory credential reference, so in-flight requests don't see a torn state.

Lambda's lifetime was shorter than every credential's TTL. The application code never had to think about refresh because the process never lived long enough for refresh to matter. The general pattern: any time-bounded resource the runtime was hiding – credentials, signed URLs, OAuth tokens, TLS certs – becomes application code once the runtime stops hiding it.

Re-fetching credentials on auth error would also have worked: retry once with fresh credentials, and if it still fails something is broken. Scheduled refresh has its own benefits. Rotation gets handled ahead of time instead of being discovered through failed connections, and an auth error during normal operation stays a signal that something is actually wrong.

Lambda as a runtime contract

None of this is an argument against Lambda – the same kind of contract exists under any runtime, including the one we moved to.

What this migration made visible is that Lambda is a runtime contract, not just a deployment target. The contract says: each concurrent execution environment gets isolated filesystem and memory, any in-process pools and caches are scoped to that environment, SNS-triggered work gets a retry policy outside your code, and the environment's lifetime is short enough that many accumulators never reach steady state. Most of this is documented somewhere – the filesystem and memory isolation, even the Lambda retries. Some of it isn't, like your code inheriting its own pools and caches. But none of it is gathered into the one list that matters – the set of guarantees your code is quietly relying on.

Code written under that contract can rely on it without naming it. The reliance shows up as the absence of code – no reconnect logic, no per-process work limit, no cache eviction, no retry wrappers, no concurrency cap on bulk replays. When the contract changes, the absences become bugs.

The Law of Leaky Abstractions says what's underneath your abstraction eventually shows through. With Lambda it went the other way. Our bugs disappeared into the runtime. You still have to learn what the abstraction hides. Lambda just let us postpone it.

The practical test for anyone planning a similar migration is to list every guarantee your current runtime provides that your code doesn't explicitly request. Each item is a candidate bug under the new runtime. Each of the eight cases above came from a line on it.

For years the system ran on Lambda with these assumptions intact – some of them latent bugs, some responsibilities Lambda had been handling for us. The runtime never made any of it visible. It just absorbed it, and nobody noticed. Any production system on a managed runtime is making the same bet, whether the team running it knows it or not.

Appendix A: The eight, at a glance