In 2022 I replaced the cryptographic library under DynamoDB’s TLS termination — OpenSSL 1.0.2 out, AWS-LC in — and deployed it across every AWS region with zero downtime. Customers noticed nothing except faster handshakes: median TLS handshake time dropped 12–35% depending on region, and FIPS endpoints dropped as much as 80%. My work was featured in the AWS re:Inforce 2022 keynote. This post is the internal ops-win announcement rewritten for the public, including the part these announcements usually bury: how hard it is to swap the crypto library under a database that is never allowed to go down — and the two ways the rollout bit back.
Where TLS terminates in DynamoDB
Every DynamoDB request arrives over TLS. Termination happens on-host: each request-router host runs an Nginx-based TLS termination stack in front of the router process, and in 2022 that stack was built on OpenSSL 1.0.2. (The request-router fleet is public architecture — see the DynamoDB USENIX ATC ‘22 paper.)
flowchart LR
C["Client SDK"] -->|TLS| T["On-host TLS termination<br/>Nginx + crypto library"]
T --> R["Request router"]
R --> S["Storage nodes"]
T -.->|"this box is the post"| T
The fleet serves hundreds of billions of requests per hour in the largest region alone. About 99% of those ride existing connections; roughly 1% are new connections that need a full TLS handshake — certificate verification, cipher negotiation, key exchange. That 1% matters more than it sounds: a median GetItem is about 3 ms round trip, and a full handshake added about 6 ms on top. If your request is the one opening the connection, the handshake is your latency.
The trigger: Graviton made it worse
The fleet migrated from M5 (x86) to M6g (Graviton, ARM) instances. P99 request latency regressed. The root cause was new-connection establishment: TLS handshakes got slower on ARM, because OpenSSL 1.0.2’s ARM performance wasn’t good. The handshake tax that had been tolerable on x86 stopped being tolerable.
One extra constraint shaped everything: government customers use FIPS endpoints with different TLS configurations, and any fix had to keep FIPS compliance.
Why AWS-LC and not OpenSSL 1.1.1
I benchmarked two candidates: upgrade to OpenSSL 1.1.1, or switch to AWS-LC, AWS’s open-source fork of BoringSSL/OpenSSL. AWS-LC won on three grounds:
- One library for both stacks. The standard and FIPS termination stacks could share a single cryptographic library instead of maintaining two, which removes a whole class of future operational pain.
- OpenSSL 1.1.1 announced it would not provide FIPS support. That made it a dead end for the FIPS fleet on its own.
- ARM performance. AWS-LC’s ARM support beat OpenSSL 1.1.1 on Graviton hosts in my benchmarks — and Graviton was the reason this project existed.
End to end — benchmark, implement, test, deploy globally — took five months, working with the AWS-LC team. Global deployment completed September 12, 2022.
Why this was hard
“Swap the crypto library” sounds like changing a dependency line. It was five hard problems bundled into one project:
Modifying Nginx source. AWS-LC is not a drop-in replacement for OpenSSL 1.0.2. Nginx’s TLS layer is written directly against OpenSSL’s API, and making it speak to a different cryptographic library meant modifying Nginx source and its OpenSSL glue — then proving the modified stack behaved identically under production traffic patterns. This was surgery on the TLS path itself, not a version bump.
A legacy build system. The stack had been building against OpenSSL 1.0.2 for years, and the build had years of assumptions baked into it — header locations, link order, version checks, packaging. Getting Nginx to compile and link cleanly against a different crypto library through that legacy build chain was its own project, finished before any performance work could even be measured.
Amazon’s internal infrastructure. This isn’t a container you rebuild and redeploy. The termination stack ships through Amazon’s internal build, packaging, and deployment machinery to every region and partition — commercial, GovCloud — each with its own rollout and compliance constraints. And the FIPS fleet doubles the surface: same integration, different TLS configuration, government workloads on the other end.
Cryptography. Correctness here is not “the tests pass.” The cipher suites offered, the session caching semantics, the FIPS module behavior — all of it is a de facto contract with millions of clients, and a mistake either breaks connections at scale or quietly weakens security. One of those failure modes is loud; the other you might not find for years. Both had to be ruled out before the first production host saw the new library. Luckily I had good scientists to work with — the cryptographers behind AWS-LC — and I leaned on them for exactly these questions.
Deployment sensitivity. The fleet’s alarms are tuned tight enough that deployments themselves can trip them, and the session-cache difference meant each deployment wave generated a surge of full handshakes that looked exactly like a problem. The deployment had to be engineered as its own system — more on that below. Luckily I had great teammates who helped me monitor the deployment waves. Thanks, Shane and Robert.
Any one of these is a routine engineering problem. All five at once, under a database that is never allowed to blink, felt like mission impossible at the time.
The results
Median (p50) TLS handshake time, before and after, by region:
| Endpoint | p50 before | p50 after | Improvement |
|---|---|---|---|
| us-east-1 | 6.2 ms | 5.06 ms | 20% |
| ap-northeast-1 | 8.5 ms | 6.5 ms | 25% |
| eu-west-2 | ~8 ms | ~5.5 ms | 31% |
| us-gov-west-1 (FIPS) | 22 ms | 4.5 ms | 80% |
| us-gov-east-1 (FIPS) | 19 ms | 4.2 ms | 78% |
| us-east-1 (FIPS) | 4.25 ms | 1.7 ms | 42% |
Across all standard regions the p50 improvement ranged from 12% to 35%. The FIPS numbers are the striking ones: government customers went from a 22 ms handshake to 4.5 ms.
What bit us
Ops-win announcements usually stop at the table above. The lessons section is the useful part.
Cipher suite incompatibility. The TLS cipher suites in OpenSSL 1.0.2 and AWS-LC didn’t line up. Shipped naively, some clients would have failed to negotiate a connection mid-rollout. I caught it in testing and resolved it with the AWS-LC team before customers could be affected — but “swap the crypto library” is never just a swap. The library’s supported-cipher list is part of your public behavior.
Session caching behaved differently. AWS-LC handled TLS session caching differently from OpenSSL. My fix was to deploy in a way that reset the session cache — which meant a temporary surge of full handshakes on freshly deployed hosts, which meant deployment-sensitive alarms firing and real operational pain for the on-call. I shipped it, wrote up the incident honestly, and left a tracked follow-up to make the alarms deployment-aware. The general lesson: when you change a component, the cache behavior around it is part of the migration, not an implementation detail.
The hard part: global deployment, zero downtime
The results table is not the achievement. The deployment is.
DynamoDB has no maintenance window. It serves hundreds of billions of requests per hour, thousands of AWS services and customer systems sit on top of it, and government workloads depend on the FIPS endpoints. The cryptographic library sits in the path of every one of those requests — it runs before any request logic gets a chance to. That makes this the worst kind of change to ship: enormous blast radius, invisible when right, catastrophic when wrong. A subtly wrong cipher list means clients can’t negotiate a connection at all. Mishandled session caching means a wall of full handshakes slamming freshly deployed hosts. At this scale, either one is a global incident on one of the most depended-on services in the cloud — not a bad afternoon.
So the rollout had to be engineered as carefully as the code. Staged deployment wave by wave across hosts and regions, the session-cache reset built into the deployment procedure, alarms watched through every wave, rollback ready at every step. Benchmark, implement, test, and deploy took five months end to end, finishing globally on September 12, 2022. Downtime: zero. Customer-visible errors: zero. The only thing anyone outside could observe was that handshakes got faster.
That is the impact I’m proudest of. Every new connection to one of the largest databases on the planet got measurably quicker; FIPS users stopped paying a 22 ms tax they didn’t know they were paying; and the Graviton migration was unblocked, so the fleet kept its price-performance win without the handshake regression. The math the library computes never changed — every handshake is the same cryptography, computed faster, swapped in underneath a live database that was never allowed to blink.