In 2022 I replaced the cryptographic library under DynamoDB’s TLS termination — OpenSSL 1.0.2 out, AWS-LC in — and deployed it across every AWS region with zero downtime. Median TLS handshake time dropped 12–35% depending on region, and FIPS endpoints dropped as much as 80%. There were no customer-visible errors. My work was featured in the AWS re:Inforce 2022 keynote. This post is the internal ops-win announcement rewritten for the public, including what the integration required and the two problems the rollout surfaced.
Where TLS terminates in DynamoDB
Every DynamoDB request arrives over TLS. Termination happens on-host: each request-router host runs an Nginx-based TLS termination stack in front of the router process, and in 2022 that stack was built on OpenSSL 1.0.2. (The request-router fleet is public architecture — see the DynamoDB USENIX ATC ‘22 paper.)
flowchart LR
C["Client SDK"] -->|TLS| T["On-host TLS termination<br/>Nginx + crypto library"]
T --> R["Request router"]
R --> S["Storage nodes"]
T -.->|"this box is the post"| T
The fleet serves hundreds of billions of requests per hour in the largest region alone. About 99% of those reuse existing connections; roughly 1% are new connections that need a full TLS handshake — certificate verification, cipher negotiation, key exchange. A request that opens its own connection pays the full handshake cost on top of the normal round trip, and that handshake cost was larger than the round trip itself.
Trigger: the M5 to M6g migration
The fleet migrated from M5 (x86) to M6g (Graviton, ARM) instances. P99 request latency regressed. The root cause was new-connection establishment: TLS handshakes got slower on ARM, because OpenSSL 1.0.2’s ARM performance was poor.
One further constraint applied to any fix: government customers use FIPS endpoints with different TLS configurations, and FIPS compliance had to be maintained.
Choosing between OpenSSL 1.1.1 and AWS-LC
I benchmarked two candidates: upgrade to OpenSSL 1.1.1, or switch to AWS-LC, AWS’s open-source fork of BoringSSL/OpenSSL. I selected AWS-LC for three reasons:
- One library for both stacks. The standard and FIPS termination stacks could share a single cryptographic library instead of maintaining two, which removes one source of ongoing maintenance.
- OpenSSL 1.1.1 announced it would not provide FIPS support. That ruled it out for the FIPS fleet on its own.
- ARM performance. AWS-LC’s ARM support beat OpenSSL 1.1.1 on Graviton hosts in my benchmarks, and the Graviton migration was the reason for the project.
End to end — benchmark, implement, test, deploy globally — took five months, working with the AWS-LC team. Global deployment completed September 12, 2022.
What the integration required
The change involved five separate problems:
Modifying Nginx source. AWS-LC is not a drop-in replacement for OpenSSL 1.0.2. Nginx’s TLS layer is written directly against OpenSSL’s API, so targeting a different cryptographic library meant modifying Nginx source and its OpenSSL glue, then demonstrating that the modified stack behaved identically under production traffic patterns.
A legacy build system. The stack had been building against OpenSSL 1.0.2 for years, and the build carried assumptions from that period — header locations, link order, version checks, packaging. Getting Nginx to compile and link cleanly against a different crypto library through that build chain was its own work item, completed before any performance could be measured.
Amazon’s internal infrastructure. The termination stack ships through Amazon’s internal build, packaging, and deployment machinery to every region and partition — commercial, GovCloud — each with its own rollout and compliance constraints. The FIPS fleet doubles that surface: the same integration with a different TLS configuration, serving government workloads.
Cryptography. The cipher suites offered, the session caching semantics, and the FIPS module behavior form a de facto contract with millions of clients. An error there either breaks connections at scale or weakens security without producing a visible failure. Both outcomes had to be ruled out before the first production host ran the new library. I worked through these questions with the cryptographers behind AWS-LC.
Deployment sensitivity. The fleet’s alarms are tuned tight enough that deployments themselves can trip them, and the session-cache difference meant each deployment wave generated a surge of full handshakes with the same signature as a fault. The deployment procedure had to be designed as its own system, described below. Shane and Robert monitored the deployment waves with me.
The results
Reduction in median (p50) TLS handshake time, by region:
| Endpoint | p50 reduction |
|---|---|
| us-east-1 | 20% |
| ap-northeast-1 | 25% |
| eu-west-2 | 31% |
| us-gov-west-1 (FIPS) | 80% |
| us-gov-east-1 (FIPS) | 78% |
| us-east-1 (FIPS) | 42% |
Absolute before-and-after latencies are omitted; only the relative reduction is shown.
Across all standard regions the p50 improvement ranged from 12% to 35%. The FIPS endpoints improved the most.
Two problems the rollout surfaced
Cipher suite incompatibility. The TLS cipher suites in OpenSSL 1.0.2 and AWS-LC didn’t line up. Deployed as-is, some clients would have failed to negotiate a connection mid-rollout. I found this in testing and resolved it with the AWS-LC team before customers could be affected.
Session caching behaved differently. AWS-LC handled TLS session caching differently from OpenSSL. I deployed in a way that reset the session cache, which produced a temporary surge of full handshakes on freshly deployed hosts, which in turn tripped deployment-sensitive alarms and added on-call load. I shipped it, wrote up the incident, and filed a tracked follow-up to make the alarms deployment-aware.
Global deployment, zero downtime
DynamoDB has no maintenance window. It serves hundreds of billions of requests per hour, thousands of AWS services and customer systems run on top of it, and government workloads depend on the FIPS endpoints. The cryptographic library runs in the path of every request, before any request logic executes, so a defect in it affects the entire service. An incorrect cipher list means clients cannot negotiate a connection at all. Mishandled session caching means a surge of full handshakes against freshly deployed hosts. At this scale either failure is a global incident.
The rollout was therefore engineered as its own system: staged wave by wave across hosts and regions, with the session-cache reset built into the deployment procedure, alarms watched through every wave, and rollback available at each step. Benchmark, implement, test, and deploy took five months end to end, finishing globally on September 12, 2022. Downtime was zero and there were no customer-visible errors. The only externally observable change was faster handshakes.
Every new connection to one of the largest databases in operation got measurably faster, with the largest reduction on the FIPS endpoints. The Graviton migration was unblocked, so the fleet kept its price-performance gain without the handshake regression. The cryptography itself is unchanged; the same computations run faster.