From inevitable overprovisioning to the “on-demand” tax: why DynamoDB is bloody hard to cost-control From inevitable overprovisioning to the “on-demand” tax: why DynamoDB is bloody hard to cost-control I recently built a DynamoDB cost calculator with the specific goal of helping potential ScyllaDB customers understand the true cost of running DynamoDB. Now, if you step back and look at my goal, it doesn’t make much sense, right? If somebody is already using DynamoDB, wouldn’t they already know how much it costs to run the technology at scale? built a DynamoDB cost calculator Naively, this is what I thought too, at first. But then, I started to peel back the inner workings of DynamoDB cost calculations. At that point, I realized that there are many reasons why teams end up paying hundreds of thousands (if not millions) of dollars to run DynamoDB at scale. The main thing I found: DynamoDB is easy to adopt, but bloody hard to cost-control. My workmate Guilherme and I delivered a webinar along these lines, but if you don’t have time to watch, read on to discover the key findings. delivered a webinar along these lines The first common misunderstanding is precisely what DynamoDB charges you for. You’ve probably already heard terms like Read Capacity Units and Write Capacity Units, and get the gist of “You pay for what you use” in terms of number of reads and writes. But let’s start with the basics. DynamoDB writes are expensive… If you look at pricing for on-demand capacity, you’ll see that a read request unit (RRU) costs $0.125 per million units, and a write request unit (WRU) costs $0.625 per million units. So, writes are 5 times more expensive than reads. I don’t know the exact technical reason, but it’s no doubt something to do with the write path being heavier (durability, consistency, indexing etc) and perhaps some headroom. 5x does seem a bit on the steep side for databases and one of the first traps from a cost perspective. You can easily find yourself spending an order of magnitude more if your workload is write-heavy, especially in on-demand mode. pricing for on-demand capacity Speaking of which…there’s the other mode: provisioned capacity. As the name suggests, this means you can specify how much you’re going to use (even if you don’t use it), and hopefully pay a bit less. Let’s check the ratio though. A Read Capacity Unit (RCU) costs $0.00013 per RCU and a Write Capacity Unit (WCU) costs $0.00065, so writes are unsurprisingly 5 times more expensive than reads. So even in provisioned mode, you’re still paying a 5x penalty on writes. Thus, is significant, especially for high-volume write workloads. No provisioned discount on writes for you! provisioned capacity You’re not provisioning requests, you’re provisioning rates… Here’s the catch: provisioned capacity units are measured per second, not per million requests, like in on-demand. That tripped me up initially. Why not just provision the total number of requests? But from AWS’s perspective, it makes perfect business sense. You’re paying for the ability to handle N operations per second, whether you use that capacity or not. the ability to handle So if your traffic is bursty, or you’re over provisioning to avoid request throttling (more on that in a bit), you’re essentially paying for idle capacity. Put simply, you’re buying sustained capacity, even if you only need it occasionally. Just like my gym membership 😉 Reserved capacity… So here’s the deal: if you reserve capacity, you’re betting big upfront to hopefully save a bit later. If you’re confident in your baseline usage, AWS gives you the option to reserve DynamoDB capacity, just like with EC2 or RDS. It’s a prepaid 1 or 3 year commitment, where you lock in a fixed rate of reads and writes per second. And yes, it’s still a rate, not a total number of requests. One gotcha: there’s no partial upfront option; it’s pay in full or walk away. Let’s look at a simple use case to compare the pricing models… Say your workload averages 10,000 reads/sec and 10,000 writes/sec over an hour. On-Demand pricing: Writes: $22.50/hr … 10,000 * 3600 * 0.625 / 1MReads: $4.50/hr … 10,000 * 3600 * 0.125 / 1M Writes: $22.50/hr … 10,000 * 3600 * 0.625 / 1M Reads: $4.50/hr … 10,000 * 3600 * 0.125 / 1M (5x cheaper than writes, as usual) Provisioned pricing (non-reserved): Writes: $6.50/hr … 10,000 * $0.00065Reads: $1.30/hr … 10,000 * $0.00013 Writes: $6.50/hr … 10,000 * $0.00065 Reads: $1.30/hr … 10,000 * $0.00013 Provisioned with 1-Year Reserved: Writes: ~$2.99/hrReads: ~$0.59/hr“Hey, where’s the reserved math?” I hear you. Let’s just say: Writes: ~$2.99/hr Reads: ~$0.59/hr “Hey, where’s the reserved math?” I hear you. Let’s just say: You take the reserved pricing for 100 WCUs ($0.0128/hr) and RCUs ($0.0025/hr), divide by 730 hours in a month, divide by 12 months in a year, divide again by 100 units, multiply by your needed rate… then round it, cry a little, and paste in the “math lady” meme. Or better yet, use our calculator. My point is: Provisioned is ~3.4x cheaper than on-demandReserved is ~7.5x cheaper than on-demandOn-demand is for people who love overpaying, or loathe predicting Provisioned is ~3.4x cheaper than on-demand Reserved is ~7.5x cheaper than on-demand On-demand is for people who love overpaying, or loathe predicting Btw, AWS recommends on-demand for: AWS recommends on-demand Traffic patterns that evolve over timeSpiky or batchy workloadsLow utilization (drops to zero or below 30% of peak) Traffic patterns that evolve over time Spiky or batchy workloads Low utilization (drops to zero or below 30% of peak) Which is basically every real-life workload — at least for the customers of ScyllaDB. So yes, expect to pay a premium for that flexibility unless your traffic looks like a textbook sine wave and you have a crystal ball. It’s not the size of the item, but it is… Here’s another trap. It’s one that you might not hit until you use real application data…at which point you’ll immediately regret overlooking it. In DynamoDB, you don’t just pay per operation; you pay per chunk of data transferred. And the chunk sizes differ between reads and writes: Writes are billed per 1KB (Write Request Units or WRUs)Reads are billed per 4KB (Read Request Units or RRUs) Writes are billed per 1KB (Write Request Units or WRUs) Reads are billed per 4KB (Read Request Units or RRUs) So if you write a 1.1KB item, that’s 2 WRUs. Write a 3KB item? Still 3 WRUs, every 1KB (or part thereof) gets counted. Reads work the same way, just at 4KB boundaries. Read a 1KB item? 1 RRU. Read a 4.1KB item? That’s 2 RRUs. Isn’t rounding up fun? I’m sure there’s strong technical reasons for these boundaries. You can see the trap here. Combine this with the 5x cost of a write compared to a read, and things can get nasty quickly, especially if your item size straddles those thresholds without you realizing. It’s probably ok if you have a fixed item size in your schema, but definitely not ok with the types of use cases we see at ScyllaDB. For example, customers might have nested JSON or blob fields which can shrink or grow with usage. And remember, it’s actual item size, not just logical schema size. Overprovisioning, because you have to … Another pain point, and devious omission from AWS’s own calculator, is the need to overprovision when using provisioned capacity. It sounds counterintuitive, but you’re forced to overprovision – not because you want to, but because DynamoDB punishes you if you don’t. In provisioned mode, every request is subject to strict throughput limits because, if you recall earlier, a fixed rate is what you’re paying for. If you slide past the provisioned capacity, you’ll hit ProvisionedThroughputExceededException. I love the clarity of this type of exception message. I don’t love what it actually does, though: request throttling. There’s a small 300s window of burst capacity that retains unused read and write capacity. But beyond that, your app just fails. ProvisionedThroughputExceededException 300s window of burst capacity So, the best way to counter this is to overprovision. By how much? That warrants an “it depends” answer. But it does depend on your workload type. We added this functionality to our calculator so you can dynamically overprovision by a percentage, just to factor in the additional costs to your workload. Obviously, these costs can add up quickly because in practice, you’re paying for the peak even if you operate in the trough. If you don’t provision high enough capacity, your peaks risk being throttled, giving you customer-facing failures at the worst possible time. Before we move on … If there’s a recurring theme here, it’s this: DynamoDB’s pricing isn’t inherently wrong. You do pay for what you use. However, it’s wildly unforgiving for any workload that doesn’t look like a perfect, predictable sine wave. Whether it’s: The 5x write cost multiplierThe 7.5x on-demand cost multiplierOpaque per-second provisioned ratesPunitive rounding and artificial boundaries of item sizesOr just the need to overprovision to avoid face-planting during peak load The 5x write cost multiplier The 7.5x on-demand cost multiplier Opaque per-second provisioned rates Punitive rounding and artificial boundaries of item sizes Or just the need to overprovision to avoid face-planting during peak load …You’re constantly having to second guess your architecture just to stay ahead of cost blowouts. The irony? DynamoDB is branded as “serverless” and “fully managed” yet you end up managing capacity math, throttling errors, arcane pricing tiers, and endless throughput gymnastics. Having observed many of our customer’s spreadsheet forecasts (and AWS Cost Explorer exports) for DynamoDB, even mature teams running large-scale systems have no idea what the cost is…until it’s too late. That’s why we built a calculator that models real workloads, not just averages. Because the first step to fixing costs is to understand where they’re coming from. In my next blog post, I walk through some real-world examples of customers that switched from DynamoDB to ScyllaDB to show the true impact of traffic patterns, item sizes, caches and multi region topologies. Stay tuned or skip ahead and model your own workloads at calculator.scylladb.com. my next blog post skip ahead and model your own workloads calculator.scylladb.com Model your own DynamoDB workloads on our new cost calculator Model your own DynamoDB workloads on our new cost calculator About Tim Koopmans About Tim Koopmans Tim has had his hands in all forms of engineering for the past couple of decades with a penchant for reliability and security. In 2013 he founded Flood IO; a distributed performance testing platform. After it was acquired, he enjoyed scaling the product, business and team before moving on to other performance-related endeavors.
