Having previously been on several managed PostgreSQL providers and now on AWS Aurora -- Aurora has been pretty great in terms of reliability and performance with large row counts and upsert performance.
However, Aurora isn't cheap and is at least ~80% of our monthly AWS bill. I wonder how it is cheaper than Heroku's previous offerings? Is it Aurora Serverless v2 or something like that to reduce cost? Aurora billing is largely around IOPS, and Heroku's pricing doesn't seem to reflect that.
Heroku Postgres has always been priced on platform convenience with very high margins. It's been many years now so I don't remember the exact numbers, but I moved a few databases from Heroku to AWS and reduced my DB costs ~90% (magnitude ~900/mo --> ~100/mo) for roughly the same specs. They probably have a lot of margins to eat into before they need to adjust prices.
We're using the highest tier Postgres instance at my work for one of our legacy Heroku apps and it costs thousands over what we'd pay for the equivalent on AWS directly.
I'm assuming this means that they are not providing any sort of guarantee on the amount of RAM available and packing these instances as tightly as they can.
“Amazon Aurora Serverless is an on-demand, autoscaling configuration for Amazon Aurora. It automatically starts up, shuts down, and scales capacity up or down based on your application's needs.”
Everything on Heroku is billed with a huge margin, plus as they're probably a partnered customer by now their pricing is a fraction of the average AWS customers pricing. I've been at companies on both sides of the partner pricing list and the difference is huge.
Aurora has treated us well. We make a self-hosted product that requires Postgres; our sales/customer engineering folks just started telling people to use Aurora, and it hasn't caused any problems despite the fact that all of our tests run against stock Postgres. Can't complain. Though a VM with Postgres would be plenty for our needs, and cost thousands of dollars less a month. But, HA is nice if you want to pay for it.
Yeah, that’s what we use as well but I don’t think that addresses the underlying instance cost? I’m not familiar with Serverless v2 though, if that’s what this is using.
Just curious, does Aurora scale down at all in price, i.e. if I have a test instance that's hardly ever used, does it ever end up being cheaper than a classic RDS instance?
Xata is (like Heroku) based on Aurora, but offers database branching and has different pricing. That should be ideal for lightly-used test instances, because you only pay for storage, and 15GB are included in the free tier.
You're thinking of Aurora Serverless, but the typical Aurora customer isn't using the Serverless offering. Additionally, the original version of Aurora Serverless scaled to 0, but v2 doesn't.
Came here to say this. Aurora is good but also very expensive. If your queries are not very well tuned you will pay through the nose for I/O that would be unnoticeable in other Postgres implementations. For a very modest installation I saw DB bills go down from $3,000 a month on Aurora to about $100 for self managed Postgres.
“Not hard” is very relative. Is it hard to run a Postgres database? No.
Is it hard to set up monitoring, automatic failover (without data loss), managed upgrades, tx wraparound protection, clustering and endpoint management, load balancing… and probably a bunch of other things I’m not thinking of? Yes.
So figure it out. I don’t understand why “ugh this is hard, I’ll pay someone else” has become the norm. You’re working in one of the most technically advanced fields in the world; act like it.
Most people aren't doing anything advanced. Also this has nothing to do with not wanting to do "hard things", that's ridiculous, it's a postgres cluster, you're not doing a PhD in math. People do it because there's limited time and no business advantage to operate postgres clusters. Use the time on what your business actually does.
> that's ridiculous, it's a postgres cluster, you're not doing a PhD in math.
It's not as difficult as a PhD (I assume; I only got as far as an MS), but based on what I've witnessed, it's up there in complexity. There are dozens of knobs to turn – not as many as MySQL/InnoDB to be fair, but still a lot – things you have to know before they matter, etc.
> People do it because there's limited time and no business advantage to operate postgres clusters. Use the time on what your business actually does.
I've seen this argument countless times for SaaS anything. I don't think it's accurate for a database. Hear me out.
For most companies, the DB is the heart. Everything is recorded there, nearly every service's app needs it (whether it's a monolith or micro service-oriented DBs), and it's critically important to the company's survival. Worse, the same skills necessary for operating your own DB generally overlap heavily with optimally running a DB, by which I mean if you're good at things like DB backup automation, chances are you're also good at query optimization, schema design, etc.
It's that latter part that seems to be missing from many engineering orgs. "Just use Postgres," people say; "just add a JSONB column and figure out the schema later," but later never comes. If your business uses a DB, then you do not have the luxury of running one poorly. Spend a few days learning SQL, it's an easy language to pick up. Then spend a few days going through the docs for your DB, and try the concepts out in a test instance. Your investment will be rewarded.
You're at a more surface level than what I'm talking about. Your advice at the end is just common sense advice for anyone using any tool. It doesn't mean you should spend time implementing your own custom backup process with ability to go back to a specific point in time, configurable in 1 minute. The amount of work needed to operationalize postgres in the same way and expose it to other teams in a company will take long enough that you won't get an Aurora-like experience in less than a quarter with a full team. What could they be doing instead to your product?
All I'm saying is it has nothing to do with difficulty. In my job for example we self-hosted HBase which is a beast compared to postgres, implemented custom backups etc, all because there was no good vendor for it. Postgres is much simpler and we always just used RDS and then switched to Aurora for the higher disk limits when it was launched. If there's a good enough vendor, you're just stroking your ego re-implementing these things when you could move on to the actual thing the business wants to release.
I've also seen senior engineering leads "proving" self hosting "saves money" but then 2 companies working on the same type of problem in the same industry with a similar feature set, on one side we had 5 people maintaining what on the other company it took 6 teams of 4-8 people. So it depends if you'd like to have a lot of your labor focused on cutting costs or increasing revenue. And they never include the cost of communicating with extra 5 teams and the increased complexity and slowness to release things this creates, while also being harder to keep databases with current versions, more flimsy backup processes, etc.
Ps: we got rid of hbase, do yourself a favor and stay away
> Your advice at the end is just common sense advice for anyone using any tool.
Common sense isn't so common. I've met a handful of devs across many separate companies who care at all how the DB works, what normalization is, and will read the docs.
> It doesn't mean you should spend time implementing your own custom backup process with ability to go back to a specific point in time, configurable in 1 minute.
If by implement you mean write your own software, no, of course not. Tooling already exists to handle this problem. Off the top of my head, EDB Barman [0] and Percona XtraBackup [1] can both do live backups with streaming so you can backup to a specific transaction if desired, or a given point in time.
Or, if you happen to have people comfortable running ZFS, just snapshot the entire volume and ship those off with `zfs send/recv`. As a bonus, you'll also get way more performance and storage out of a given volume size and hardware thanks to being able to safely disable `full_page_writes` / `doublewrite_buffer`, and native filesystem compression, respectively.
> If there's a good enough vendor, you're just stroking your ego re-implementing these things when you could move on to the actual thing the business wants to release.
Focusing purely on releasing product features, and ignoring infrastructure is how you get a product that falls apart. Ignoring the cost of infrastructure due to outsourcing everything is how you get a skyrocketing cloud bill, with an employee base that is fundamentally unable to fix problems since "it's someone else's problem."
> Ps: we got rid of hbase, do yourself a favor and stay away
HBase and Postgres are not the same thing at all. If you need the former you'll know it. If people convince management that they do need it when they don't, then yeah, that's gonna be a shitty time. The same is true of teams who are convinced they need Kafka when they really just need a queue.
My overall belief, which has been proven correct at every company I've worked at, is that understanding Linux fundamentals and system administration remains an incredibly valuable skill. Time and time again, people who lack those skills have broken things that were managed by a vendor, and then were hopelessly stuck on how to recover. But hey, the teams had higher velocity (to ship products with poor performance).
Have you ever been paid to do work before? There’s a price at which a business will prefer to pay to have SaaS/PaaS solve a problem. Allocating engineering hours to setting up and maintaining a Postgres cluster has a cost. You’ll want someone senior on it. Their time could be well over $100/hour. And that’s assuming your business is small enough to only need one DBA part time. A business that’s spending a ton on Aurora might need 3 specialists. Now you’re talking about hundreds of thousands of dollars per year. It could be better to just pay AWS.
However, at large scales cloud won’t make sense anymore. They do have a markup and eventually what you’re paying in markup could instead buy you a few full time employees.
Yes, many times, which is why I've developed this opinion.
> However, at large scales cloud won’t make sense anymore. They do have a markup and eventually what you’re paying in markup could instead buy you a few full time employees.
The issue is once you've finally realized this stuff matters, and have hired a DB team, I can practically guarantee that your schema is a horror show, your queries are hellish, and your product teams have neither the time nor inclination to unwind any of it. Your DB{A,RE}s are going to spend months in hell as they are suddenly made the scapegoats for every performance problem, and are powerless to fix anything, since their proposals require downtime, too much engineering effort, or both.
Hence my statement. Learn enough about this stuff so that when you do hire in specialists, the problems are more manageable.
You need to do things that are appropriate for a small company when you’re a small company. And then if you become a large company you change things to suit your new scale.
All of the troubles you described sound like bad management. I’m sorry if you’ve had to go through that. DBAs that are setting up a replacement are going to need time to do that right and expectations need to be set that this is a tricky problem.
You clearly don’t know what aurora is or does if you think people can just run their own. It’s not a regular Postgres setup, and nothing exists that’s equivalent for self hosted.
However, Aurora isn't cheap and is at least ~80% of our monthly AWS bill. I wonder how it is cheaper than Heroku's previous offerings? Is it Aurora Serverless v2 or something like that to reduce cost? Aurora billing is largely around IOPS, and Heroku's pricing doesn't seem to reflect that.