So figure it out. I don’t understand why “ugh this is hard, I’ll pay someone else” has become the norm. You’re working in one of the most technically advanced fields in the world; act like it.
Most people aren't doing anything advanced. Also this has nothing to do with not wanting to do "hard things", that's ridiculous, it's a postgres cluster, you're not doing a PhD in math. People do it because there's limited time and no business advantage to operate postgres clusters. Use the time on what your business actually does.
> that's ridiculous, it's a postgres cluster, you're not doing a PhD in math.
It's not as difficult as a PhD (I assume; I only got as far as an MS), but based on what I've witnessed, it's up there in complexity. There are dozens of knobs to turn – not as many as MySQL/InnoDB to be fair, but still a lot – things you have to know before they matter, etc.
> People do it because there's limited time and no business advantage to operate postgres clusters. Use the time on what your business actually does.
I've seen this argument countless times for SaaS anything. I don't think it's accurate for a database. Hear me out.
For most companies, the DB is the heart. Everything is recorded there, nearly every service's app needs it (whether it's a monolith or micro service-oriented DBs), and it's critically important to the company's survival. Worse, the same skills necessary for operating your own DB generally overlap heavily with optimally running a DB, by which I mean if you're good at things like DB backup automation, chances are you're also good at query optimization, schema design, etc.
It's that latter part that seems to be missing from many engineering orgs. "Just use Postgres," people say; "just add a JSONB column and figure out the schema later," but later never comes. If your business uses a DB, then you do not have the luxury of running one poorly. Spend a few days learning SQL, it's an easy language to pick up. Then spend a few days going through the docs for your DB, and try the concepts out in a test instance. Your investment will be rewarded.
You're at a more surface level than what I'm talking about. Your advice at the end is just common sense advice for anyone using any tool. It doesn't mean you should spend time implementing your own custom backup process with ability to go back to a specific point in time, configurable in 1 minute. The amount of work needed to operationalize postgres in the same way and expose it to other teams in a company will take long enough that you won't get an Aurora-like experience in less than a quarter with a full team. What could they be doing instead to your product?
All I'm saying is it has nothing to do with difficulty. In my job for example we self-hosted HBase which is a beast compared to postgres, implemented custom backups etc, all because there was no good vendor for it. Postgres is much simpler and we always just used RDS and then switched to Aurora for the higher disk limits when it was launched. If there's a good enough vendor, you're just stroking your ego re-implementing these things when you could move on to the actual thing the business wants to release.
I've also seen senior engineering leads "proving" self hosting "saves money" but then 2 companies working on the same type of problem in the same industry with a similar feature set, on one side we had 5 people maintaining what on the other company it took 6 teams of 4-8 people. So it depends if you'd like to have a lot of your labor focused on cutting costs or increasing revenue. And they never include the cost of communicating with extra 5 teams and the increased complexity and slowness to release things this creates, while also being harder to keep databases with current versions, more flimsy backup processes, etc.
Ps: we got rid of hbase, do yourself a favor and stay away
> Your advice at the end is just common sense advice for anyone using any tool.
Common sense isn't so common. I've met a handful of devs across many separate companies who care at all how the DB works, what normalization is, and will read the docs.
> It doesn't mean you should spend time implementing your own custom backup process with ability to go back to a specific point in time, configurable in 1 minute.
If by implement you mean write your own software, no, of course not. Tooling already exists to handle this problem. Off the top of my head, EDB Barman [0] and Percona XtraBackup [1] can both do live backups with streaming so you can backup to a specific transaction if desired, or a given point in time.
Or, if you happen to have people comfortable running ZFS, just snapshot the entire volume and ship those off with `zfs send/recv`. As a bonus, you'll also get way more performance and storage out of a given volume size and hardware thanks to being able to safely disable `full_page_writes` / `doublewrite_buffer`, and native filesystem compression, respectively.
> If there's a good enough vendor, you're just stroking your ego re-implementing these things when you could move on to the actual thing the business wants to release.
Focusing purely on releasing product features, and ignoring infrastructure is how you get a product that falls apart. Ignoring the cost of infrastructure due to outsourcing everything is how you get a skyrocketing cloud bill, with an employee base that is fundamentally unable to fix problems since "it's someone else's problem."
> Ps: we got rid of hbase, do yourself a favor and stay away
HBase and Postgres are not the same thing at all. If you need the former you'll know it. If people convince management that they do need it when they don't, then yeah, that's gonna be a shitty time. The same is true of teams who are convinced they need Kafka when they really just need a queue.
My overall belief, which has been proven correct at every company I've worked at, is that understanding Linux fundamentals and system administration remains an incredibly valuable skill. Time and time again, people who lack those skills have broken things that were managed by a vendor, and then were hopelessly stuck on how to recover. But hey, the teams had higher velocity (to ship products with poor performance).
Have you ever been paid to do work before? There’s a price at which a business will prefer to pay to have SaaS/PaaS solve a problem. Allocating engineering hours to setting up and maintaining a Postgres cluster has a cost. You’ll want someone senior on it. Their time could be well over $100/hour. And that’s assuming your business is small enough to only need one DBA part time. A business that’s spending a ton on Aurora might need 3 specialists. Now you’re talking about hundreds of thousands of dollars per year. It could be better to just pay AWS.
However, at large scales cloud won’t make sense anymore. They do have a markup and eventually what you’re paying in markup could instead buy you a few full time employees.
Yes, many times, which is why I've developed this opinion.
> However, at large scales cloud won’t make sense anymore. They do have a markup and eventually what you’re paying in markup could instead buy you a few full time employees.
The issue is once you've finally realized this stuff matters, and have hired a DB team, I can practically guarantee that your schema is a horror show, your queries are hellish, and your product teams have neither the time nor inclination to unwind any of it. Your DB{A,RE}s are going to spend months in hell as they are suddenly made the scapegoats for every performance problem, and are powerless to fix anything, since their proposals require downtime, too much engineering effort, or both.
Hence my statement. Learn enough about this stuff so that when you do hire in specialists, the problems are more manageable.
You need to do things that are appropriate for a small company when you’re a small company. And then if you become a large company you change things to suit your new scale.
All of the troubles you described sound like bad management. I’m sorry if you’ve had to go through that. DBAs that are setting up a replacement are going to need time to do that right and expectations need to be set that this is a tricky problem.
