Parquet files are smaller than row based storage in a database (but not those databases with focus on strong compression).
And for backup - the files are probably easier to just copy to multiple disks for redundancy, as opposed to database dumps and incremental backups which at the Petabyte scale will be a pain.
Correct, but RustFS is the only drop in replacement (just migrated) Garage and Seaweed are nice (didn't look into Ceph) but you have to re-ingest. RustFS was just plug and play albeit a few minor API differences.
Also Apache licensing gives some peace of mind after the musical chair license game before they finally landed on only paid AIStor offering.
Not that I know of but a lot of people took tat as a sign of extensive vibe coding and wrote Rustfs off completely. I personally installed seaweedfs, rustfs and garage at different points and plan on checking out rustfs again in the future. If anybody is interested, the most robust of them seems to be seaweed by the way.
there is a reason why people develop for S3: a lot of enterprise data is there. people ingest there from various sources. and it's not just parquet usually, it's multivendor sources writing to an iceberg catalog.
nobody will run minio on AWS other than hobby projects and small demos.
I regularly work with iceberg datasets in the double digit TB range per dataset. keep that in mind when you think about sizes. databricks, snowflake, large enterprise vendors: they are targeting these sizes.
they exist, sure. And I'm sure it can handle PB+. on prem is an existing market, however, if you reread my comment I talked about running minio on AWS because S3 is too expensive - just doesn't make sense to do.
I've yet to met a Fortune 100 who isn't mostly using either on prem or a large hyperscaler (S3/Azure/GCS).
The large enterprise vendors are not prise-sensitive. They're on AWS because you never get fired for picking AWS, and there isn't really any other choice for these vendors regardless of AWS ripping you off.
At this point S3 is a standard interface. All sorts of cloud providers and open-source projects provide S3. If you're on AWS, price isn't the reason. You pick AWS because you don't see your company taking a risk with anything else.
S3 doesn't mean expensive. AWS does. But AWS users are fully locked-in, they'll pay whatever the price is.
There is a scale between prise-sensitivity and risk-averseness, from my point of reference large companies are much more risk-averse than they are price sensitive. Of course this will vary, CTOs exist in all sort of different environments.
Price is not the reason people chose AWS. Some companies use Azure. The current startup at $WORK uses yet another smaller Cloud. And yet AWS sill has the clear lead in market share. That's because price is far from the only factor, and not even the main factor.
> There is a scale between prise-sensitivity and risk-averseness, from my point of reference large companies are much more risk-averse than they are price sensitive.
That's not true. It's just the way things work "saving money" isn't part of the KPI. Enterprise teams get a budget. If you "saved" you don't get it back. So unless there's a legit need it's ALWAYS easier increasing than cutting it.
It's not about risk. It's about power. They are price sensitive but in a way that doesn't matter to the bottom line i.e. if I can cut my AWS storage bill by 10% and then spend it on random tokens I'd do it.
They definitely havent. Tech side of companies is a Cost Center. And the main question the CEO/CFO makes to the CTO every week is "how can we reduce our AWS bill?" , even before the how was your weekend ? One.
Parquet files are smaller than row based storage in a database (but not those databases with focus on strong compression).
And for backup - the files are probably easier to just copy to multiple disks for redundancy, as opposed to database dumps and incremental backups which at the Petabyte scale will be a pain.