> There were no real transactions, no ensured consistency
Which is the right way to do it, because transactions don't extend into the real world. If you need to wait for the consequences of a given event, wait for the consequences of that event. Otherwise, all you really care about is all events happening in a consistent order. It's a much more practical consistency model.
> and no referential integrity
The problem with enforcing referential integrity is how you handle violations of it. Usually you don't really want to outright reject something because it refers to something else that doesn't exist yet, so you end up solving the same problem either way.
> There was also no authentication or authorization, because a default-configured deployment of Kafka from Confluent happily neglects such trivial details.
Pretty common in the database world - both MySQL and PostgreSQL use plaintext protocols by default. Properly-configured kafka uses TLS and/or SASL and has a good ACL system and is as secure as anything else.
> It was a nightmare to code against once you left the fantasy world of functional programming nirvana and encountered real requirements. It meant pushing a whole series of concerns that isolation addresses into application code... or not addressing them at all.
My experience is just the opposite - ACID isolation sounds great until you actually use it in the real world, and then you find it doesn't address your problems and doesn't give you enough control to fix it yourself. It's like when you use one of those magical do-everything frameworks - it works great until you need to customise something slightly, then it's a nightmare. Kafka pushes more of the work onto you upfront - you have to understand your dataflow and design it explicitly - but that pays off immensely.
> It was a GDPR nightmare.
Really? I've found the exact opposite - teams that used an RDBMS had to throw away their customer data under GDPR, because even though they had an entry in their database saying that the customer had agreed, they couldn't tell you what the customer had agreed to or when. Whereas teams using Kafka in the way you describe had an event record for the original agreement, and could tell you where any given piece of data came from.
> Really? I've found the exact opposite - teams that used an RDBMS had to throw away their customer data under GDPR, because even though they had an entry in their database saying that the customer had agreed, they couldn't tell you what the customer had agreed to or when. Whereas teams using Kafka in the way you describe had an event record for the original agreement, and could tell you where any given piece of data came from.
This is absolutely wonderful! Unfortunately, this team decided to store data subject to GDPR deletion requests in Kafka, where deletion is quite difficult. It was a problem, when trying to do deletion programmatically, across many teams using the same set of topics.
The real nightmare came when this team, obsessed with the power of infinite retention periods, encountered PCI-DSS. You see, the business wanted to move away from Stripe and similar to dealing with a processor directly, in order to save on transaction fees. So obviously they could just put credit card data into Kafka...
Yeah fair enough. I'd argue that this is kind of a double standard (a traditional RDBMS may well be copies of "deleted" data on dirty pages, and may well leave that data on the physical disk indefinitely, for much the same reasons as Kafka does - it just makes it a bit fiddlier for you to access it), but your legal team may decide that it's required.
I don't think your overall scorn is warranted - there are bigger problems that are endemic to RDBMS deployments, and the advantages of a stream-first architecture are very real - but there are genuine difficulties around handling data obliteration and it's something you need to design for carefully if you're using an immutable-first architecture and have that requirement.
Which is the right way to do it, because transactions don't extend into the real world. If you need to wait for the consequences of a given event, wait for the consequences of that event. Otherwise, all you really care about is all events happening in a consistent order. It's a much more practical consistency model.
> and no referential integrity
The problem with enforcing referential integrity is how you handle violations of it. Usually you don't really want to outright reject something because it refers to something else that doesn't exist yet, so you end up solving the same problem either way.
> There was also no authentication or authorization, because a default-configured deployment of Kafka from Confluent happily neglects such trivial details.
Pretty common in the database world - both MySQL and PostgreSQL use plaintext protocols by default. Properly-configured kafka uses TLS and/or SASL and has a good ACL system and is as secure as anything else.
> It was a nightmare to code against once you left the fantasy world of functional programming nirvana and encountered real requirements. It meant pushing a whole series of concerns that isolation addresses into application code... or not addressing them at all.
My experience is just the opposite - ACID isolation sounds great until you actually use it in the real world, and then you find it doesn't address your problems and doesn't give you enough control to fix it yourself. It's like when you use one of those magical do-everything frameworks - it works great until you need to customise something slightly, then it's a nightmare. Kafka pushes more of the work onto you upfront - you have to understand your dataflow and design it explicitly - but that pays off immensely.
> It was a GDPR nightmare.
Really? I've found the exact opposite - teams that used an RDBMS had to throw away their customer data under GDPR, because even though they had an entry in their database saying that the customer had agreed, they couldn't tell you what the customer had agreed to or when. Whereas teams using Kafka in the way you describe had an event record for the original agreement, and could tell you where any given piece of data came from.