I have not worked with FAANG or any project that requires massive web scale. Most of my career I have worked with resource constraint devices. So when I decided to write a web app, it was a horror show for me. Even though I have a CS background and understand any tech when I read and see it, it becomes extremely difficult to navigate the maze of different options available and the many things one is forced to consider to get a web app running.
What I have ultimately realised is that having a middle path to software architecture development is the optimal way i.e. build critical sections on your own and outsource others. For example, at the start do not think about horizontal scalability, it is a nightmare scenario to handle even if outsourced. So concentrate of vertical scalability with the help of a hosting provider. Horizontal scalability is a luxury that you can afford when you have dedicated people to manage it. There is a reason people developing Postgres have resisted the temptation to support horizontal scalability. It is a very hard problem to solve.
So if you are starting small and at an early stage then concentrate on how you will be able to own the data, how you will be able to own the business logic, and how you will be able to migrate from one service provider to another avoiding any vendor lock-in.
High availability, fault tolerance, always-on architectures are very difficult to achieve and maintain. There is a reason why FAANG spend so much resources on this and still get it wrong.
In case you want to go that route, then for some practical advise refer to https://sre.google
After evaluating lots of HA and fault-tolerant solutions and thinking about it for some time, personally I have come to the conclusion that having a single node with postgres, backed by streaming replication for backup, with dynamic vertical scaling using a hosting service where I can easily migrate my setup to other hosting provider is the way to go.
If your journey has made you to come to some other conclusion then please do share.
>High availability, fault tolerance, always-on architectures are very difficult to achieve and maintain.
Ironically, the only reason this is true is because generally people working on those pretty much cobble shit together instead of writing correct code, so you have subsystems of code talking to subsystem of code, and inevitably shit goes wrong.
Build your system correctly to the use case, from the ground up, and you will have a much easier time.
Of course the counter argument to this is the age old "what if developers make mistakes, its better to use off the shelf software thats already tested and verified". But the real answer to that isn't to use off the shelf software, its to not be or hire shit developers.
What I have ultimately realised is that having a middle path to software architecture development is the optimal way i.e. build critical sections on your own and outsource others. For example, at the start do not think about horizontal scalability, it is a nightmare scenario to handle even if outsourced. So concentrate of vertical scalability with the help of a hosting provider. Horizontal scalability is a luxury that you can afford when you have dedicated people to manage it. There is a reason people developing Postgres have resisted the temptation to support horizontal scalability. It is a very hard problem to solve.
So if you are starting small and at an early stage then concentrate on how you will be able to own the data, how you will be able to own the business logic, and how you will be able to migrate from one service provider to another avoiding any vendor lock-in.
High availability, fault tolerance, always-on architectures are very difficult to achieve and maintain. There is a reason why FAANG spend so much resources on this and still get it wrong.
In case you want to go that route, then for some practical advise refer to https://sre.google
After evaluating lots of HA and fault-tolerant solutions and thinking about it for some time, personally I have come to the conclusion that having a single node with postgres, backed by streaming replication for backup, with dynamic vertical scaling using a hosting service where I can easily migrate my setup to other hosting provider is the way to go.
If your journey has made you to come to some other conclusion then please do share.