This issue is much worse if you don't have swap. What happens, I think, is that as memory allocated by processes grows to fill the available RAM, it starts to push out memory that doesn't technically need to be in RAM, like cached file pages. Which accounts for some of the slowdown, until it reaches the code itself, which is 'just' a memory mapped file. So eventually most of the code that is actively trying to run is being pushed out of RAM and must be loaded in as it executes, slowing everything to a crawl and generally creating a death spiral. If you have swap the kernel can decide to put other pages onto disk and keep the more important stuff in memory. Or you can run something like early-oom which stops things from getting to that point in the first place (albeit in a somewhat brute-force manner).
Dealing with low-memory situations elegantly is pretty hard: firstly Linux uses memory overcommit by default, in part because the semantics of fork imply very large memory commitments which are almost never realised, and in part because a lot of software does the same because it's the default. Secondly, managing allocation failures is often tricky and ill-tests, and often requires co-ordination between different systems. The DE could, though, in principle, put running applications in a container which would prevent them from using above a certain amount of memory, but the results are similar to early-oom in that the result of reaching the limit is almost certainly the termination of the process using the most memory.
Yes, but the problem, I feel, is the priority of what gets pushed out to RAM on Linux.
You could split the processes into 2 categories:
1: applications that are doing tasks the user wants.
2: OS processes that the user needs to interact with in order to terminate applications.
There is an argument for applications taking priority: the user wants to do a task, if you move application out of RAM, the task is going to take longer.
But to me OS processes, including the graphical shell (taskbar, windowing system, etc.), should have priority: if an application hangs on I/O, the user NEEDS to be able to use the taskbar in order to terminate the application, otherwise they're going to have to wait who knows how long for the application to finish its task (or just hard reset the computer).
I don't know anything about how Linux handles memory, but the impression I have is that it has its priorities wrong, or it may not even have a way to configure priorities (unlikely), or maybe there is a to prioritize what is kept in memory but it only splits kernel/userspace memory so DE's that sit in userspace don't get priority (i.e. it's inadequate for a graphical operating system).
To be frank, as a desktop Linux user my biggest fear is that the Linux kernel is perfectly capable of prioritizing kernel/userspace memory, but it has no way to prioritize DE's. In other words, that the "graphical OS" use case of Linux is a second-class citizen, a feature bolted on top of GNU/Linux/Systemd. Because that would mean a lot of things are considered only from the perspective of a Linux server. This is only my imagination talking, since I'm not really involved with how Linux works. But to be fair I was never involved with how Windows worked either, and I never doubted it considered desktop a primary use case.
The issue is, if you don't have swap, then it's not matter of prioritisation: there are some things in RAM that can't be pushed out, regardless of how unimportant they are (so the only recourse is to terminate some processes, usually the ones using the most RAM. Linux the kernel by default tries really had not to do this, which is why there are userspace applications like earlyoom to do it, which is probably the better location for such logic).
And yeah, you can adjust the priorities and the latency/throughput tradeoff (even per-application to some extent), but it's a difficult thing to get right in general (what works for one use-case might make another a lot worse). I don't know of any DE that really tries to adjust this, though (not because the kernel can't, but probably because no-one on the DE side has really prioritised it or they have tried and it hasn't made a noticable difference).
A specific process can use mlockall to keep all its mapped memory resident and prevent swapping or page cache eviction. That's what earlyoom does so that it can stay responsive when memory gets low. It's unfortunately underutilized in other infrastructure. It's also all-or-nothing: everything stays resident until it's munlocked regardless of how frequently it's used.
I had hoped that something like Linux Pressure Stall Information (PSI) would become more useful for low-memory scenarios. E.g. you could put critical processes in a cgroup that could rate-limit swap-outs/evictions so that it was always responsive. There are some cgroup knobs that affect reclamation, but you need a really good guess about how much memory something needs, which makes it hard to use.
Dealing with low-memory situations elegantly is pretty hard: firstly Linux uses memory overcommit by default, in part because the semantics of fork imply very large memory commitments which are almost never realised, and in part because a lot of software does the same because it's the default. Secondly, managing allocation failures is often tricky and ill-tests, and often requires co-ordination between different systems. The DE could, though, in principle, put running applications in a container which would prevent them from using above a certain amount of memory, but the results are similar to early-oom in that the result of reaching the limit is almost certainly the termination of the process using the most memory.