The fundamental problem here is that the kernel can't time-travel into the future far enough to make sure no higher-priority processes will try to allocate more memory, as the system has to make the decision to return ENOMEM at the time of the allocation. There is a reasonable(-ish?) default to deny allocations that request more memory than currently available (using the Linux definition of available, that is total memory without the sum of resident sets of all processes and a bit more). Of course, this limit only works properly if other processes do not add anything to their working sets, but again, the kernel's fortune-telling abilities are quite limited.
It can't look into the future, but it certainly could use the information from the past.
Has this application already allocated 90% of the memory? Has it been steadily growing the allocation without releasing much back? Well then, why let the system run out completely? Why not stop it at say 10% or 5% left?
For a desktop OS that would be very unusual in my experience.
For a server, perhaps, but I'd say even for a server it would be better that say the core application returns a 500 internal error or whatever than forcing the system to start killing random processes.
