I'm not an expert, but the application's memory usage being opaque to the kernel sounds vaguely like the problems Azul had scaling its Java runtime to large (>2GB) heaps.

Azul recently open sourced some of their kernel patches:

http://www.managedruntime.org/faq

But that is about all I know.