You are missing the point. The idea is not to determine exactly what's going on, because that doesn't matter. You mentioned the casual user before, but all these details are only important to specialists trying to debug an issue with a specific application.
When you run out of RAM, it is completely irrelevant which application is thrashing, because the problem is that you run out of RAM. It doesn't matter if the reason is that an app is accessing a memory mapped file or modifying a copy-on-write page, when underlying reason is that you ran out of RAM.
Activity Monitor is perfectly suitable to find out why you ran out of RAM. Oh, Mathematica is using 2GB of RAM? Maybe I try closing that. It doesn't matter if "Real Mem" actually counts some memory twice, it is still a useful measure.
EDIT: Yes, I assumed that the problem is memory pressure. Since this is a common reason for "The whole system becomes sluggish", it seemed reasonable to start testing for this.
Fantastic argument, user is having a problem, but it doesn't matter what's going on, let the user try arbitrary things which I already explained that are folklore and why they make no sense, maybe it works.
If that's you advocate, let's stop here with this discussion, as there's no common ground.
When you're having a problem, first you try to understand it in order to try to solve the root cause. Applying rules of thumb like "kill top RSS process" are as sensible as rules of thumb regarding running repair permissions, sizing paging files or hoping arbitrary herbs cure arbitrary diseases.
Activity Monitor is useless because it's impossible to assess how a specific action will affect the system. Users should understand what's going on when they kill a process and the tools should help them to do so. When people do something, they should understand it, even casual users. Activity Monitor exposes data that's not understood by most users, although it leaves the impression that it does.
Just for trivia, memory pressure, hasn't been the primary reason for "the whole system become sluggish" for a few years already.
1) Nowhere have I suggested to "kill top RSS process".
2) If you start Activity Monitor, you immediately see if memory pressure could be the problem (well, after 30 seconds, because it takes time to start Activity monitor if you ran out of memory)
3) If you ran out of memory, looking at which apps use much RAM is useful. And it's not as unpredictable as you make it seem. Quitting an app, you'll free at least the private memory, and you might free some of the shared memory, and other processes (e.g. Window server) might also free some more memory as a consequence. This might not be a precise prediction, but it's not quite "impossible to assess".
I do not know how prevalent memory pressure is for other users. It's been the primary reason of "sluggish computer" for me. If you know more about this topic, I'd be thrilled to hear other possible explanations beyond "it's more complicated than that".
Your argument basically calls for casual users to stop being casual and to begin to become experts. Activity monitor certainly does help casual users understand what's going on. For the casual user using a system normally (meaning having a browser open, checking mail, writing in a word processor) seeing that application X is using the most memory means that's the problem to them and they're correct to assume they should kill it. It may be hit or miss but that's all they know and in most cases things get fixed that way.
As professionals we often forget what it is to be a casual user. Asking a casual user to learn what you explained as folklore is simply too much to ask. Everyone who uses a computer should at least be technically literate to a degree but that means understanding the basics. To a casual user the basics are: my computer has a processor that executes tasks, it has RAM that stores data for quick retrieval, and a hard disk for long term storage. Each application uses a percentage of my finite RAM and when it runs out my system slows down. Therefore logic dictates that if I kill the app taking the up the most RAM my computer will go faster.
That's all they usually know. We understand that Activity Monitor lies to us and killing random processes is voodoo but we also have to take into account how we use our machines. The casual user will be able to solve their problem by killing processes more often than people like us will because of the way they use their systems plus there is a placebo effect for them. When they kill a process they often feel like the system just got faster regardless of whether it really did.
I liken it to driving a car. Ask some random person about fuel economy. Their thinking is "high octane fuel has more energy per gallon therefore if I use it I'll get better fuel economy". They might even know the relationship between tire inflation and fuel economy too if you're lucky. Ask a professional driver about those things and they'll look down at the average person like they're crazy. They know all how octane, oil, air filtration, shocks, struts, aerodynamics, etc, etc. all contribute to better fuel economy. "If only the average driver knew what I knew, then they'd save a ton on gas" theyd think. But alas, that's too much to expect so we just have to make sure they get the basics and it's up to the professionals to provide the average person with something that just works and do our best to be one step ahead of users by anticipating their usage patterns. This applies to hardware/software engineering, car manufacturing, and anything else. You just can't expect the user to learn or even take an interest in even a quarter of what we know.
When you run out of RAM, it is completely irrelevant which application is thrashing, because the problem is that you run out of RAM. It doesn't matter if the reason is that an app is accessing a memory mapped file or modifying a copy-on-write page, when underlying reason is that you ran out of RAM.
Activity Monitor is perfectly suitable to find out why you ran out of RAM. Oh, Mathematica is using 2GB of RAM? Maybe I try closing that. It doesn't matter if "Real Mem" actually counts some memory twice, it is still a useful measure.
EDIT: Yes, I assumed that the problem is memory pressure. Since this is a common reason for "The whole system becomes sluggish", it seemed reasonable to start testing for this.