It's hard to be sure you would have no leg to stand on. It would be an emotionally charged conversation, and misinterpretation would be likely. If the hiring manager, for instance, said anything about a legally protected characteristic (like age or marital status, for example), it could be easily construed as illegal discrimination. One could make the communication written to avoid certain types of misunderstanding, but then you'd give the lawyers an actual document to play with.
There is no upside. Remember that the hiring manager can be sued personally, not just the business.
Finally, consider another factor. Most of the time there are several qualified candidates and one may stand out. That does not mean the others were bad, just that one candidate clicked. It is of negative value to say "Nothing personal, but we really hit it off with Mr. Foobar and not you." Silence is actually better than pronouncing an under-informed judgement on someone based on a series of half-hour interviews.
There is no upside. Remember that the hiring manager can be sued personally, not just the business.
Finally, consider another factor. Most of the time there are several qualified candidates and one may stand out. That does not mean the others were bad, just that one candidate clicked. It is of negative value to say "Nothing personal, but we really hit it off with Mr. Foobar and not you." Silence is actually better than pronouncing an under-informed judgement on someone based on a series of half-hour interviews.