From specific to general, from general to specific

In various conversations, both on the Net and in real life, the following pattern emerges:

X: Group A does this (or is that, or is better/worse than group B in something).
Y: Not, they don't. I know quite a lot of people from group A who don't do this.
X: Well, obviously I didn't mean all of the group A, I was saying that statistically group A is significantly more likely to do this thing than group B.

X kind of does have a point here. One can say "men are taller than women" without having to qualify that one means that men are on average taller than women, not that every single man is taller than every single woman. It is more convenient to say "men are on average taller than women" to distinguish this from the case when a statement is true in case of almost every man, as in "men have penises and women don't", but IMO the statement "men are taller than women" is not in any way an abuse of English language, even if you don't add "on average".

Speaker Y and other similar ones (there is quite a lot of them) are often blamed for derailing the conversation, purposely misunderstanding what the other speaker meant to say, and failure to understand that the conversation does not concern specific cases, but the general case. These accusations can be fair or not, depending on the context.

What is interesting is that there is quite a lot of people with the opposite problem, ones who clearly have difficulties with going from the general to the specific, and while a lot of people point out to them that they are wrong in other ways, I've seen very few people explain to them in general that this is neither logical nor a sensible way to conduct a conversation.

A conversation like this:

X: Women prefer men who have a car.
Y: There is quite a lot of women who like men who don't have a car.
X: Yeah, yeah, I was not talking about every woman in the world, but on average women prefer men with cars.

is quite understandable, from both sides and on many levels. The participants are not sure how universally X's statement was meant or understood, and are seeking to understand each other. X might be annoyed because in X's opinion X's statement is obviously not meant to mean all women, just the majority. X often accuses Y of failure to understand that such general statements are not meant to apply to every member of a class. Obviously not meant to.

The problem is, this is not in fact obvious. And the reason it is not obvious is that by this point Y, and probably also X, has participated in many conversations that go the following way:

Y: I've met a few new guys lately, but nobody I am really interested in so far.
Z: That's because you would only date a guy with a car.
Y: No, I don't care about that, they were just not my type.
Z: That's just what you are saying, but I know you want a guy with a car. Women always do.

Z is being entirely unreasonable here: he/she is, in fact, taking the implied statement "women prefer men who have a car" to mean every single woman, and is clearly not ready to accept an example to the contrary. However, I have never seen anyone accuse Z of being unable to understand that general statements so not apply to every specific case. This accusation is usually reserved for Y in the previous conversation.

So: why do the people who try to deny a general statement as if it were a universal statement by citing specific counterexamples get accused of not understanding the nature of general statements, and why do people who try to extend a general statement into a universal statement by denying existence of counterexamples not usually get accused of the same?