There seems to be a perception among many men that sexual harassment mostly exists on some sort of continuum with harmless flirting. And, to be fair, some of the complaints from women do seem to verge on stigmatizing basically normal behavior. It may even be true that some small number of these women are essentially using the scarlet letter of “harassment” to penalize guys who they resent having to turn down.

But when I talk about sexual harassment, I’m not talking about this. I’m not talking about guys at work who put me in an awkward position by asking me out. Nor even about the fellow I briefly worked with who used to show me pictures of skimpily clad magazine models and say “You’d really look good in this.” He wasn’t an abusive jerk, but a shy, awkward fellow who needed a lot (!!!) of work on his approach. When I belatedly made it clear that I wasn’t going to date him, he immediately stopped with the pictures.

No, when I say "harassment," I’m talking about … well, this is a family column, so actually, I can’t repeat most of what I’m talking about. But let’s just say that when you are on your knees under someone’s desk in order to check the network connection, and the owner of that desk starts a sentence with "while you're down there," he has not inadvertently stumbled over some near-invisible social line he wasn’t aware of.

The sort of men who make these remarks don't do this kind of thing because they think it is all right; they do it because they can get away with it. That is the kind of abuse that Carlson and others are alleging.

I do not know whether Ailes is guilty or innocent; that’s what courts are for. But I was surprised to find that Rivera actually thought “I’ve never seen any sign of it myself” was relevant to the question of Ailes's guilt or innocence. Does Rivera consider himself so irresistible that anyone with the potential to sexually harass would be sure to sexually harass him? Like he's some kind of canary in the sexual harassment mine? "Oh, don't worry about Roger; if he were a lech, I'd be the first to know."

He showed a stunning lack of imagination, or feigned it, to defend his friend. Stunning -- but not rare. Everyone is shaped by his or her own experiences, and men are less likely to have been sexually harassed. So are women a decade younger than me, whereas women a few decades older became inured to quite crass abuses. Think "Mad Men"-level sexual harassment

Related: Employers expect increases in LGBT discrimination cases

Everyone naturally extrapolates from their own experience to assess the likelihood of some claim. I was shocked when a black friend told me that clerks followed her around stores. What she said was completely alien to my own experience. But after she told me, I did observe it happening occasionally. Previously, presumably, I had not noticed, because it wasn’t happening to me.

That matters because it affects how we assess public policy. If you rarely or never see sexual harassment, then it can be hard to believe a group that says that it’s really common and that legal redress is required.

We forget that if one group is particularly likely to be targeted, then even a small number of people who abuse their power can create, for the targeted, a near-universal experience of being harassed. And so we end up shouting past each other, the targeted on one side shouting “this is rampant, do something!” and the unaffected on the other angrily denying that they’ve done anything wrong. Conflicts that should be framed as Americans against a small number of abusers instead get framed as men against women.

Sexual harassment is not, of course, the only context in which this occurs. We don’t need to believe that all cops, or even most cops, abuse their power, to understand that as soon as power is created, it will be abused by at least some of the people who wield it.

Related: 82 victims, one company

And if those people perceive that it is wiser to target black men than middle-aged white women, the middle-aged white women will have no idea that this is going on, while the black men will grow to see every cop as a potential threat. So instead of coming together to solve a joint problem, we divide into sides, separated by our own experience.

I do think that this attitude can be overcome. Both victims and the groups from which most of the victimizers are drawn need a refresher on probability: “Most sexual harassers are men” is not the same statement as “most men are sexual harassers.” And the righteous majority of men, or police officers, probably has more in common with victims of sexual harassment, or victims of police brutality, than with the perpetrators.

Copyright 2018 Bloomberg. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.