And Republicans are more likely than Democrats to think workers misuse the paid time off. Two-thirds of Republicans think abuse is common, compared with 46 percent of Democrats, the survey found. "They do seem to think more people abuse the benefit," said Juliana Horowitz, one of the authors of the survey.

That might explain why Republicans, in general, are less supportive of paid family leave, she added. Around three-quarters of Republicans support paid maternity leave, compared to 90 percent of Democrats. There's less support among workers of both political persuasions for paid paternity leave: Around 57 percent of Republicans support paternity leave, versus 79 percent of surveyed Democrats.

And Americans' leeriness of other people's paid leave-taking is reflected in their reluctance to take very much of it themselves: More than half the workers Pew surveyed said they had taken less time off than they really wanted or needed to after they had a child.

The negative attitude can keep workers from taking time off, said Ellen Galinsky, president of the Families and Work Institute, a nonprofit that researches family and work. "People don’t feel safe in using it largely because of attitudes like that," she said. "Although we recognize how important leave is, unless there is a culture that supports it, people are going to worry."

When Pew asked leave-takers of all kinds why they returned to work early, just under half the respondents said they feared risk losing their job. Around a third thought taking more time off might hurt their chances for advancement. Others cited feelings of guilt: 41 percent said they felt badly about co-workers taking on additional work.

Those fears aren't unsubstantiated. New moms often return to jobs with fewer responsibilities and face bias in the workplace, resulting in lower pay — a phenomenon known as the "motherhood penalty." For every child a woman has, she earns 4 percent less over her lifetime, research from University of Massachusetts sociologist Michelle Budig has found.

Men, on the other hand, get a "fatherhood bonus" — a 10 percent bump in earnings when they have kids. Those effects were reflected in the Pew survey: Only 13 percent of men said taking parental leave had a negative impact on their careers, compared to a quarter of women.

Without flexibility, parents find it hard to conform to rigid schedules — pushing new moms in particular out of the workplace. About 12 percent of women who go on maternity leave don't come back, the Pew research found. The survey didn't ask why, or if these women had discussed the plan with their employers.

"When people don't come back, it’s often that it’s not a good situation," said Galinsky. "If you don't feel valued and supported, you're less likely to want to come back." Companies that offer longer leave and more flexible schedules and that encourage new parents to take leave tend to retain more women workers, she said. "People tend to come back to places where they are respected."

Copyright 2018 Bloomberg. All rights reserved. This material may not be published, broadcast, rewritten, or redistributed.