The solution to that in our case, at least, is that the stories aren't just handed down with no explanation from a product manager. Instead, there's a team-wide "estimation session" every week where everyone (dev, PM, and QA) attends and we take the higher-level feature ideas, including any initial supporting documents (market research, mockups, desired use cases) and break them out into smaller, estimated stories. At that point everyone has a chance to ask for more detail and clarify any concerns they have, point out potential difficulties, suggest alternatives, etc. So stories are only scheduled for work once they've passed that initial stage where everyone was involved; if things are too vague, or technically impossible, or otherwise unready for development, they don't make it out of the estimation session.

It definitely still requires good product managers, but again the idea is that no matter how good the spec or whatever is you'll still always have questions, and it's better to just build that fact into the process and get people used to asking questions rather than having people make assumptions. And the best way I've seen to make sure that the PM has the answer to those questions is the estimation session, since if the PM doesn't know what they want well enough to answer the questions quickly as you develop, you'll find that out in the estimation session before anyone is tasked with working on the feature. That at least largely avoids the situation where a developer starts working on something and finds out the PM really didn't have a great idea of what they wanted after all, or that it's technically infeasible, which (in my experience) is just a really bad situation to be in.

The story approach still presents endless ways to screw up, but it's far better than any other approach we've tried over the last 7 or so years.