This depends entirely on who the users are. If a fellow developer tells you something, listen to him! If someone who doesn't know what a CPU is tells you something, try to figure out what he really wants. But even then, if he tells you something simply doesn't work well for him, don't dismiss it as him not understanding your vision.
The problem with some projects (e.g. GNOME) is that they consider all feedback to be equally invalid, and the self-appointed "designers" are always right. This is the result of the Apple-inspired worship of almighty "design," and is contrary to the very idea of personal computing empowering people.
"Listen to fellow developers" can be very dangerous too. It's how you end up with programs that have menus nested 4 levels deep and thousands of options that will only be used by the developer who recommended/implemented it and a few other people (at huge cost to the other 99% of users).
Let me clarify my statement: you should of course ALSO listen to them (but you should take what is said with grain of salt and try to understand the real motives; once you understand that, you can propose other - often better - solutions).
However the real feedback comes from what is not said out loud. Users might dismiss the problems they have as their incompetence, but only you can be the judge of that. If 3 people in a row make the same mistakes, I can guarantee you it is not their incompetence you are seeing... ;) That is why observing the users is so important, much more than just listening to them.
