Why wouldn't you sample a random set of potential post office users?
If you wanted to know what parents think of public schools, would you survey only the parents who currently send their kids there, or would it also be informative to know the opinions of the parents whose children educated elsewhere?
It would depend on what type of data you're collecting. If you're looking to understand the quality of a service, non-users are worthless, you need to talk to people who've spent actual time with it. If you're looking to get impressions of a service, then sure, ask anyone and everyone.
Just because someone doesn't currently use X, it doesn't mean they've never used X.
And even if someone has never used X, it doesn't mean they haven't done extensive research about X, before concluding X doesn't meet their quality bar.
Before deciding whether to send my son to an SFUSD school, I had a call with the principal of the school and asked questions specific to my son's situation. Do you think my conclusions are worthless for understanding the quality of the service?
Not useless by any means, but if given the choice between somebody who interviewed the principal and a somebody else who's son actually attends, I'd prefer the latter every time. Wouldn't you?
Also, while I get your point, you're in the very unique position of informed non-users. That's a valuable category if you can access it, but it's the most difficult. (How can you determine which library non-attendants have done their research?)
OK, so we agree that your previous statement was overstated, right?
"If you're looking to understand the quality of a service, non-users are worthless"
given the choice between somebody who interviewed the principal and a somebody else who's son actually attends, I'd prefer the latter every time. Wouldn't you?
Not every time, no. You seem to be suggesting that no single informed non-user could be more useful than the least useful user.
"If you're looking to understand the quality of a service, non-users are essentially worthless."
I'm suggesting that, in absence of some out-of-band signaling mechanism, the expectation value for usefulness of individual feedback is orders of magnitude greater for users than non-users.
Put differently, if I happened to be interested in this SFUSD school as well, is there some list of people like you I should interview?
Put differently, if I happened to be interested in this SFUSD school as well, is there some list of people like you I should interview?
No, but I can point you to a FB group where you can ask for anyone that faced the same decision before. Some of them will have gone one way, and some the other. You can learn from both.
I'm suggesting that, in absence of some out-of-band signaling mechanism, the expectation value for usefulness of individual feedback is orders of magnitude greater for users than non-users.
This might be true for things that most people don't need (like ski boots) but it's not true for things that most people use, like the post office. That was the point I was making here: https://news.ycombinator.com/item?id=42861928
If you got to the post office to find out about people's experience, your sample will be biased: you will mostly be interviewing people who use the post office much more often than the median post office user.
I go to the post office 1-2 times per year. People like me will be both:
- underrepresented in your sample, and
- be the least likely to have adapted to any difficulties getting to know how the post office works
Some folks who are heavy users of parcel services will have stopped using the post office completely. They will be totally unrepresented in your sample, even though you'd like to know why they use UPS instead of USPS.
I do see your point and agree it can be valuable data. What I'm arguing is specifically that the economics of it are extremely unfavorable.
Having a relevant Facebook group is about the best case scenario. I guess email might work? Maybe letters? I can't imagine calling or canvasing.
On top of all that I still think the data from actual users is still going to be more valuable overall, even the negative stuff. I'd prefer somebody who attended the school and had a bad experience than somebody who had a bad impression and went elsewhere.
But... I'll agree it's not worthless. It just strongly depends on the type of data you want to collect. For impressions and onboarding, it's a different story. Here's the goal of this study:
> NYPL regularly surveys its patrons to understand and improve how the Library fits into—and adds value to—their lives. We strive to identify the unique power of public libraries, pinpointing precise mechanisms of positive impact, so that we can preserve and strengthen that impact.
If you were to survey people in a place where the library mainly serves as shelter for people without homes, you would conclude that the most important thing is that, and there's no point maintaining collections of books.
If you believe that public libraries serve some purpose, you have to start with that, rather than just using gradient descent to get to somewhere. Otherwise why wouldn't libraries evolve into hospitals, or podcast studios, or brothels?
If you wanted to know what parents think of public schools, would you survey only the parents who currently send their kids there, or would it also be informative to know the opinions of the parents whose children educated elsewhere?