20% LOL! If only it were that low. I used to be a reputation management company, at one time we were responsible for 10% of the reviews in Silicon Valley. There were metro's where we were about 80% of the reviews.
When you think that the majority of companies have less than 10 reviews, and most owners can't resist doing a review, and having a friend do a review you are at 20% really fast.
When you combine that with organizations like Blackwaterops.com (my old company) that used to do 1000s of reviews a day. 20% is really low, and wrong.
Just out of curiosity, do those "reputation management" companies realize that in serving their clients by making fake reviews, they help destroy end-consumer trust?
In many ways, it's the old SEO argument all over again - gaming a system to increase ratings eventually erodes trust in that system, making those increased ratings less valuable. At the same time, refusing to game the system means your business loses potential eyeballs/customers/etc, so one is left with the old rock vs hard place situation.
I don't do reputation management anymore. Now I build tools to do real reputation analysis, and fight the very things I used to do. I'm like Darth Vader in the last few moments of his life... Only I plan to live a long while.
At Stremor.com we have technology that "breaks" SEO so we have better search results at http://www.samuru.com than you would have using link authority. We have tech that can detect writing styles to tell that 6 reviews all came from the same person. We can do this for things like YELP or Amazon product reviews. We also do this to detect web content that was a Paid Review.
> Now I build tools to do real reputation analysis, and fight the very things I used to do. I'm like Darth Vader in the last few moments of his life...
Really this just reads like you figured out a way to get paid to play both sides against each other. Hey, I'm not judging, in fact I say more power to you for figuring out how to make a buck, but casting yourself as some sort of redeemed hero seems a bit over the top.
so can I pass Samuru a link to a particular Amazon product and it'll tell me how many of the reviews have a similar writing style therefore possibly being fake?
That would be an interesting service to provide. More data points for companies (x people were interested enough to check reviews with us) and end-consumers get a better feeling for the reviews. I'm not sure how you'd turn a profit with it, unless the data points for companies prove valuable enough. I also don't know how you'd keep from turning such a service into yet another system to be gamed.
We don't expose that as an API in that way. We are rolling out features, but I am not sure flagging reviews as fake will be something we make "public" instead we'd license that to a company like amazon. Otherwise on the small percentage we get wrong we might have some sort of liability.
> Just out of curiosity, do those "reputation management" companies realize that in serving their clients by making fake reviews, they help destroy end-consumer trust?
The thing with Tragedy of the Commons is that recognizing the fact that it exists doesn't change the fact that defection is still the optimum reward-maximizing strategy.
> In many ways, it's the old SEO argument all over again - gaming a system to increase ratings eventually erodes trust in that system, making those increased ratings less valuable.
And not gaming the system means that you get less short-term returns while everyone else games the system, gets better short-term returns, and still erodes public trust in the system.
> do those "reputation management" companies realize that in serving their clients by making fake reviews, they help destroy end-consumer trust?
Don't really see the relevance this is advertising and most industries in general.
Look at the clothing industry, we all could be wearing sacks right now, but people realised if I wear a nicer sack than everyone else it'll up my born attractiveness and so the race begins, back to where we started worrying about clothing as well as dealing with our born attractiveness.
Do the people participating in these industries know they are not improving the world and destroying things like customer trust, probably not. But that's most people.
I know that most people just look at the stars and make a decision.
Other people read the good reviews and determine to make their decision from that.
I read the bad reviews. 1 to 3 star reviews. I'd rather go to a place with a bifrication of bad and good reviews where the bad reviews are stuff I don't care about...
My theory is that if a place sucks it will definitely have bad reviews. A place that is great will still have a few, but they will be over stupid stuff (i.e. I expected something different etc.)
When my friend's mother opened her restaurant in Koreatown, they received, amongst other welcome-to-the-neighborhoods: phantom Yelp reviews (describing tickets and events that never occurred) and a string of big table reservations (no shows). One such "review" complaining about a rat infestation was written by a then Yelp Elite member rather than an obvious sockpuppet.
What happens when a trusted reviewer gets recognised as such, and starts getting offers to do faked reviews for cash? This is a classic example in other mediums, eg. radio. Trust is a double-edged sword.
I read the bad reviews and the reasons for the negatives.
If they add up, I'll take note. If they don't, I'll presume a fake.
The failure of review services (Amazon, Yelp, and others) to police their reviews adequately really hurts these services. Otherwise they're little more than a yellow pages or directory.
To me what this means is you can't trust a positive signal, but you still can trust a negative one. If it's really that bad, then they're clearly not on the ball.
There was as much if not more negative gaming as positive. The best way to hurt a competing self storage services is to post a review that mentions their rat problem.
In addition to gaming competitors, there are unreasonable customers who can't be trusted. So, technically, the review is from an actual customer, but still cannot be relied upon.
And, the problem is compounded by the fact that more people are apt to write a review if they have a negative experience. This can really skew the results in a negative direction.
The whole paradigm of reviews as they are currently implemented is just plain broken.
When using Trip Advisor I skip over reviews where the reviewer has only 1 review. I assume they are written by competitors. A hotel could have many pages of favorable reviews, yet the reviewer with only 1 review under his/her belt will likely report something disgusting like bed bugs or blood stains.
Conversely, a hotel probably has more than one competitor so if they're posting negative reviews they may well review all of them. Someone who has a particularly bad (or good) experience with a hotel may well not have signed up before but could feel compelled to do so for that experience.
Note that I'm not speaking from experience here, just playing devil's advocate a bit :).
I wouldn't consider the owner's friend's review to be "fake." As for the owner him/herself, that might feel a bit unethical, but I still wouldn't call it "fake."
If there exists a material connection between the owner and the reviewer which is not disclosed in the review then the FTC considers this to be deceptive advertising. A material connection is defined to be something "that is, important to a consumer's decision to buy or use the product." See here: http://www.business.ftc.gov/documents/bus35-advertising-faqs...
So while you might not consider such a review as fake, my understanding is that the FTC would.
Can you articulate exactly why you wouldn't value that review? I would only consider a review "fake" and discount it if I thought the reviewer hadn't actually experienced the item being reviewed, or was externally incentivized (e.g. paid) to compose the review. Merely discounting a review for being "biased" doesn't seem wise to me. Presumably, the childhood friend of Obama actually did enjoy the book, and the friend of the restaurant owner probably does enjoy the restaurant, even if those preferences are influenced by their relationship with the item owner/producer.
> Merely discounting a review for being "biased" doesn't seem wise to me.
To me, fake and biased are not too far apart considering the context that I expect reviews on books, restaurants, etc. to come from random people who don't have any particular reason to like or dislike the product outside of the quality of the product itself. I believe friends & family are too close to the product to give a review without external-factors influencing. It's true that the friend could have truly enjoyed the product, but I still think the chances that their review was influenced by their friendship is high enough to make the review suspect.
> I expect reviews on books, restaurants, etc. to come from random people who don't have any particular reason to like or dislike the product outside of the quality of the product itself.
The problem with that expectation is that voluntary reviews already self-select for people willing to write reviews, which will almost certainly introduce strong biases.
Hmm, I consider the people who are passionate enough about their experience to write a review to be the exact people I want to hear from. Of course, I'm aware of the tendency of people to complain more often than complement. If everything goes well with a product, it's like "ah, nice." But when it fails, it causes feelings of being ripped-off and to resolve those feelings means complain-on-the-interwebz.
But, I guess that's the best we've got, right? Otherwise how do we go about getting reviews for anything? I definitely think forcing people to do reviews that they didn't want to write would produce mostly garbage.
Hi everyone, co-author of the paper here. I wanted to clarify the 20% figure. We actually have no way of telling how many, or which reviews are fake. We do not directly observe review fraud, and we clearly spell this out in the paper. The 20% figure represents the percentage of filtered reviews on Yelp. These are reviews that Yelp finds suspicious enough to not publish. Some filtered reviews may be fake, and some might just be false positives. Similarly, Yelp's filter might miss some fake reviews, and end up publishing them. (See http://www.yelp.com/faq#filter_wrong). I hope this makes the distinction between fake and filtered clear.
Our main goal with this paper was to analyze the economic incentives behind review fraud, and for this we used filtered reviews as a proxy for fake reviews. bobf provides a good summary of our key findings so I won't repeat them. For those interested, you can read more here: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2293164
Wow, that is dramatically different than what the headline suggests. The "20% are fake" headline implies "20% of visible reviews", but instead I'm hearing you say, "20% of reviews are killfiled before visibility."
And, it's not just the headline. Consider the following:
>A Yelp spokeswoman says its software helps filter many fakes before most users get to read them.
When combined with the headline, that really makes it sound like two discrete groups: the fake 20% that make it to Yelp, vs. the "many" that are filtered before making it to Yelp.
The difference is so vast that the headline now feels like link-bait.
Filtered reviews are a terrible proxy for fake reviews!
Yelp is widely believed (among business owners) to wield review filtering/deletion as a club to "encourage" small businesses to buy advertising on yelp.com.
The economic incentives behind review fraud seem pretty obvious to me.
That's why we checked whether advertisers are getting preferential filtering treatment. (They aren't according to our data.) Read my previous comment, or better yet read the paper where we also explain possible pitfalls in the analysis.
You might want to rethink that then; I found one of my reviews filtered once, I have also been a "Yelp Elite" member so I am really unclear what criteria they use for the filtering. I also found some clearly fake reviews for the same place where my review was filtered. It was weird that they filtered my review as well because (I think) it was fairly balanced and mostly positive...
I don't know what goes into Yelp's filter either. I also agree that it can make mistakes. At the same time, I do think that the filter is more likely to catch a fake review than it is to catch a real review. This is the assumption our analysis relies on. (Not sure what I should be rethinking...)
While I read such anecdotes, I cannot rely much on them for research. Instead, what we did for our paper was to collect thousands of filtered reviews, and check their characteristics against reviews Yelp publishes. We find that filtered reviews are shorter, more likely to be from one-time reviewers, more likely to be extreme (1- or 5-stars), and so on. I realize this does not constitute absolute proof but at least it's evidence based on data. Check the paper for more stats.
Beyond that, and as a counter-anecdote, I have read hundreds (if not thousands!) of filtered reviews over the past year and in my experience they are more likely to be fake. Furthermore, one implication of challenging our assumption is that you are suggesting consumers would be better off reading, and basing their decisions on filtered reviews since -- by your assumption -- they are more likely to be genuine. As I said, I cannot offer proof, but given what I have seen I believe the filter to be more likely to catch a fake review than a real review.
From my own anecdotal experience, the filtering is crap. I just reviewed a local restaurant recently. It happens to be one of the best restaurants in town. It has only one review on yelp, a 1-star review, from someone who seemed to have unusual expectations. It also have 2 filtered 5-star reviews from people who loved it.
From what I can tell, all 3 reviews were kosher. Unfortunately because yelp incorrectly filtered the two 5-star reviews, the restaurant has an unfounded poor reputation on yelp. I left a 5-star review, and as far as I can see my review isn't filtered (at least, not when I view it). Perhaps that is because I'm a long-time yelp user.
Argh, I am so sick of seeing this article. From the study, 16% of all Yelp posts are identified by Yelp as fake, hidden from view, and not counted into a business's average score.
This tells us nothing about the percentage of non-removed Yelp reviews that are fake, because all non-removed reviews are assumed, in the data, to be real.
This is linkbait and a complete misrepresentation of the data.
While I agree with you about the title of the article, it is not true at all that "all non-removed reviews are assumed, in the data, to be real." In fact, in our analysis we explicitly allow for the filter misclassifying reviews (that is to say filtering real reviews, or publishing fakes.)
Sorry about my imprecise wording. I wasn't intending to criticize your study, which did acknowledge potential misclassifications (and was very interesting and deserves attention on its own merits).
I'm criticizing MW's interpretation, which assumed that your proxy was very accurate in terms of providing the proportion of fake reviews, but went on to imply that your proxy was highly inaccurate by talking about fake but visible reviews.
I'm sure there will be a lot of discussion here about the accuracy of the 20% figure, but that actually doesn't seem as important as the other findings from the research this article cites ("Fake It Till You Make It:
Reputation, Competition, and Yelp Review Fraud", http://people.hbs.edu/mluca/FakeItTillYouMakeIt.pdf).
The interesting points from the research include:
1) Low ratings increase incentives for positive review fraud, and high ratings decrease them.
2) Having more reviews reduces incentives for positive review fraud.
I would call all of those points extremely obvious and hardly worth a whole study.
If you have low ratings, of course you're tempted to get some fake positive reviews. If you have a lot of reviews, adding a few positive ones will not affect the overall score much--it's a drop in the bucket. Chain restaurants, well, everyone knows what McDonald's is, what would they gain from posting fake reviews?
Businesses that claim their Yelp page have more interest in Yelp? Shouldn't it be the other way around, businesses who have decided they're interested in Yelp will claim their page? Claiming your Yelp page is a sign that you care; businesses that do not care about Yelp will not commit review fraud.
Who else would post negative fake reviews but competitors? I guess reputation management companies could post negative fake reviews in an attempt to drum up business.
An increase in competition encourages more negative fake reviews? I mean, if a business has no competitors, who has an interest in posting fake reviews? This ties to #5, where competitors are the ones who post negative reviews. More competitors, more fraud.
It's nice that these points seem obvious to you, but until you have data and a sound statistical analysis, people won't have any reason to believe them. Lots of "obvious things" turn out to be true, but lots turn out to be false too.
Whenever my dad and I are looking for a place to eat I pull up Yelp. He grumbles about the possibility of fake reviews, but I tell him that as long as you look at the sample size (n=25 or more is usually safe) then you shouldn't have a problem. I've found amazing restaurants (and many other services) through Yelp that I wouldn't have found otherwise.
Sure, there have been times where I've eaten at a highly-rated restaurant and thought to myself that there must have been some positive review fraud at work. At the end of the day I have tried something new, and I still have my whole life ahead of me to try other things. Nothing is forcing me to go back to said restaurant. And I generally leave a review questioning the merits of the establishment.
The positives of Yelp outweigh the negatives and the NYC regulation is a step in the right direction.
Does anyone really go by the star ratings on Yelp? I read the reviews but I don't pay any attention to the overall ratings. One of my favorite places to go is a 2-star, and our local Outback is rated a 5.
Plus with the fact that there's no standard, some people rate everything a 4 or 5 star while others reserve a 5 star for the best places in the area (or even the genre) to eat.
What I know is true: I've reviewed many places, yelp 'filters' my reviews as fake even though they are legitimate. So 20% of the ones they don't filter are fake, but several of the ones they do filter aren't fake? Sounds like the site is just fake.
Yelp exists basically because it shows up in Google search results.
Google should be promoting sites that provide value to their users.
Considering the quality of Yelp, I really don't think they should be showing up in the search results.
However, it would take a huge culture change at Google for Yelp to stop showing up in search results, IMO. Even if the quality of their reviews continues to degrade.
I run a small review site and have to deal with issues surrounding fake reviews all the time. Every now and then someone makes a rookie mistake and it gets picked up by my system and automatically filtered but as the years have gone on, I've found people are becoming better at gaming us.
I believe there is value in reviews, especially at a local level; I've found out about some really great local businesses based on the reviews but I'm so tired of people trying to game us. I'm tired of business owners who would rather argue with me for months on end and threaten legal action to have a totally legitimate negative review removed, or have their friends post glowing reviews, rather than just providing a good service and recognising they can't please everyone.
I wish there was a sure fire way of determining a persons intention when they write a review but it's impossible, so this problem will remain unsolved.
You could review them and say they offered a good discount with everybody saving face and you between the lines writing that the discount was the only thing worth mentioning about the place ;)
Another problem I've seen with Yelp is reviews that aren't actually for the intended place. My father-in-law owns a few restaurants. One of them has >300 reviews (solid 4 stars), but shares part of its' name with another restaurant with >1500 reviews (3.5 stars, with a heavy downward trend).
Most reviews posted for his restaurant are either 4 or 5 stars, with the majority of lower reviews referencing the other restaurant either by name, menu items, or decor. I'm not sure if the issue is one of UI design, mobile app, or poor search handling - although I heavily suspect search is the culprit.
Unfortunately there doesn't seem to be a way to address these mis-posted reviews, that I've seen. I would love to hear any suggestions that would help though.
Change your name. Seriously. Even without Yelp, you don't want potential customers hearing via word of mouth about how bad your restaurant was when it really was the one down the street.
Names are relatively cheap to change and if you select a name that is related (like say, shortening it or incorporating part of the old name in the new one) and communicate what is going on with regulars, you'll limit any short-term decline in business.
That's a perfectly reasonable suggestion, but I don't foresee it happening. They've been there for >10 years, and have been very successful. The name isn't actually similar, and they're in very different parts of the city. There generally doesn't seem to be any real confusion except for a few Yelp reviews.
I worked at a review site and have decided that if I ever open a restaurant I will name it 'null' or 'NullPointerException.' The hilarity that would ensue from this would be priceless.
For the record: if you go and flag a review, in addition to the usual "abusive/TOS violation" type of stuff, you can also report that it "doesn't describe a personal consumer experience." It's not super obvious, but that's the easiest way to dispute someone who's not actually reviewing your restaurant.
I tend to use that to flag those jerkface reviews that are basically "I would never go there" rants.
I find the same thing. I think the difference is that since 4sq is soliciting tips instead of reviews, they're harder to convincingly game. If I'm looking for a restaurant in an unfamiliar city and see "definitely try the squash risotto with balsamic drizzled" that means a lot more to me than "everything here was yummy!" Without a star rating to game with 1- or 5-star reviews, there's less noise to sift through.
That said, I assume that their 0.0-10.0 ratings they introduced a few months ago are basically random.
"You would expect it to be in Yelp's interest to demonstrate its reviews are credible, though I haven't seen that. Competition and articles like this should motivate them in that direction. In the meantime, they benefit from charging businesses to allow positive reviews to appear and to suppress negative reviews (if they're doing that, it would explain why they don't focus on credibility)."
I think that you haven't been looking for it, then: that exact thing is hugely important to Yelp and is why they filter out so many reviews in the first place!
(Mostly answering to the deleted grandparent here.)
"In the meantime, they benefit from charging businesses to allow positive reviews to appear and to suppress negative reviews (if they're doing that, it would explain why they don't focus on credibility)."
We try to address this issue in our paper. Take a look at Section 3.4. Short summary: we do not find any noticeable differences in filtering between advertisers and non-advertisers, but there are limitations to our analysis.
And when Yelp does filter reviews, they routinely are accused in forums like this one of running a pay-to-play scheme.
A business owner hears from a Yelp sales guy about the benefits of sponsoring their page (top placement for a good review of their choice) and hears it as a "threat" to hide all those one shot five star reviews they talked their regulars into writing last month.
>And when Yelp does filter reviews, they routinely are accused in forums like this one of running a pay-to-play scheme.
Exactly. And, as well they should be. No business should base its revenue on taking money from those it reviews. Period. I'm not sure why this is even a gray area. Even if it runs as clean an operation as possible, it invariably will come off as extortionate at some point.
The BBB has been running this scam for so long and it amazes me that the FTC hasn't put them out of business.
As a small business owner, I have been on the receiving end of their extortion. A single complaint in 5 years, and they extort you into responding through their system, thus adding value to their business. They literally threaten that if you do not respond, they will be "forced" to give you a negative rating. They know nothing about your business and could not assess the good faith validity of any response, yet have no problem making representations about you to the world, while asking you for money.
This whole third-party review service just lends itself to abuse all the way around.
"There is one site where users share opinions with greater transparency than Yelp or Google: Facebook. The world’s biggest social-networking site prohibits the use of fake names."
Uh, I don't really think Facebook's real name policy has any effect here (it's policy is actually a lot weaker than Google in practice afaik). I do think people share their real opinion on Facebook more often but I think that's because of Facebook's huge audience. People basically want to share their real opinions with their real friends (not necessarily their friends there under real name). And people don't really have that many opinions about just plain products. And when people do, they mostly feel kind of sleazy sharing those opinions with the world for commercial purposes. Facebook is always trying to get people to integrate their socializing with advertising/commercial-recommendations but it has been unsuccessful and would probably start to truly lose users if it really forced people's hands.
The problem is most people care so little about the products they buy that they'd rather not write a review. That's unlikely to change.
My father-in-law is a veterinarian, and he got a call one day from a Yelp advertising sales person, who made it clear that if he didn't buy advertising from them, negative reviews would start showing up. So I just treat Yelp as legal extortion.
I'm a business owner and don't want my business to be listed on Yelp — can I have it removed from Yelp?
Consumers have the right to talk about what they like (and don't like) about a meal they ate, a plumber they hired, or a car wash they visited. We don't remove business listings, so your best bet is to engage with your fans and critics alike, and hear what they have to say.
The real news is that NY state is prosecuting fake paid reviews as fraud, the WSJ is reporting on it, and that HBS released a study with quantification of it. Also, this is another sign that the "Wild West" days of the Internet have ended.
"20% of Yelp reviews are fake" is what is being presented by the author as 'real news'.
If he wanted to go your way as the 'real news' he would have headlined it sometime like:
"NY state starts prosecuting fake Yelp reviews" or something along those lines.
I recently posted 2 reviews on Yelp (I'm not very active on it; this was probably my 2nd and 3rd ever). I gave rave reviews for both. One is filtered, one is not. I see a 50% failure rate in filtering, from my extremely small sample size. Oh, and one of them was in NYC (the non-filtered one). The filtered one, well the chef knows who I am, since I said the same thing to him at dinner that I posted in the review.
Unfortunately that behavior pattern is exactly one that abusive users exhibit as well: innocuous review, long delay, glowing review of "target" business.
It doesn't matter. Over time, the value of all un-curated reviews approaches zero. One of the oldest review platforms, IMDB, being the prime example of this.
I don't often agree with Nicholas Carr, but in this particular case, the wisdom of the crowds fails miserably, with or without astroturfing.
> There is one site where users share opinions with greater
> transparency than Yelp or Google: Facebook. The world’s
> biggest social-networking site prohibits the use of fake
> names.
Facebook isn't that transparent. You can't scrape Facebook.
And I wouldn't be at all surprised to find out that 20% of Facebook profiles are fake. About 80% of the friend requests I've received over the last year have been from people I don't know; the majority of which are clearly bots.
Anyone want to speculate on what percentage of Angie's List reviews are fake? On one hand you have to pay to leave a review. On the other hand, Angie's List gets paid if a business registers twenty fake accounts to post fake reviews.
Flagged - The headline is VERY misleading. Read the comments from the co-author. 20% of reviews are voluntarily filtered by Yelp. Though the issue is certainly real.
When you think that the majority of companies have less than 10 reviews, and most owners can't resist doing a review, and having a friend do a review you are at 20% really fast.
When you combine that with organizations like Blackwaterops.com (my old company) that used to do 1000s of reviews a day. 20% is really low, and wrong.