ONLINE RATINGS SERVICES

Consumer Reports: The truth about Angie’s List, Yelp, and more

Online ratings services promise unbiased reviews of local businesses. Do they deliver?

Published: September 2013

Need to find a good plumber, hairdresser, or auto mechanic? If you’re like a lot of people, you’re happy to turn to online ratings services to get a recommendation.

More than 100 million consumers checked ratings for companies in 97 U.S. and international markets in just the first three months of this year, according to Yelp. Some 2 million households subscribe to Angie’s List for its version of the skinny on local service providers in 244 markets in the U.S. And Google now gives you local business ratings, whether you want them or not, by serving them up while you’re busy navigating Google Maps.

Sure, it can be convenient to find out what others think of a handyman’s skills before you hire him to retile your kitchen. But how trustworthy are the opinions? Quality controls are necessary to ensure accuracy and reliability, because the scores can be manipulated by self-interested parties. So we decided to learn the methods of five leading rating providers serving the San Francisco Bay area: Angie's List, Consumer's Checkbook, Google+ Local, Porch, and Yelp, as well as the Better Business Bureau. (None of these services offers ratings everywhere.) Unsettling surprises came fast and frequently. Here’s what you should know about the companies we examined, listed in alphabetical (not ranked) order.

Angie's List

• Years in business: 18
• Structure: Publicly traded company
• Cost for consumers: Varies; $46 a year in San Francisco
• How it works: Companies set up free online profiles or involuntarily get one when a member rates them. When they get two reviews and a B average or better, and there are no alerts about them, they can pay to advertise and must offer discount coupons that “position your business to rotate on page 1 of search results,” Angie’s List says.
  Consumers subscribe to gain access to highly rated service providers. After they hire a business, they’re asked to rate it based on nine measures. They also have to affirm that the information they provide is true and accurate and that they’re not an employee, competitor, or in any way related to the service provider.
• Caveats: We think that the ability of A- and B-rated companies to buy their way to the top of the default search results skews the results. Cheryl Reed, a spokeswoman for the company, disagrees. “We don’t believe that,” she says. But Angie’s List marketing materials intended for businesses say that companies that advertise get “an advantage of increased exposure” that “can propel you ahead of your competition.” They get 12 times more profile views than companies that don’t buy ads.
  Angie’s List encourages businesses to solicit reviews by giving customers free, postage-paid forms, stickers on thank you notes, and Web links embedded in e-mail invoices. But experts who study survey techniques say that can create a bias for positive reviews.
  Angie’s List misleads consumers by prominently promising that “businesses don’t pay” and that it’s a consumer-driven service supported by membership fees. But almost 70 percent of the company’s revenues come from advertising purchased by the service providers being rated. Angie’s List tells consumers that it provides “reviews you can trust,” and takes steps to detect and remove fraudulent positive and negative reviews. But company investment disclosures say that “we cannot guarantee the accuracy of our reviews.”

Consumers' Checkbook

• Years in business: 37
• Structure: Nonprofit
• Cost for consumers: $34 for two years
• How it works: Local companies are involuntarily rated by the Center for the Study of Services, based on surveys of Consumers’ Checkbook’s own subscribers, Consumer Reports subscribers, and CSS’ own research. (Consumer Reports gave CSS $25,000 in matching funds to get started in the mid-1970s and has allowed Checkbook to survey its subscribers in the seven metro areas it covers.)
  CSS’ own research includes conducting undercover price shopping and gathering complaint data from regulators (such as the state attorney general’s office) and the Better Business Bureau. Subscribers access detailed ratings online and in a biannual print magazine covering their metropolitan area.
• Caveats: We found little to fault here, except that in some cases a business rating may be based on as few as 10 users. But Checkbook provides complete transparency and guidance about how to assess those ratings compared with companies with more users. And since those numbers come from known subscribers objectively surveyed, they don’t carry the same tarnish as reviews from people who voluntarily review for some unknown motive.

Google+ Local

• Years in business: 1
• Structure: Publicly traded company
• Cost for consumers: Free
• How it works: Anyone can search Google+ Local by city, state, and business type to find reviews, which are simple one- to five-star ratings and commentary. Google is mum about how an overall rating is calculated beyond saying that it’s based on user ratings—no details on how users are verified—“and a variety of other signals to ensure that the overall score best reflects the quality of the establishment,” according to the Google+ Local website.
  To write a review, you must create your own personal Google+ profile, typically using your real name. That does provide something of a reality check, because you can click on the reviewer and see how many reviews she has written, and click further to her Google+ profile and scan or read all her reviews to assess whether she tends to be effusive or disgruntled. Businesses create their own pages to connect with their customers and listen in.
• Caveats: Google encourages businesses to reward their fans with coupons and to try to resolve customer-service problems. We can’t argue with that. But from a ratings reliability standpoint, this can skew the ratings positively, because assuaged customers can always delete their previously negative reviews. Google didn’t respond to two requests for an interview.

Porch

• Years in business; structure: 1.5 years in beta, launched September 17, 2013
• Structure: Privately financed startup
• Cost for consumers: Free
• How it works: Porch.com CEO Matt Ehrlichman describes the site as a “home improvement network” that lets visitors see photos and details about remodeling and repair projects that have been done in their neighborhood (drawn from 90 million around the U.S.), how much they cost, and which of some 1.5 million service providers did the work. The project’s home address is not shown, to preserve privacy.
  The 100,000 homeowners who have signed up to date can find a map showing projects in their area that service providers have listed, whether it’s a $5,000 roof repair or $50,000 kitchen remodel. Or they can browse photos for inspiration and build a scrapbook.They can also search for professionals for their specific project or repair. We haven’t been able to try out the site ourselves—Porch did not grant us early access—so we’ll provide an update when we get a look after launch.
• Caveats: Ehrlichman says Porch is like a LinkedIn resume for service providers, which means the service providers control their profiles and are the source of all the key information. By design this is a “positive endorsement model,” says Ehrlichman. Unfortunately, then, what makes one professional seem better than another will be driven by how many endorsements he or she solicits and gets from customers. If you’re looking to learn about possible problems, you won’t find them here.
  This also means you must rely on the contractors themselves to tell you how good they are, in an industry that has historically not fostered much trust. For example, last year, 8.6 million consumers found it wise to inquire at the Better Business Bureau to check the reputations of roofing contractors, general contractors, plumbers, and construction/remodelers, who wound up in the top 10 for all BBB inquiries and rated 16th through 42nd, respectively, for complaints out of some 5,000 business categories tracked.
  Another problem: While Yelp and Angie’s List learned early on that bogus user reviews were an Internet reality that could undercut their credibility and can’t be ignored, Porch has no systems in place to guard against them. There are also no sanctions for service providers who get caught gaming the system, nothing to prevent contractors from buying endorsements with discounts or freebies, no background checks, and no mechanism that enables its 27 employees to verify the details of 1.5 million contractors and their 90 million projects in the system.
  “We don’t need it at this point. I absolutely imagine that we’ll have people that are actually working on this as we go on,” says Ehrlichman. 

Yelp

• Years in business: 9
• Structure: Publicly traded company
• Cost for consumers: Free
• How it works: Anyone can look up a company on Yelp to see its overall rating and individual reviewers’ ratings and comments. If you want to write a review, you need only set up a user name and provide your e-mail address and ZIP code. That creates a profile where your reviews are gathered. The more prolific you are, the more trusted and “known” you become in Yelp circles.
  Yelp uses software that aims to filter out bogus reviews and keep legitimate ones, but it hasn’t been objectively tested, says Kristen Whisenand, a company spokeswoman. About 70 percent of Yelp’s 36 million reviews have been posted unfiltered; 22.5 percent have been filtered and separated from the overall average rating for each company but can still be read if you want; and 7.5 percent have been deleted.
  Companies appear on Yelp involuntarily, but they can claim their page and gain access to tools that let them contact reviewers publicly or privately through the site to work out problems.
• Caveats: A company’s ability to make amends with a negative reviewer—while good from a customer’s perspective—undercuts the integrity and accuracy of the ratings, because placated gripers can change their review at any time. “We see plenty of people go back and delete reviews or update them,” Whisenand says. Yelp can also remove reviews that businesses and individuals flag as violating content rules.
  Some data reveal that user reviews skew toward the positive. Researchers at the University of Texas at Dallas and California State University, Long Beach, found that people who rented movies online were five times more likely to post a positive review than a negative one. And 66 percent of Yelp’s reviews for the first quarter of 2013 were four or five stars. Services like Yelp could adjust their ratings to eliminate such measurable bias, but Yelp says it simply averages the raw unfiltered ratings to calculate a company’s overall score.
  Rated service providers can buy sponsored search results and put them at the top of the list for someone searching for, say, a restaurant in Los Angeles. That’s not unusual among search engines, and Yelp identifies the listing as sponsored, but that still gives one paid advertiser per search an advantage over the natural results that Yelp otherwise promises.

What about the Better Business Bureau?

The Better Business Bureau is not a review site, per se. Ratings result from the dispute-resolution process, and staffers verify that complainants are real customers. It also offers user reviews now, but they don’t factor into a company’s rating. Consumers can check BBB company ratings online, find details of individual complaints, and get help in mediating disputes that can’t be worked out directly with a company; they looked up businesses at bbb.org almost 125 million times last year.

• Years in business: 101
• Status: nonprofit
• Cost for consumers: Free
• How it works: Companies get a profile when a consumer files a complaint; 985,000 were logged last year. Ratings are calculated using a mathematical formula based on how the company responds to each complaint, how promptly, and how well it’s resolved. Complaint volume and patterns factor in, too. Businesses can also apply for BBB accreditation after they maintain a grade of B or higher, and if they pledge to adhere to BBB standards. Accreditation itself doesn’t influence the company’s grade but allows it to use the BBB logo in its marketing.
• Caveats: Local companies aren’t the only ones subject to BBB scrutiny. Last March the BBB of Los Angeles was itself expelled from the national organization for failing to adhere to standards on handling complaints, accreditation, and reporting on businesses after a two-year investigation following a report by ABC’s “20/20” program. A new local BBB chapter has taken its place to repair the damage.

1 plumber, 5 ratings

Here’s how one plumbing business serving the San Francisco Bay area was rated by these services as of late July. (Porch launched after we looked at this plumbing service.)

Angie’s List: Overall grade of F on the strength of just one unhappy customer, who also gave the company the same failing grade for responsiveness and punctuality.
Better Business Bureau: A+ rated; two complaints resolved; accredited BBB business since 2003.
Consumers’ Checkbook (Center for the Study of Services): Of 63 consumers surveyed, 40 percent said it was “superior” for overall performance, 27 percent said it was “adequate,” and 33 percent rated it “inferior.”
Google+ Local: No user reviews or ratings.
Yelp: 2.5 stars out of a maximum of 5 based on 20 reviews.

Editor's Note: This article appeared in the October 2013 issue of Consumer Reports Money Adviser.