Chevrolet’s have gotten under the skin of TV watchers for years now, and even inspired that are almost as famous. Yet they soldier on, and in one recent spot, Chevrolet tries to convince “real people” that they may have made the wrong choice by purchasing a Honda, Toyota or Ford. Using some dubious survey data, Chevy is now claiming that they make “more reliable” cars than their competition.

If you haven’t yet seen the ad currently being blasted across the airwaves, here it is, and it’s exactly what you would expect:(

Naturally anyone that has had experience with Hondas or Toyotas and Chevrolets may have their bullshit meter pegged at the “most reliable” claim. While General Motors has substantially boosted quality in recent years, the big Japanese automakers are still tough to beat.

So how exactly does Chevy come to that conclusion? Through a survey conducted by a market research company called Ipsos. Chevrolet , but here is the rundown:

This study, performed by Ipsos, surveyed owners of Chevrolet, Buick, GMC and Cadillac versus competitive brands in the area of parts replaced or repaired in the past 12-months – excluding fluids, filters and those related to accident/collision.

For the purpose of this study, reliability is defined as the percentage of vehicle owners who reported they have not repaired or replaced any vehicle components in the past 12 months (excluding fluids, filters and those related to accident/collision).

Using a nationwide sample, a total of 840,979 non-sponsored letter invitations were mailed to 2015 Model Year vehicle owners (time in service starting December 2014 to June 2015). Via the survey, respondents were screened to ensure they still owned or leased the vehicle and ensure they were the original owners of the vehicle. All data was weighed to reflect the population of model ownership.

In total, 48,679 surveys were completed. Brands included in the survey: Acura, Audi, BMW, Buick, Cadillac, Chevrolet, Chrysler, Dodge, Fiat, Ford, GMC, Honda, Hyundai, Infiniti, Jaguar, Jeep, Kia, Land Rover, Lexus, Lincoln, Mazda, Mercedes-Benz, Mini, Mitsubishi, Nissan, Ram, Subaru, Toyota, Volkswagen and Volvo.

I made multiple requests to Chevrolet to send me a full copy of the report. They did not oblige, but they did send me a “summary” that clarified a few points.

First, the survey was sent in 2018 to owners of 2015 model year cars, and the reporting of the “repairs in the past 12 months” relates to those 2015 cars in their third year of service. Chevy reiterated that “Independent statisticians reviewed the materials and concluded with 95% confidence thatChevy’s percentage of no parts repaired or replaced is better than Toyota, Honda, Ford or 23 other brands.”

When it comes to the components themselves, Chevrolet said:

The owners were asked what parts they have replaced or repaired on their vehicle in the last 12 months. They were given a listing of over 50 parts that might have been replaced as well as an “other” catch-all. Examples of these parts are engines, transmissions, AC components, fuel pump, radio, spark plugs, batteries, shocks/struts, etc.

Anyone who has studied research and statistics will tell you that it’s not difficult to form the conclusions you want to based on your use of what is called an “operational definition.” For this particular study, Ipsos is operationally defining reliability as the repairs a car has had within a 12 month period once the car has reached three years old.

In their summary, Chevrolet cited that this operational definition of “reliability” is consistent with other definitions such as the one from the SAE.

SAE (from JA1000/1 “Reliability Program Standard”)

“Reliability—The ability of a product to perform a required function, under stated conditions, for a stated period of time.”

However, just because a car had components replaced over a 12-month period doesn’t necessarily mean the product was not performing its required function, but an owner of a vehicle may consider a car “unreliable” if they have to continually make repairs on something. Also, it is that “period of time” component to SAE’s definition that is a matter of perspective when determining a car’s “reliability.”

When people go to buy a reliable car, they want something that is not going to break down and cost them money well beyond the expiration of the warranty. The number of components replaced within one year doesn’t paint a complete picture of how that car will hold up over a much longer period of time.

Setting aside the fact that , fewer than 49,000 respondents completed them out of almost 840,000. What Chevrolet did not provide is a breakdown of those respondents by brand, because if we were looking at the percentage of repairs made by various brand owners this could make a difference.

For example, if only 1,000 of those 49,000 respondents were Honda owners and 100 of them reported repairs within that 12 month period, that means that Honda would have a 10 percent repair rate. But if 15,000 of the respondents were Chevrolet owners and 1,200 of them reported repairs, that is an 8 percent repair rate. Yet this distribution was not provided so it’s difficult to ascertain exactly how those percentages compare within each sample size of brand owners.

Another key factor that was not presented was the number of miles driven for each respondent. Owners who put more miles on their car within a three-year period are more likely to have to replace components, but that doesn’t necessarily mean that car is unreliable.

Data can be collected and organized in a way to tell whatever story you want it to tell. Not long ago we published some data from . The list was dominated by Hondas and Toyotas, but not one GM car made the cut. And in , Consumer Reports ranked both Honda and Toyota well above Chevrolet that occupied the lower part of the list along with several other domestic brands.

But like Chevrolet’s survey, this data is also self-reported based on the ownership experiences of Consumer Reports subscribers.

While leasing is more popular than ever these days, a lot of people still want to buy new cars for…

I also reached out to, as they are a respected reference when it comes to these kinds of “quality” rankings and asked them to comment on the methodology behind Chevrolet’s study. They declined. (Recall that in this very same series of ads.)

Are Chevrolet models “more reliable” than Hondas and Toyotas? It all depends on the data you are looking at. . And that is why these send the message that they are always playing defense.