The Lists Are Not Data. They’re Products.
Search for the best airline in the world, or the best resort on a stretch of coast you’ve never heard of, and you get the same thing every time: a tidy top ten, gold badges, a “winner” crowned in eleven different categories, and a shiny seal that somehow ends up on the homepage of every property in the top five.
It looks like analysis. It reads like science. A lot of it is advertising with a spreadsheet stapled to it.
Here’s the part the lists never explain: the ranking business is a business, and the thing being ranked is usually you.
How Airline Rankings Actually Get Built
Most airline rankings blend three kinds of inputs: hard operational stats, subjective survey scores, and editorial opinion. Only the first category is really auditable.
The inputs you’d expect
- On-time arrival percentage
- Cancellation rates
- Baggage mishandling per thousand passengers
- Fleet age and cabin refresh cycles
- Seat pitch and configuration
- Complaint volume per passenger boarded
- Route network reach
The inputs nobody audits
- “Cabin comfort” scores from small panels of travelers who flew a complimentary premium seat
- “Crew friendliness” ratings — a vibe, sampled from a handful of flights
- Reader polls where the vote count is never published
- Editorial scores weighted however the editor feels that quarter
Where the numbers get quietly massaged
On-time performance is the most gamed metric in aviation. Here’s how:
- Schedule padding. Airlines add buffer minutes to the published block time. A flight that “arrives early” may have simply been given a longer scheduled duration than the route requires.
- Cancellation over delay. A cancelled flight isn’t a late flight. Depending on how a ranking treats the two, it can be smarter to scrub the flight than to run it four hours behind.
- Regional outsourcing. Regional partners’ performance sometimes lands in a different bucket than the mainline brand’s.
- Complaint dilution. Complaints per passenger improves automatically when a carrier adds millions of passengers who never bother filing anything.
And seat “comfort” scores are usually collected in the best cabin on the newest aircraft. That tells you nothing about the row you’ll actually be sitting in.
How Resort Rankings Actually Get Built
Star ratings are all over the place
Some rating systems send an actual inspector. Others let a property self-assess against a checklist, pay a fee, and receive a plaque. Both produce the same shiny four-star logo. The consumer can’t tell which is which because nobody labels it.
The review score problem
- Review gating. Guests get asked for a review while standing at the front desk with a welcome drink in hand — and only happy guests get the follow-up email.
- Incentives. Discounts, upgrades, or loyalty points offered in exchange for a five-star write-up.
- Sample size theater. A property with 14 reviews and a 4.9 average is not better than one with 4,000 reviews and a 4.3. It’s just newer.
- Stale scores. A score built over six years says nothing about the renovation that happened, or didn’t, last season.
The average star rating is nearly useless. The distribution is what tells the story. Look at how many one and two-star reviews exist, and when they were posted.
The photos are not the property
Wide-angle lenses make a standard room look like a suite. Property photos are often taken the week after a refurbishment. The room type shown is usually the top category, not the one you’re booking. And “ocean view” has been stretched to include a sliver of blue visible past a parking structure.
The Hidden Economics of “Best Of” Lists
Nobody runs a ranking operation out of charity. The money comes from a few predictable places:
- Affiliate commissions. You click the winner, you book, the list publisher gets a cut. The ranking exists to route your click.
- Badge licensing. Properties pay to display the award logo on their own marketing.
- Sponsorship and ad packages. Often sold to the same companies competing for the top spots.
- Event and gala revenue. Award ceremonies are ticket-and-table businesses.
None of this automatically makes a list wrong. But it does mean the ranking has an incentive structure, and you should read it the same way you’d read a restaurant review printed on a restaurant’s own menu.
Build Your Own Scorecard Instead
This is the actual answer. Ten minutes of work beats a decade of awards.
Airline scorecard
- On-time rate, adjusted. Compare scheduled block time against actual flight time on the specific route you fly. Padding shows up fast.
- Cancellation rate as a separate number. Not folded into delays.
- Seat pitch in inches. Not “comfort.” Comfort is a marketing word; pitch is a measurement.
- Irregular operations handling. When a flight dies, do they rebook you on a competitor, or only on their own metal three days later? Do they cover a hotel?
- Refund behavior. How fast does money come back when they cancel?
- Fleet consistency. A great flagship cabin means nothing if it’s on one route.
- Loyalty program reality. Award availability on the dates you’d actually travel, and whether points expire.
Resort scorecard
- Read only the two and three-star reviews. Five-star reviews describe a good mood. Middling reviews describe the building.
- Filter to the last 90 days. Anything older is history, not information.
- Look at guest-uploaded photos, especially ones taken in low light with no filter.
- Get the full fee list in writing. Mandatory daily fees, parking, internet, and “included” breakfast that isn’t.
- Check the room category map. Know exactly which category your booking lands in.
- Satellite view. Construction sites, highways, and industrial neighbors don’t appear in the marketing shots.
- Ask two questions directly: Is anything under renovation right now, and is anything seasonal closed during my dates?
Red Flags in Any Ranking
- No published methodology
- Methodology published, but category weights are hidden
- The winner in three categories also happens to be an advertiser
- “Voted by readers” with no vote count
- No date on the page — you might be reading a list from four years ago
- Every category winner is a different company, and all of them are large
- Photos of awards, not photos of the product
The Only Ranking That Matters
The best airline is the one that flies your route on time and rebooks you fast when it doesn’t. The best resort is the one whose worst review describes something you can personally live with.
Everything else is a badge someone paid for. Use the lists as a starting point to find names you didn’t know existed — then throw the rankings away and score the options against your own priorities, your own dates, and your own tolerance for nonsense. Update the scorecard after every trip.
That’s a ranking nobody can sell you, and it’s the only one that won’t lie.