How techies get phone camera reviews wrong…

A camera isn’t the same thing for every consumer.

Even though standalone cameras have become less popular for consumer point and shoot use, we still see an incredible variety of hardware and form-factors. There is no universal agreement on what a camera “is”. People still have entirely unique needs where those differences matter to the individual.

This isn’t just some oblique idea of specialty or “pro” use. At a basic level, someone’s specific needs will be better fulfilled by a tool designed for those needs.

Yet in the world of smartphone camera reviewing, we crouch a significant portion of the conversation behind some idea of “The Avuraj CunZoomer™”. We build phone camera conversations around a fallacy that there is some common or objective agreement on what a camera SHOULD be.

It’s only at the most regressively basic level that we might find some agreement.

You should be able to push a button that results in an image being captured.

Beyond that, cameras should specialize. We should see different pros and cons. That’s what makes cameras exciting.

Balancing Expectations

More often than not, the style of output you are most familiar with will be the output you prefer.

If you’ve predominantly shot with an iPhone, and are familiar with Apple image processing, moving to another platform will FEEL different. It’ll look “wrong” compared to what you’re used to.

Too often, we subjectively grade cameras on ill-defined criteria. We then take those comparisons, and rate them against the number of people who use each camera. If more people buy iPhones, then the iPhone will “win”.

It’s a debate fallacy: Appealing to the majority.

“More people own iPhones, therefore, more people must prefer this style of camera.”

If a different camera processes image data in a different way, it will automatically be categorized as a “loser” in a showdown. If we start with the above fallacy, then every other phone will automatically be “incorrect”.

“We know iPhones have ‘good color’ and the color is different on this other phone, so it has ‘less good color’.”

It’s not difficult to see when reviews start with this fallacy as an “objective” truism.

The most popular phone is already the “best”. Why would the reviewer need to dedicate much time or effort in disproving that conclusion? It’s much easier to begin the review process knowing the more popular phone has already “won”. All you need to do is show how the less popular device is simply different in processing image data.

Remember, in tech review land, “different” output is the same as “inferior”.

Controls Matter

This extends to the control of a camera. I use this analogy a lot:

Take a wedding photographer who has shot on Canon cameras for decades. On the day of a wedding, swap all their gear for Panasonic. The photographer’s output WILL be poorer. Is Panasonic hardware inferior to Canon? Of course not. We immediately recognize that the artist needs to be familiar with those tools to produce the same quality.

We regularly see when a reviewer is unfamiliar with less popular phone brands. We’ll often see how they get small details wrong.

It’s also a giveaway when reviewers speculate on features instead of just showing those features in action. It’s easy to dismiss competing features on less popular brands. Properly showing those features would take time to learn about all the settings on a different phone.

“Why would anyone want to use [useful photo or video feature]? If it’s not on an iPhone, it must just be a gimmick! If a feature is ‘worth it’, it will be on an iPhone!”

That idea lines up with a particularly lazy trend in deciding that all phones should be judged primarily on the out-of-the-box full-auto output.

If an iPhone doesn’t have advanced camera features out of the box, the only “fair” way to compare it against a phone that DOES have advanced features is to IGNORE all of the other phone’s advanced settings.

We HAVE to be “fair” to the predetermined “winner” phone.

In-between reviews, reviewers don’t keep using the smaller brands, they fall back to iPhone or Galaxy.

(I can confirm this. When I’m between reviews, I tend to fall back on LG or Sony devices for MY familiarity in using them as tools to shoot my videos.)

Just as with our above photographer analogy, the same holds true with phone cameras.

The more you use one phone camera, the more familiar you are with it, the less you need to THINK about the operation of that camera. You can more easily focus on your composition, and you will arrive at a better photo with less mental effort.

Even for more casual point and shoot situations, that familiarity contributes significantly to the final image created. I spelled this idea out more directly while reviewing the Pixel 5A camera app here:

Using a new phone for a couple weeks during a review embargo is woefully insufficient to compare against the phone a reviewer REALLY lives with. That’s not really what that camera will be capable of in the hands of someone who OWNS that phone. It’s a disingenuous way to highlight differences between phones.

That’s a massive challenge for producing meaningful camera reviews and comparisons. The USER matters more than the hardware and specs. It takes time to build that proficiency. It takes effort to not only learn about those differences, but be able to communicate those differences to an audience.

If you’re confident that the “popular phone” AUDIENCE will want to hear that their phone has the better camera, why take the time to get proficient at using a less popular phone? That effort might reduce your final profits if you come to a conclusion which contradicts the bias of your core audience.

We also wouldn’t want to risk those PR relationships with larger manufacturers. Getting access to the major phone releases is critical to a reviewer’s bottom line, but that’s a topic for another editorial…

New Technologies Are Exciting

Phone cameras have evolved aggressively over the last several years. We’ve seen an explosion in sensor sizes, image processing hardware, resolution, and dynamic range.

What makes these conversations MUCH more difficult though, is when new technologies are casually compared using marketing bullet points. Instead of taking the time to better understand what new tech brings to the table, we see “NEW” and equate that with “BETTER”, and then trash any solution which isn’t similarly “NEW”.

Sensor size is a brilliant example of how reviewers pick and choose their winners to satisfy their audiences.

Let’s get nerdy with some numbers!

The Pixel 5 used a 1/2.55” camera sensor. That’s smaller than the sensor in an iPhone 12 Pro Max which uses a sensor around 1/1.87″

Both phones have smaller sensors than the Galaxy S21 Ultra. The Samsung main sensor is 1/1.33”, which if you understand basic fractions is MUCH larger than either the Pixel or the iPhone.

Yet in MANY camera conversations, the general agreement from numerous reviewers, the Pixel 5 was simply outclassed.

It didn’t matter that an iPhone 12 Pro Max or S21 Ultra were almost twice the price of the Pixel 5. The Google phone WASN’T EVEN WORTH COMPARING. Google’s formidable investment in software processing was deemed “not worth it”. Apple AND Samsung TOGETHER had ushered in the era of larger camera sensors!

(Of course, techies with any kind of longer term memory would probably point out that manufacturers like Huawei had been riding the large sensor train for a couple years already, but rarely received the kind of praise for photography improvements that Samsung and Apple are currently receiving. But I digress…)

Those reviewer assertions were rarely backed up with much photo evidence. Bigger sensors are newer. Newer is better. Google was simply too old to even be worth comparing. Even at a reduced price, a Pixel couldn’t possibly be “worth it”. Better luck next year Google!

Reviews get sloppy when we take this marketing stuff for granted.

It didn’t matter that the iPhone 12 Pro Max sensor is closer in size to the Pixel than the Galaxy. Reviewers conflated Apple and Samsung, stapling both to the same “larger sensor” idea. There’s a larger combined audience of Samsung and Apple owners than Google owners. The Pixel needed to be the “loser”. Reviewers didn’t need to show their work. They just regurgitated the specs.

The Verge went out of their way to highlight this in their Galaxy S21 Ultra review:

“Some of you might be surprised that I haven’t mentioned the Pixel 5 yet. Well, it has fallen behind. Both the S21 Ultra and the iPhone 12 Pro Max have switched to physically larger sensors and it has revealed the limits of computational photography.”

Notice that the author here didn’t mention the iPhone 12 Pro at $999, which uses the EXACT same sensor size as the Pixel 5. You might have missed that little omission.

Why detail that, when it might just upset iPhone owners? You NEED the Pixel to be the loser. Don’t have Apple and Samsung fans fight each OTHER! You can keep BOTH of those communities happy by picking on the Google nerds.

Just as a fun little exercise, let’s see how those differences play out.

These are obviously not the real life sizes (because phone sensors are silly tiny compared to standalone mirrorless cameras), but they are properly to scale. Here are the relative size differences between a Pixel 5, iPhone 12 Pro Max, and Galaxy S21 Ultra:

• Pixel 5 (4032×3024 @ 1.4μm) = 5.64mm x 4.23mm (7.05mm diagonal)
• iPhone 12 Pro Max (4032×3024 @ 1.7μm) 6.85mm x 5.14mm (8.56mm diagonal)
• Sony Xperia 1iii (4032×3024 @ 1.8μm) = 7.26mm x 5.44mm (9.07mm diagonal)
• S21 Ultra (12000×9000 @.8μm) = 9.6mm x 7.2mm (12mm diagonal)

The Galaxy truly does represent a noticeable difference in photo performance, low light, and depth of field. The The Note 20 Ultra and S21 Ultra main sensors are almost twice as large as the iPhone 12 Pro Max, and three times larger than the iPhone 12 Pro.

The practical surface area difference between a Pixel 5 and the most expensive iPhone is much closer. The 12 Pro Max enjoys around a 50% surface area advantage over the 12 Pro and Pixel 5. That advantage isn’t as stark as the comparison between a 12 Pro Max and the larger cameras found on Samsung, Xiaomi, and OnePlus mega-sensor phones.

There is NO parity between an iPhone 12 Pro Max and a Galaxy S21 Ultra for those elements of photography that are influenced by sensor size.

If you are discussing larger sensors, you’d most want to highlight image characteristics like low light performance and depth of field. The S21 Ultra is simply in a different league compared to the iPhone 12 Pro Max.

If you are discussing sensor technology and pixel binning, again, the S21 Ultra represents a significant difference in how that computational photography functions compared to the more traditional sensor design of the iPhone.

For both of the above points, and what MOST reviews completely missed, the most expensive iPhone of 2020 lived in a photo “tier” ahead of (but closer to) the premium-mid-ranged price of the Pixel 5 than it did the most expensive Samsung.

Reading the above Verge quote again, it rings… incomplete…

Why would reviewers disregard the Pixel 5 when it competes directly against the smaller iPhone 12 Pro? At the very least, shouldn’t the iPhone 12 Pro be included in the above statement? The 12 Pro was a significantly more expensive phone than the Pixel 5. The Verge asserts we’ve reached the limits of computational photography, so wouldn’t that also be true for the iPhone 12 Pro, the iPhone 12, and the iPhone 12 Mini?

Should we apply motive to that kind of review inconsistency? Why go SO far out of the way (in a Samsung review no less) to marginalize the Pixel?

Resolution is another area where it’s easy to casually spit out numbers, but still arrive at incorrect conclusions.

One of the major recent advances in phone photography, pixel binning is a complicated new technology to describe.

Instead of a traditional arrangement of red, blue, and green sub-pixels, pixel binning combines those sub-pixels into larger blocks. It is technically correct to say a camera has all the dots to be called a 48MP or 108MP camera, but that’s not really how these cameras are most likely to operate.

Push the shutter on a OnePlus 9 Pro (with a 48MP sensor) and the output resolution will look familiar. The OnePlus groups sub-pixels in four-block squares. Divide the total resolution by the blocks of sub-pixels, and we arrive at a 12MP final output resolution.

108MP on the Galaxy S21 Ultra sounds really impressive, but that sensor bins in groups of NINE sub-pixels. The final output resolution will be 12MP.

I’ve seen numerous complaints about the current XPERIA 1iii having ONLY a 12MP sensor, but if we refer to the above sensor size chart, the XPERIA main camera sensor is a tiny hair larger than the 12 Pro Max. It’s larger than the last generation of 12MP sensors. The iPhone 12 Pro Max, XPERIA 1iii, and Galaxy S21 (non Ultra) main camera sensors are all similar. There’s no fancy pixel binning, but the sensors are around a 50% surface area improvement over the previous generation of camera sensors.

We have to discuss cameras holistically.

There is no ONE simple metric which can predict better output.

Packing on more pixels, increasing sensor size, changing the subpixel arrangement. Those are all individual technological differences, but we can choose different COMBINATIONS of technologies to highlight different areas of photography.

Pixel binning sensors might be better at HDR processing, but they can often suffer slower and less accurate auto focus.

Bigger sensors are a boon in dark conditions, but if we don’t have control over our aperture, bright scenes can be a challenge to capture without blowing out highlights. This is problematic for video. The main way we control exposure on a phone is using a faster shutter speed, which can make video look REALLY choppy.

Moving the Goal Posts

There is no ONE way to grade a camera which will properly predict how all consumers might enjoy it.

The XPERIA 1iii is a fun example of this. Sony’s sensors are less exciting from an HDR standpoint. Why would Sony use those “REGULAR” sensors on their expensive phones?

Sony’s innovations in phone camera tech primarily highlight speed and auto-focus.

It’s hack reviewing to make the same tired arguments about XPERIAs being “hard to use” because the camera app is laid out like a traditional camera.

“Who has time to fiddle with manual modes???”

Yet, we rarely hear about the phenomenally fast burst photo rate and the insanely spooky-good auto-focus. When paired with the solid Auto mode on the Sony camera app, an XPERIA offers one of the fastest ways to capture a moment happening right in front of you.

Instead of using AI to paint in a bunch of saturated colors, Sony uses AI for Skynet-grade subject identification.

Take two EXPENSIVE phones: a Galaxy S21 Ultra and an XPERIA 1iii. What matters more to you?

• Better HDR processing and low light performance, but with clumsier auto focus.
• Better autofocus and faster burst rate, but photos in dark conditions will be grainier.

Separate the conversation from the YouTube SEO popularity of each brand, and I doubt you’d arrive at one consistent answer that all consumers would agree on.

People staging nicer portraits would probably gravitate towards the phone with a larger sensor. Parents trying to capture action shots of their kids would probably want the faster auto focus. Everyone will have their own needs to consider.

Each camera brings its own pros and cons.

The joy of that last sentence is digging through all the nooks and crannies to find what a camera’s strengths are. If you’re curious about tech, the discovery is the fun part. I find it frightfully dull to keep reducing camera conversations to the lowest common denominator auto use. You miss all the crazy new fun stuff.

Reviewers can still educate.

We go through a hype spiral every year. It’s marketing’s job to make bold claims about improvements. It’s a reviewer’s job to test those claims.

Consumers seem increasingly exhausted by this cycle.

They might see the BIG CLAIMS about amazing new performance, but if we continue to test at the lowest level use, we never get to examine or realize those claims. There is no single number score that can really rank cameras. There’s no simple auto mode analysis that will predict if a camera is the right fit for an individual’s needs.

I understand the desire for a simple chart of scores. We want to easily know if something is “gooder” or “badder”, but no tech works like that. There’s always nuance.

It’s my not-so-humble hypothesis that consumers are often underwhelmed with new tech because they often buy kit that doesn’t properly satisfy their needs. You’d be frustrated with your clothes if you constantly bought the wrong size. Tech feels a lot like that today.

There’s a little glimmer of magic when you really do find that solution that properly fits your needs. Finding that solution takes more energy, but I think people are genuinely rewarded when they take that little bit of extra time to learn about what they buy.

Hopefully, along the way, reviewers will find some of that fun too.