The Alpha Chronicles: the case of the disappearing car
2.10.2020
Translation: Veronica Bielawski
Click. I attempt to take a picture of a racecar speeding by. I end up with a serene landscape shot and not a car in sight, courtesy of the AI. Although it’s no work of art, the picture gives some insight into how a smartphone camera works.
Every now and then, I get my hands on software that is, according to the vendor, «almost finished». These are some of the best moments in my job. Unfinished software is the perfect way to get a good look behind the scenes. Unfortunately, most smartphone manufacturers seem to be embarrassed by their almost-ready-but-not-quite-there-yet software. Or maybe they think that no one will want to buy their phones once they find out that the software was once incomplete.
I really don’t get it. Because it’s a really interesting opportunity for tech nerds. It allows you to find out how a phone does certain things, what the AI is «thinking» and why a manufacturer has made a particular decision.
I really do relish alpha and beta software. Also, my dear manufacturers, I’m quite certain that my readers are intelligent enough to know that the final software they end up getting on their phone will be complete.
With that out of the way, let’s take a look at Oppo.
The Oppo Reno 4 Pro entered the market today. I’ve had it since last week and took it along to the 24-hour race at the Nürburgring. It was wet. And freaking awesome.
The case of the disappearing racecar
Night falls. The Oppo Reno 4 Pro is hailed as a beast of night photography. And night video recording. Time to put that claim to the test. I'm to the right of the finish line. I hear an engine racing in the distance, see the lights of the car. In any smartphone, there’s a delay of a few tenths of a second between the press of the shutter button and actual shot. So, I snap the picture while the racecar is in the leftmost third of the frame.
The result:
There’s a racecar in this picture. Technically speaking. But the AI ignored it. How’d that happen?
The shot I actually wanted would have looked something like this, ft. my absolutely glorious Photoshop skills:
But because the alpha software on my Oppo is not yet 100% compatible with the camera, the AI misunderstands the data it gets from the camera.
The anatomy of a smartphone snap
A photo taken with a smartphone is not a photo in the traditional sense. It’s a snapshot – an image consisting of several pictures put together by the AI. It usually combines quite a large number of shots, but for our example, I’m assuming it’s just three.
A photo from the user's perspective
I press the shutter button when the car is in the left third of the frame.
The Oppo actually takes the picture when the car is in the middle third.
And when the car is in the right third, it’s too late.
A photo from the smartphone's perspective
From the moment you launch the camera app, your smartphone is already calculating things. It looks for stationary points and decides what kind of scene it’s dealing with. Some possible scenes are «night», «blue sky» or «cat».
Then, the phone gets input from the user to take the picture.
Over the course of the next few tenths of a second, the smartphone takes as many pictures as it can.
Since a racing car is pretty dang fast, it may already have zoomed out of frame while the phone is still shooting.
And when the racecar starts a new lap, the phone’s AI is still hard at work. It analyses the gathered data – the three or more shots – and pieces something together. It then deletes the raw data. A single shot taken on the Oppo Reno 4 Pro is 10,905 kilobytes in size. That’s quite big. If there are three shots, that’s around 30 megabytes of raw data, give or take a few. With 12 shots, it would be 120 MB. Nobody needs that much data noise.
So, the AI does its best to decipher what the user actually wants to see in the picture. It recalls the initial settings, and so probably thinks you want a clean, undisturbed landscape. And the AI has enough material to work with. The three to twelve shots it gets are enough to piece together a completely car-free street.
Here’s what an early picture might look like:
And here’s what a later picture might be:
So, the AI takes the leftmost and rightmost thirds from the first shot and the middle third from the second shot. These are then merged, the colours optimised and voilà – you end up with a serene nighttime landscape. Basically, the AI’s thought process is: the car is an unwanted disturbance. It’s too fast and too small, so away with it. So, while the AI feels super proud of itself, the photographer of the Nürburgring is in for a laugh.
Further experimentation proved that the AI is perfectly capable of capturing the lights of the speeding car – just not when it’s right in the centre of the shot.
So there you have it. An update with improved camera software was released this morning.
Journalist. Author. Hacker. A storyteller searching for boundaries, secrets and taboos – putting the world to paper. Not because I can but because I can’t not.
Smartphone
Follow topics and stay updated on your areas of interest
6 comments
later