This week, Apple debuted a new iPad Pro. It has a little more power than the previous model, and a keyboard with a trackpad. Neat. But its most consequential upgrade is the one that will likely get the least use, at least on a tablet: a lidar scanner.
If you’ve heard of lidar it’s likely because of self-driving cars. It’s a useful technology in that context because it can build a 3D map of the sensor’s surroundings; it uses pulses of light to gauge distances and locations in a similar way to how radar uses radio signals. In an iPad Pro, that depth-sensing will be put in the service of augmented reality. But it’s not really about the iPad Pro. Apple put a lidar scanner in a tablet to prepare, almost certainly, for when it puts one in a pair of AR glasses.
Speculation is cheap. But at this point Apple’s AR ambitions are hardly speculative; a brisk stroll through the innards of the company’s iOS 13.1 and Xcode 11 last fall referenced a set of smart glasses. A patent the company was granted last summer for a headset-based “mixed reality system” specifically extols lidar as essential to the experience. In November, tech news site The Information reported that Apple executives had targeted a 2022 release of an AR headset, followed by AR glasses the following year.
“I fully expect each of Apple’s emerging AR technologies to be dry-runs for whatever future headset they’re building,” says developer Steven Troughton-Smith, one of the first to spot the headset breadcrumbs in iOS 13.1. “While they have interesting uses on iPhone and iPad, they’ll really start to make sense when paired with stereo head-mounted AR.”
Even Apple seems pressed to come up with a compelling argument for lidar in the iPad Pro. While its press release ticks off all the perks—speed, accuracy, motion capture—it mainly highlights its Measure app as a beneficiary. It has a Ruler View now. Other apps like Ikea Place and Hot Lava will use lidar to transform your rooms with Swiss commodity design and pulsing hot magma, respectively.
Yes, lidar is well-suited for that kind of work. “There are a lot of benefits for AR applications. You need to know where the physical objects are in order to place virtual objects on top,” says Gordon Wetzstein, who heads up Stanford University’s Computational Imaging Lab. “It helps you to understand the environment around you, and understand where’s the floor, where’s the table, where’s the chair, where do I put my Pokémon on this visual objects. That’s something that really requires precise depth definition from the mobile device, ideally with low power and low latency.”
But it’s even more appropriate when you leave your apartment and step out into the world, which introduces complications like walking around. “This isn’t your run of the mill ‘stand still and point your phone at something’ AR experience,” says Jessica Brillhart, director of the Mixed Reality Lab at the University of Southern California’s Institute for Creative Technologies. “Now you can walk and AR in a much more seamless way. Virtual objects will now be easily occluded—meaning assets will be much more baked into the world you’re moving through, plus they’ll continually map to your perspective and position within space relative to their placement.”
That becomes even more important when you offload AR sensors to eyeglasses rather than a tablet or phone. Rather than having to reference narrow slices of a virtual world through a smartphone screen, a lidar scanner will enable experiences that truly envelop you. Apple’s lidar component also doesn’t take up much space.
“The technology is ideally suited for all small scale devices, says Wetzstein. “It’s low power, it’s lightweight, it’s small scale, it gives you high-quality depth. You can probably combine a number of things together so they wouldn’t interfere with one another,” because lidar works by identifying individual points in space rather than an entire room all at once.
So … why is it on an iPad Pro again, the least mobile of Apple’s mobile computers? Think of it as a headset head start, for both the supply chain that provides the component and the developers who need time to figure out what to do with it. “iPad feels like an especially odd place for it, but it may just be that suppliers aren’t ready to meet iPhone demand yet,” says Troughton-Smith. “It also gives Apple a chance to get developers to build experiences now that they can eventually show off whenever the next iPhone is introduced.”
That last bit will be especially important. While Apple has spent the last several years pushing its ARKit framework—devoting big chunks of stage time to it at its annual Worldwide Developers Conference—the tech has yet to go mainstream outside of a few specific cases, most notably Pokémon Go. If and when Apple does finally push into a new product category, it needs fully baked experiences to show off along with it. Getting lidar in front of developers through the iPad Pro will help familiarize them now, helping to stave off a potential chicken-and-egg problem later.
“Immersive media has a content problem more-so than a hardware problem,” says Brillhart. “Most immersive technologies fail if engaging content is lacking, which is practically every piece of hardware right now. If I were Apple, I would be trying to seed this ecosystem now so that when I did release hardware, I would be avoiding a similar fate.”
Whatever form Apple’s head-worn AR takes, and when, remains unclear, especially given the disruptive long-term effects of Covid-19. But when it does come, you can expect lidar to play a critical role. The iPad Pro is its dress rehearsal.
More Great WIRED Stories
- How UFO sightings became an American obsession
- Deliver us, Lord, from the startup life
- Going the distance (and beyond) to catch marathon cheaters
- Plane contrails have a surprising effect on global warming
- Can you spot the idioms in these photographs?
- ? If AI’s so smart, why can’t it grasp cause and effect? Plus, get the latest artificial intelligence news
- ✨ Optimize your home life with our Gear team’s best picks, from robot vacuums to affordable mattresses to smart speakers