Niantic is expanding the developer platform to access its next-generation augmented reality tools using its Niantic Lightship platform. The platform will now be available to just about any third-party game developer.

Previously referred to as the Niantic Real World Platform, Niantic Lightship is offering broader access to the new Niantic Lightship Augmented Reality Developer Kit (ARDK), enabling developers to build their own immersive, imaginative, and unique AR applications.

Niantic Lightship encompasses the full Niantic platform, including the suite of tools and services that power Niantic games, from Ingress to Pokémon Go to the company’s partnership with Nintendo to bring the Pikmin universe to location-based and AR gaming. With Niantic Lightship, the company wants to set the industry standard for mapping and shared, planetary-scale AR experiences.

Developers can now sign up at Niantic.dev to join the private beta of the Niantic Lightship ARDK, as well as to learn more about Niantic Publishing.

“We are hoping it’s going to help creators and storytellers to build their own AR experiences on our platform,” said Kei Kawai, Niantic’s vice president of product management, in an interview with GamesBeat. “We always wanted to build a platform to enable people to build a immersive AR experiences. And we are finally going to get there. It’s been a long journey.”

The Niantic Lightship ARDK

Above: The Niantic Lightship is an AR platform.

Image Credit: Niantic

Niantic launched the Niantic Lightship ARDK as a private alpha earlier this year for hundreds of gaming and non-gaming developers. They have engaged with the company’s engineering, product development, and support teams to test the limits of what’s possible with the technology.

“We’ve brought some technologies to this SDK that will make that a lot easier for developers so they don’t have to solve those problems,” he said. “Also, 5G is here and we think it will help scale AR. We’ve added a lot of Niantic secret sauce on top of it.”

Niantic has incorporated that feedback to help shape the AR tools, features, documentation and core functionality of the Niantic Lightship platform.

As a result of this work, the ARDK private beta incorporates a host of advanced tools, including enhancements in three key areas. It has real-time mapping through advanced meshing, which combines smartphone camera technology with a neural network to map an environment in real time into a mesh of repeating tessellated triangles, resulting in a machine-readable representation of the physical world. In the Niantic Lightship ARDK, meshing makes “physics” possible for virtual objects.

“It’s like understanding your environments better in an augmented way,” he said. “Our whole mission is getting people out into the world to explore the world. So it’s really focused around natural things. So understanding where the sky is, where the water is, where the ground is, foliage is, so that your characters or your virtual content can then act naturally with it.”

In addition, semantic segmentation improvements in the beta distinguish between varied characteristics of a space — ground, sky, a building, etc. — so that virtual objects can look, feel and move in that space in realistic ways. Lightship can automatically segment different natural outdoor objects in a scene, enabling AR content to interact with specific surfaces.

More robust multiplayer functionality allows developers to benefit from colocalization, networking and synchronization improvements, enabling immersive multiplayer experiences where up to eight players can share the same AR experience in the same real world space at the same time. Peer-to-peer messaging and back-end server features are built in, so developers can focus on sculpting the shared experience players will have. While this kind of content anchoring is currently ephemeral, Niantic knows that long-term virtual content anchoring will enable developers to build future AR experiences that persist.

“The whole idea is we want to have shared multiplayer experiences,” Kawai said.

It’s not easy to do this. Doing a shared multiplayer experience is a tough computer vision problem. Getting the visuals right without time delays, or latency, is also a big challenge, Kawai said.

What makes the platform unique?

Above: Niantic’s ARDK lets AR objects interact naturally with the real world.

Image Credit: Niantic

Kawai said that meshing is a foundational technology in the AR space, which Niantic enables on more devices than anyone else by leveraging common smartphone cameras.

Niantic’s approach creates meshes using the RGB color sensors found in most cameras, rather than requiring LiDAR scanners only available on some high end devices. The meshing technology scans the environment around a player, creating in real time a 3D map that allows any Lightship platform experience to place virtual content in the right physical place, combining context from the real world environment with the developer’s environment. The history of Niantic’s approach to meshing goes back to the earliest days of the company and incorporates the work of acquisitions including Matrix Mill and 6D.ai.

The tech blends with phones that have LiDAR (laser-based sensors), including iPhone 12 Pro models, allowing developers using the Niantic Lightship ARDK to create dynamic maps for experiences without worrying about the specific devices, empowering the best experiences any hardware is capable of delivering.

What’s next?

Niantic said the future of AR can only be more fully realized once our devices have a 3D map of the world. Niantic is building the Virtual Positioning System (VPS) and advanced tools that allow AR experiences to leverage geolocation and persistent content anchoring with precision, while also supporting global scale.

“We’re hearing the excitement from developers who’ve been playing with features around real-time mapping or meshing technology,” he said. “It gives players the ability to doreal-time mapping of your environment with a smartphone camera. You can have a virtual ball that rolls on the real floor and bounces against a wall. It will bring that virtual content to life.”

He said Niantic will strive to make the AR technology work on as many devices as possible.

“We’re seeing billions of photos in AR in Pokemon Go, taken by and shared by people,” Kawai said. “I can’t speak much about the big titles, but the whole of our games will be running on the platform and they will have capabilities available to them to make something unique and different and beautiful and delightful.”