The recently-released iOS 13 brings with it a host of advanced capabilities, many of which are specially optimized for its newer A12 and A13 lines of chips. Advances in iOS 13 and hardware advances combine to create a platform that makes building augmented reality applications extremely straightforward.

What is Augmented Reality?

Augmented reality combines real-world and virtual-world elements into a single, (hopefully) seamless experience. The augmented reality app that many of us would recognize is Pokemon Go, which allows players to look for Pokemon in the “real world.” More sophisticated games and applications integrate into your video feed, literally overlaying the virtual world on top of the physical world captured by your camera. For instance, in Void Vector AR, you dodge enemy attacks by moving around physically in the real world.

An Integrated System

While augmented reality (AR) has been enabled for quite some time in iOS, iOS 13 has taken it a step further by providing developers more robust and straightforward tools for building the apps. It has introduced two technologies—ARKit and RealityKit—which facilitate integrating augmented reality into your application. Previously, to achieve augmented reality, the user had to tediously combine the elements of video feeds and 3D animation in an error-prone process.

The iOS 13 system combines:

• World Tracking: iOS 13 senses the movements of your device and uses them to help track your position in space. It can use its sensors to know how the device is oriented and how far it has traveled from where the session first began. Additionally, it keeps track of a variety of real and virtual objects in 3D space and knows where they need to be relative to the iPad.
• Video Feed: iOS 13 creates a very simple mechanism to read video feeds and overlay additional data.
• Object Detection: iOS 13 can detect basic objects in the camera’s field of view, and estimate distances to those objects.
• Body Tracking: iOS 13 can detect human bodies as well as their poses and can mirror those poses onto three-dimensional avatars. With just a few lines of code, you can get a 3D model to near-exactly mimic the actions of a person detected in your video feed.
• Occlusion Tracking: iOS 13 can detect people and their position in front of or behind objects, to make a more realistic combination of virtual and real-world entities.
• Image Tracking: iOS 13 can detect predefined images in order to attach additional virtual content to them. For instance, you could take an image of a painting and add a 3D version that will pop out when the camera sees that image.

These systems are integrated into a single framework to make it extremely simple for developers to access, modify, and display virtual content within a video feed.

Sessions, Scenes, Anchors, and Entities

Augmented reality in iOS 13 is oriented around sessions. An AR session manages your occupation of the real and virtual world. It uses the device’s motion-sensing hardware and camera to coordinate the position of the device in both the virtual and real worlds. Additionally, the AR scene is the virtual representation of the world. It includes both the detected physical world content and virtual world content. The AR scene is the core interface between the programmer and the world. The programmer adds objects to the scene and the session manages the spatial location of the user in the scene.

Within this world, the scene keeps track of all objects’ locations through anchors. An anchor is a point in the real world in which something is attached—either physically detected or virtually added. An Entity is a piece of virtual content, such as a 3D model, which can be rendered onto the video. The magic happens when an entity gets tied to an anchor, which causes the 3D model to be rendered in the right place in the 3D environment. You can either load entities from 3D model files, or you can build them from meshes. Apple even created an application called Reality Composer which allows for easy creation of 3D content. iOS 13’s AR system allows for 3D content to factor in ambient lighting so that lighting changes in the environment affect the shading of your 3D content.

By default, wherever you are when the session starts is the origin of your coordinate system. The coordinates are marked out such that each unit along each axis is approximately 1 meter long. The y-axis is aligned with gravity, such that movement up and down gets translated into movement along the y-axis. The x and z axes are based on the initial orientation of the device when the session starts, where the z-axis is forward and back from your initial position and the x-axis is left and right. However, once the session starts, the axes and origin of the coordinate system are fixed no matter how you move your device.

All of these pieces are conveniently wrapped together in a single module, the AR View, which comes with a session and a scene already ready to go.

A Simple Example

Creating AR worlds with iOS 13’s tools is very straightforward. Below is a Swift-language implementation of an AR application that simply presents a 1m radius sphere four meters in front of you. It will stay in its original world location even as you move about the room. This is the entire code for the entire application:

import UIKit
import RealityKit
import ARKit

@UIApplicationMain

class AppDelegate: UIResponder, UIApplicationDelegate {

    var window: UIWindow?

    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {

        // Create an ARView
       let arView = ARView(frame: .zero)

        // Grab the session and scene for ease of access
        let session = arView.session
        let scene = arView.scene

        // Create a 3D model of a red sphere
        let sphere = ModelEntity(mesh: MeshResource.generateSphere(radius: 1), materials: [SimpleMaterial(color: .red, roughness: 0.5, isMetallic: true)])

        // Create an attachment point in reality for the sphere,
        // level with the device four meters in front.
        let sphereAnchor = AnchorEntity(world: [0, 0, -4])

        // Add the sphere to the attachment point
        sphereAnchor.addChild(sphere)

        // Add the attachment point to the world
        scene.addAnchor(sphereAnchor)

        // Turn on the AR session
        session.run(ARWorldTrackingConfiguration(), options: [])

        // Make the AR view the main app screen
        let vc = UIViewController()
        vc.view = arView

        window = UIWindow(frame: UIScreen.main.bounds)
        window?.rootViewController = vc
        window?.makeKeyAndVisible()

        return true

    }

}

As you can see, with a handful of lines of code, you can easily integrate the virtual and real worlds in iOS 13.

Note that if you try to run this code directly, you will need to add an NSCameraUsageDescription to your info.plist file and remove the entirety of the UIApplicationSceneManifest and UIMainStoryboardFile keys from the info.plist file.

Limitations

While iOS 13 has some amazing features, it also has some significant limitations. The biggest one is that the tracking between the real world and the virtual world does not extend to GPS.

That is, you can see the world relative to your device, but it does not track the world relative to the world at large. While you can coordinate with the larger world through GPS, it takes quite a bit of extra work (though some third party systems are being developed to make this much easier).

Additionally, the 3D body tracking, while interesting, is not exact. It is more of a “demo” feature than a useful addition. It works really well for the “cool” factor, but not so well if you try to use it for real-world applications.

In all, however, the limitations are not such serious limits, compared to what iOS 13’s augmented reality system allows you to build quickly and easily.

Also by Jonathan Bartlett on what you can do with new tools:

You can build your own chatbot. New tools have made it comparatively easy

and

AI computer chips made simple: The artificial intelligence chips that run your computer are not especially difficult to understand