As part of my AR Mobile Robotics project I need to import 3D assets like DAE, OBJ, and STL files into RealityKit using native Swift code. While Apple’s framework natively favors the USDZ format, loading these common engineering formats at runtime is not directly supported—and existing third-party solutions didn’t quite meet the requirements for complex robotics visualization
So, I did what any good engineer would, and created my own; here presented as a pair of utilities that you can add to your XCode project using the Swift Package Manager (SwiftPM or SPM). I’ll cut right to the chase in the next section, and explain how you can add these utilities to your own project to fetch DAE, OBJ, or STL files from a URL and convert them into a ModelEntity to display in an augmented reality (AR) or spatial computing scene. More advanced users, and those who want to contribute to the open source project, can read on to learn about the architecture of the packages, and some of the features that are still to be implemented.
The Short Version
If you found this post via Google Search, you likely just want to know, what package should I download, and how do I run it? in that case, you’re in luck, with DAE, OBJ, or STL files, you can quickly generate RealityKit entities by simply:
- Adding a package via SwiftPM.
- Calling a one-line extension on
ModelEntity.
OBJ and STL file support are provided via a set of extensions I created for Apple’s ModelIO framework, and are contained in the following GitHub repository:
DAE files are, surprisingly, not supported by the ModelIO framework, though they were, at one time, partially supported by SceneKit. To use these, I needed to create a custom XML parser, which is found in this repository:
Installation
The packages may be installed using Swift Package Manager, in XCode, as follows:
- From the
Filemenu, selectAdd Package Dependencies. - In the search bar in the upper right, enter
https://github.com/radcli14/DAE-to-RealityKitorhttps://github.com/radcli14/ModelIO-to-RealityKit, depending on which format you need to support. - Make sure your project is selected in the
Add to Projectline, then click theAdd Packagebutton in the lower right. - Make sure your target is selected in the
Add to Targetline, then click theAdd Packagebutton again.
Usage
Most of the time, you will asynchronously load an entity using the static extension methods, for example, if you have a URL for a DAE file location:
let entity = await ModelEntity.fromDAEAsset(url: URL)Or, if you have a URL for a STL or OBJ file:
let entity = await ModelEntity.fromMDLAsset(url: URL)An example of a minimal SwiftUI app with a RealityView is shown below, using the ModelIO-to-RealityKit package to import a OBJ file. You could, of course, generate a similar app with DAE-to-RealityKit.
import SwiftUI
import RealityKit
import ModelIO_to_RealityKit
struct ContentView: View {
var body: some View {
NavigationStack {
RealityView { content in
if let url = Bundle.main.url(forResource: "shiny", withExtension: "obj"),
let entity = await ModelEntity.fromMDLAsset(url: url) {
content.add(entity)
}
}
.realityViewCameraControls(.orbit)
.navigationTitle("Model3DLoader")
}
}
}The code above was used to render the Blender mascot, Suzanne, with a shiny teal-colored material, from an OBJ-formatted model file, as shown at the top of this post.
A More Detailed Introduction
Apple has adopted the Unified Scene Description (USD, or USDZ when zipped) standard for importing 3D objects into RealityKit augmented reality (AR) scenes. I think this is a good design choice by Apple, it’s a high-quality modern format, and it’s best to stick to a single official format for developer consistency. Plus, the USD convention is very alive, with recent developments being its adoption into the NVIDIA Newton simulation framework, and its continued usage by Pixar.
That said, in my own project, AR Mobile Robotics (ARMOR), which includes a Unified Robot Description Format (URDF) file loader, and MuJoCo simulation solver, I need to support a variety of other 3D formats. URDF is a kinematics modeling standard, and is used in the Robot Operating System (ROS). It is a fairly simple XML format, which holds references to 3D objects via their URL. What this means is, supposing that an industrial supplier provides you the URDF of their robot, your software (in my case, the ARMOR app) needs to be able to not just interpret the XML of that file, but also whatever 3D objects files it references.
The STL, OBJ, and DAE formats continue to be de facto standards in the robotics simulation world in large part because they’ve been around a while. Their specifications are quite stable, and there is familiarity across a broad spectrum of users. They are also fairly simple, with examples of the first few lines of an OBJ and DAE file shown above. STL is somewhat similar to OBJ, so I’ve omitted it. These can also be compressed into binary formats, but this text-based format lends well to version control systems like Git. All of which is to say, while not necessarily the first choice of professional animators, I expect that DAE, OBJ, and STL will continue to be common engineering formats, and supporting them in an iOS app is a matter of learning to interpret their modeling language.
The Long Version (How it Works)
I created these package repositories, of course, for an immediate need that I faced in my own project. However, I also hope for them to be a valuable resource to others with a similar need, and as a way to give back to the open source community. In the case of the latter, I also invite others to contribute to or critique the work.
These packages work backward from the constructor for a ModelEntity where we provide the mesh and materials arguments. A mesh defines the physical locations and connectivity between vertices that define the object shape, and optionally its normals and texture coordinate (UV) at those vertices. The materials define color and other properties associated with the visual appearance of the faces of the mesh. If we can extract this same data from a DAE, OBJ, or STL file, then we can produce a RealityKit Entity.
The mesh argument is in the form of a RealityKit MeshResource. This type holds the geometric representation of the mesh in an array of MeshDescriptor. The material array can be formed from any of several different RealityKit Material types, but for our case, the PhysicallyBasedMaterial is best.
Unpacking the ModelIO (OBJ or STL) File
The starting point for the OBJ or STL file import is the MDLAsset type from ModelIO, which can be initialized from a URL of a 3D model file. Geometry for the MeshDescriptor can be unpacked from MDLMesh and MDLSubmesh children of the MDLAsset. The former contains vertex data, meaning the positions, normals, and texture (i.e. UV) coordinates of each vertex, in a buffer. The latter contains mesh primitives, meaning indices associated with each face of the mesh.
The MDLSubmesh also holds material data, meaning color or image textures for that section of the mesh (for many objects, there is only a single submesh). This data can be unpacked from the material variable, which itself contains various MDLMaterialPropertyinstances. Each of these can be converted to a form required by similar properties of the RealityKit PhysicallyBasedMaterial, which is alternately referred to as a Physically Based Rendering (PBR() material.
Unpacking the DAE File
There is a similar workflow for parsing DAE files, however, it is not a format supported by ModelIO. For this reason, I created a custom routine for unpacking the DAE files. I haven’t generated a flow diagram, but it has a similar structure, in brief summary, “find the fields in the raw XML that correspond to mesh and material data, reformat these to be compatible with the RealityKit constructors, then build a new constructor for a ModelEntity that takes these.”
My first attempt at loading entities from DAE involved creating a set of SceneKit extensions, using the SceneKit DAE loader as an intermediate stage to obtaining the RealityKit entity. This worked in the case that the DAE files are provided in the bundle at compile-time, which XCode automatically converts to scene files. However, that does not work for DAE files that are outside of the bundle and loaded at runtime. Therefore, I ended up creating a Collada data structure as a more general-purpose DAE parser.
The directories are structured as follows:
- RealityKit:: Extensions on
ModelEntitythat allow loading from DAE-formatted XML, as in the quickstart. Note that these are the only methods that are currently markedpublic. - Collada: Data structure derived from the COLLADA 1.4.1 Schema, parsed from XML using the XMLCoder package. This is loaded directly from the raw XML.
- SceneKit: Extensions on SceneKit objects to provide data that can aid in conversion to RealityKit. This section is probably obsolete for the reason I noted above, but is retained for the time being as a reference to the data structures.
Future Work
The ModelIO-to-RealityKit and DAE-to-RealityKit packages are robust for loading model meshes and materials from DAE, OBJ, or STL files. These will continue to improve with added testing, error handling, etc. What they are not is a “complete” representation of any one of these file specifications. In particular, I have made no effort thus far to parse animation, rigging, lighting, or other elements that are common to 3D models, but not necessary in my application.
As far as the materials definition itself, I suspect there is still room for improvement in the mapping between definitions created in Blender (or other 3D software), and what gets constructed into the PBR material for a RealityKit entity. Those who are familiar with PBR know that the most frequently used properties are texture files, or single colors, for diffuse color, normals, and roughness. Diffuse color is of course straightforward, but normals have different conventions, such as tangent or object space, or the positive/negative sign on each color channel. Roughness is easy to define, as it is grayscale-only, but I’ve observed that OBJ files may assume a different equation when associating roughness with the specular exponent term. These get down into the weeds of rendering math, but I expect some work may be done to improve on the visual consistency across different model export formats.
In short, my RealityKit loader scripts are be in good working condition for you to go ahead and use them in your own apps, but I also hope that you will contribute and improve on them to support the open source community!
More RealityKit Posts
If you enjoyed this post, I suggest you check out a few of my others related to RealityKit and 3D modeling in general: