Get Started with building apps for spatial computing

Fundamentals

  • By default apps launch into shared space – kinda like multiple windows on a Mac. And people are connected to their environment with passthrough 
  • Volumes allow of bounded 3D Content in a space
  • Dedicated full space – only your app and content appears in an immersive view
  • You can choose to render in a PassThrough or Fully Immersive space
  • Interactions are via Eyes or Hands by defaults.  These interactions there are a bunch of gestures covered in another blog post of mine.
  • You can use ARKit skeletal hand tracking to create app specific hand features
  • Also supports Wireless devices, game controllers, keyboards, etc.
  • SharePlay and Group Activities allow for shared space
  • Check out designing and building spatial experience 
  • There is a lot of Privacy core principles
    • System curates data and interactions and delivers hoover effect or touch events
    • System asks the user for more information
  • Everything starts with Xcode – supports everything you need including Project Management support and Platform SDK
  • You will want to look at RealityKit metrics in Instruments
  • Reality Composer Pro supports Particle Emitters which were recently added to RealtyKit
    • You don’t have to build an app to test your content on the headset if you use Reality Composer Pro
  • Unity is also supported for people who are familiar with it

Where to Start

  • New App – You can use the new App Template for the platform where you can choice for Window or Volume, and then add an entry point for Immersive Space, if you choice Space it will add a second scene called Space to show you how to integrate.
  • There are code samples being published p
    • Destination Video show you how to build a shared immersive experience.  
    • Happy Bean is a game example in immersive space with custom hand gestures. 
    • Hello world shows how to position between different environments and modes
  • To start migrating an app, the iPad variant is preferred over iPhone variant.
    • They will retain light mode style, rotation is supported (Run your iPad and iPhone apps in Share Space) to learn more.
    • I f you add a new destination you can not target and recompile for xrOS – it will take on the new characteristics, spacing, sizing, etc. 

How to build

  • Start with Hello World app to see how it is done. Note basic windows have 3D content. Some views you end up with a volume, Another view provides a full immersion style.
    • This is a great app to see the spectrum of modalities
    • Windows serve as a starting point for your app.  They are written in SwiftUI and can include 2D and3D content.
      • Just add a windowGroup to your scene
      • Model3D(named: “” ) is very much like Image(named: “” )
      • Actions are familiar gesture recognizers and a few unique ones
  • You can add the gestures to Model3D to allow the user to manipulate a 3D object
  • Volume is a new style of window, designed for 3D content, really built for a shared space, so content must remain within the Volume
    • It is a .windowStyle(.volumetric) set with size dimensions.
    • Adopt RealityView to a scene, this allows for any number of entities to be managed by SwiftUI
    • You should learn more from various Spatial Computing with RealityKit sessions
  • Spaces is an element that hides all other apps, leaving only your app visible.  Your objects can interact with the surrounding.  Check out meet ARKit for Spatial Computing.  You use the Digital Grown to dial in the level of immersion
    • There are two styles of Immersion – .mixed (layers your content on top of passthrough) and .full (only shows your content.  There is also .progressive that allows to change the level of immersion
    • Good idea to add a button to allow a user to switch modes

Expand on Swift Macros

Why Macros

  • Lots of build in code expansion like Codable and property wrappers
  • Used to allow your own language features, to eliminate tedium and boiler plate.

Design Philosophy

  • Different from C Macros
  • Distinctive use sites
  • All start with either # or @ depending on type of macros
  • Complete, type-checked, validated – macros must be all of this
  • Inserted in a predictable additive ways.  It can not delete anything
  • Should not be  Magic – you can expand the macro in line or step into it in the debugger.  Even if it is a closed source library.

Translation Model

  • Basic concept,  extract the usage from code and sends to compiler plugin, in separate process and sandbox, and returns an expansion of code – which is added to your program – which is then compiled with your code.
  • All macros have to be declared so that the compiler can do the prior step – this is pacifically the API for the macro.  You can either import it via library/framework or write it inline in your code.
    • It must define it’s role

Macro Roles

  • Role is a set of rules for the macro
    • Where it can be used
    • What types of code it expands into
    • Where the expansions are inserted
  • They are responsible for achieving the goals.
Screenshot 2023-06-07 at 17.29.53 (2).png
  • There are two that create free standing macros and 5 that are attached macros
  • An expression is a piece of code that executes and presents a result.  
  • Detailed examples for each of these types of macros are gone thru in the session
  • If two different macros are attached to the same code it doesn’t matter about ordering, since they all are based on the original code.

Macro Implementation

  • Comes after an = sign, and it is always another macro
  • Usually an external macro implemented by a compiler plugin
  • #externalMacro specifies the plugin to call and use to expand the code.
  • You need to import SwiftSyntax – in your macro code to understand the Swift code syntax.  Check out the Write Swift Macros session or the SwiftSyntax developer documentation. 
  • Import SwiftSyntaxMacros – protocols and types
  • Import SwiftSyntaxBuilder – provides convince tools
  • You should include comprehensive error codes for when someone tries to use it incorrectly 
  • You can provide Fix buttons, highlights and other information in your error message when the macro is used incorrectly

Writing correct macros

  • Name collision – you can use the makeUniqueName() method on the macro expansion context.
  • ● Swift needs to use names outside of the macro, so you need to use makeUniqueName() to solve this.
Screenshot 2023-06-07 at 18.04.31 (2).png
  • Macros can only use the information that the compiler provides to them, so you can’t create a macro that inserts the current date and time (as an example)
  • Testing – it’s an ordinary swift module so you should write normal swift unit test for it

What’s New in App Clips

These are light versions of your app, allows users to try your app without installing

There are three

New size limit

  • Need to be small to ensure instant experience
  • You can now go up to 50MB size limit, 
  • If you want to use QR codes or NFC tags, then you still have to be limited to 16MB
  • If you target iOS 15 or earlier still limited to 10MB

Default app Clip links

  • Many require only a single experience , you must provide a website to host the recognized data.
  • Configure and link your App Clips from WWDC2020 for more information
  • Default links are Apple generated URLs so you can be invoked without any setup on your end.
  • You can get information from this with NSUserActivity

Invoke From your App

  • Now it can be invoked from your app, so you can support things like food pickup in a messaging app.
  • Use Link Presentation API – via LPMetaDataProvider request
  • Or use the default App Clip link 
  • This linking behavior does not require going thru Safari

Generalize APIs with parameter packs

This is an advance topic and builds on last year’s “Embrace Swift Generics” it is highly recommended that you review this before watching this session.

What parameter packs solve

  • This is about generics and variadics
  • Code is written in values and types.  You can abstract over types by having varying types, most generic code abstracts over types and values
  • If you want variable number of parameters .. you have Variadic parameters.  But they have limitations, you can’t know the number of parameters while preserving type information.  The only way to do this now is via overloading.
    • This pattern is limiting and adds redundancy.
  • This solves the overloading pattern problem

How to read parameter packs

  • In code a parameter pack can handle any qty or types, 
  • A type pack handles different types
  • A value pack handles different values
  • They are used together – they are positional, so at the type of position 0 is related to the value at position 0.  Think of it like a collection.
  • You need to iterate to process, but each element can have a different type.  So you would define a pack like <each  Type> – this is a type parameter type.  Use singular instead of plural. So you then use repeat Function<each Type> to see how this replacement works. You will get a comma separated list of types. They can only be used In things that naturally support comma separated lists, like function parameters, etc.
  • Using in a type pack in a function value, it becomes a value pack
  • You can add constraints to a parameter pack to enable conformance
  • You an force a minimum number of arguments (even though the default is zero or more arguments).

Using parameter packs

  • To use this code, you will want to use repetition patterns like repeat (each item).evaluate() if you want a list of all the values you can wrap in parenthesis like (repeat (each item).evaluate())
  • This should make your code simpler, and also allow you to write a lot less code
  • To exit an iteration – use throwing methods to allow you to break out of the iteration loop (if needed)

Discover String Catalogs

Localization is created with String Catalogs, OS supports 40 languages today

You used to have to maintain Strings and StringDict in your project.  In Xcode 15 you will have String catalogs to simplify the process

Will update my own app to this quickly.  Complex editing is enabled to make it easier to convert strings, including ability to change strings by device.

Extract

  • Localizable strings are simply text presented to user at runtime.
  • Key (required) – usually the sting itself
  • Default value – usually defaults back to Key
  • String Comments – to provide translator context about where and how it is used
  • Table corresponding to one ore more files where translations are stored. bY default this is Localizable table.
  • String Catalog contains the entire string table in a single file.  You can create multiple String Catalogs
  • Xcode will create the catalog and try to keep it in sync.
  • You should define you localizable strings as such with LocalizedStringResource
  • Change build setting – Use Compiler to Extract Swift String to make sure that Xcode will extract for your, you can also define your own localized custom macros
  • In Interface Builder strings are automatically defined as localizable.
  • Add a info.plist to the catalog to capture values in plist files 
  • Every time you build Xcode will export to the catalog – code is source of truth and the catalog will reflect state of the localized value. It will be considered stale if removed from code an you have a translation defined.

Edit

  • The editor allows for first class support of state and progress of translation.
    • New – untranslated
    • Stale – No longer seen in the code
    • Needs Review – may require change 
    • Translated – Green check mark – no action needed
  • Pluralization is a challenge
    • You need to change the grammar in English 
    • But in other languages it may be more complex
    • You used to need a StringDict file.  New feature has built in value variations via context menu
    • Complex examples like you have X items in X packages – which may change if you have 6 items in 1 package 
    • You define substitutions for items and packages in the prior statement
  • You can define manual strings that are defined in code or a database

Export

  • To work with translators – you can export localization catalog for each language so you can send to translation team.
    • The xliff file is standard format for storing translations. 
    • There are some changes based on catalog instead of stringDict format, this may require automation tools to change
  • Change make sure export works correctly set “Localization Prefers String Catalogs” in build settings to “Yes”
  • Import will just add it back in correctly

Build

  • During Build catalogs are json files under the hood and easily diffable in source code
  • You can back deploy to any OS target
  • Source strings are not stored in final build

Migrate 

  • To migrate your existing project, pick files and targets to migrate
  • Right click and choose Migrate to String Catalog
  • You must use swift tools version 5.9 or higher

Develop your first Immersive App

This session will probably require tools not yet available… will have to come back to it once the SDK and simulator becomes available

Create an Xcode project

  • You carte an xrOS assistant – but this is not yet available.. will just watch this section until I can really follow along.
  • Windows are 2D and can be resized, shown along side other apps
  • Volumes are designed for 3D content sizable in all three dimensions by the app, not the user
  • Immersive Space, will allow you to start a starting point for your app, this is a Full space, hiding other app and enable hand tracking, you can also decide how much immersion the user has “Go beyond the Window with SwiftUI” should be checked out
  • Recommend always start in a window and provide clear entry and exit points for immersion
  • most of the code is in ContentView in the sample project

Simulator

  • Show you a launcher view, click simulates tap, click and hold simulates pinch
  • There are buttons on the bottom right to control the device
  • There are multiple scenes to simulate location and lighting environments,

Xcode Previews

  • When editing source file, the preview provide will open the canvas with a simulated device view to navigate around and see impact of your code changes 
  • There is an object mode to see content that extends beyond your app’s view.

Reality Composer Pro

  • Creates swift packages for your content 
  • Content is organized into scenes
  • Scene type for immersive space, uses inferred position of your feet, as the center of the coordinate system.
  • You can get access to additional data like hand position.  Users will be prompted to provide access to privacy data

Create an immersive scene

  • You can use drag and drop to import and position usdz files.
  • Double clicking on an object in the scene hierarchy will make it front and center of the scene editor window
  • Immersive spaces are placed at a fixed place based on when it is launched.. you move around in the space to look around

Taret gestures to entities

  • You can add a simple TapGesture().targetedToAnyEntity().onEnded { //command to run }
    • The targetedToAnyEntity  tells you which entity was tapped
  • You must have both InputTargetComponent and CollisionComponent for Tap gestures to work

Design widgets for the Smart Stack on Apple Watch

Layouts

  • There are six design layouts for consistency
  • This will use consistent font sizes and weights
  • Only show what is necessary, 10 seconds max
  • Look at the design resources page
  • Combo widget is by default and can be used for circle app launchers

Color and Iconography

  • To be recognizable and distinctive
  • By default you have dark material with white text
  • Try a SFSymbol

Sessions

  • An active state in an app with a clear start and end
  • Focus on helpful content to lead into or come after a session, like music to play

Relevancy 

  • There are moments to consider, time, location, start or end a workout
  • This will prioritize when a stack item floats to the top

Build Widgets for Smart Stack on Apple Watch – Code alone session

Design spatial SharePlay experiences

You can add in shared context with Spatial Persona’s so you can communicate effortlessly

Set the Scene

  • Understand the type of experience you are trying to recreate…this will impact content placement and interactivity.
  • You can share up to 1 window and 1 immersive space at a time, so you can have a dedicated shared window, or a dedicate room (they can’t access other apps during this)
  • Using window shared activity let’s people use their other apps and drag and drop things across from private and public windows

Start SharePlay

  • You can create entry points to allow for people to join.  Adding sharePlay means you can have more control, etc.
  • Start activities from those who have the right permissions, and provide information to the participants  who are trying to join, like “start tv”

Arrange participants

  • This is based on where the app is positioned, and one of three arrangements
    • Side by Side (think TV screen or whiteboard)
    • Surround (think table top game)
    • Conversational (think front and center or ambient)
    • Your app can support up to 4 other people.. so you can have between 2-5 participants in a shared space.

Share context and UI

  • You get a common coordinate system and space .. this is shared context. The system manages this by itself, so the synchronization needs to be synchronized by the app to make sure people can follow.  You should encourage participants to follow the first person who changes context.  Shared space by default uses share audio context.
  • Large physical motion and audio cues can help share context
  • Make sure that permissions or turn taking is not imposed to make the experience is the most natural.
  • Allow for differences that don’t affect others, like volume per person and subtitles by person
  • Active tools should be personalized too.

Enter a Full Space

  • When is it ideal when a person can go to a operate context (or full space)
  • Think of this as break out rooms designed for a specific person to view.
  • If a person steps out (like pressing digital crown and enabling pass thru) make sure they can still see the shared window

Design for spatial user interfaces

UI Foundations

  • App icons
    • Should be round and flat, with 3D effect to expand the layers.  So you should use layers to create parallax affect on other platforms (up to three layers).  1024×1024 layers will be cropped by a circle to create the 3D layers – Be careful with layers.
  • Materials
    • Need to adapt to lighting conditions and surrounding 
    • New language uses the glass material – this allows for ability to see what is behind your window.  
    • Avoid solid colors (will have to really focus on UI for my app)
    • There are system defined vibrancy will cover below.
    • Don’t stack lighter material on top of each other
  • Typography
    • All font styles use semantic names – check out “Principles of spatial design”
    • Weight has been changed on visionPro to make it easier to see
    • Two new style son vision Proc – Extra Large Title 1
  • Vibrancy
    • Key to maintain legibility across system
    • This updates in real time since the background can constantly change 
    • Use Primary for standard
    • Use secondary for description
  • Colors
    • Consider using white text or symbols
    • If you need color, use it in a background layer or the entire button

Layout

  • Ergonomics
    • This is key, content should be placed intentionally, I.e. easier to go left or right, than up or down
    • Go wider aspect ratio, instead of taller
    • Prioritize key information in the center
  • Sizing
    • Must be a target of 60pts of space, standard button is 44pts with 8 pts of space around on all sides
    • For smaller items (disclosure button) like 28 pts ,but it still has 60 pts around it
  • Focus feedback
    • System components provide a subtle highlight (hover effect) – this gives people confidence they are focusing on the right element
    • Make sure you have a small amount of space (at least 4 pts) to make sure that hove can occur
    • Ensure nested elements have relative corners – remember – inner corner radius + padding = outer corner radius 
    • Make sure to use continuous corners

From screen to spatial

  • Window
    • Use glass material instead of opaque material
  • Tab Bar
    • This will be moved to the left side of the window – vertically
    • Avoid having more than six items
  • Side Bar
    • This will automatically expand to labels if someone looks for a while
  • Ornaments
    • These are placed at the bottom slightly in front of the window, great for tool bars
    • Also adds depth to an app.  Use borderless buttons to keep it clean
    • Overlap by 20pts to make the feel like they are integrated with the main window
    • It is only recommended to disappear when focusing on a single piece of content, like watching a movie
  • Menus
    • Can expand outside of the window and they are centered by default
    • Change the button that selects them, to selected state
    • Avoid using buttons with white backgrounds unless they are selected
  • Popovers
  • Sheets
    • Modal views in the center of the app, pushing the parent view back 
    • Consider using push navigation for nested view
    • Always place close buttons on the top left corner

Design for Spatial Input

Focus is on eye’s and hands – this is personal and unique to the user.

Eyes

  • This is primary targeting method, all inputs target where you look
  • To make this we need to
    • Comfortable
      • Your canvas may be infinite, but you should design for center of field of view to minimize head and neck movements
      • Depth should be considered too, changing focus depth can cause eye strain, this is the Z position, you can move things backwards so that current action is still in the same positioning as before
    • Ease to use
      • Focus on shapes that guide to the middle of an object, so use round shapes, not sharp edges. 
      • Keep shapes flat without shadows. (Need to adjust my own items for this)
      • Right size is at least 60pt, (which includes spacing between items).  Use standard components as much as you can.  Check out Design for spatial user interfaces.
      • Scaling (larger as it go away, and smaller as it get’s closer to keep the target the same) this is default dynamic scale behavior.  Fixed scale changes target area.  Used dynamic scale as much as possible.
      • Keep UI oriented at the user to minimize eye strain – check out “Principles of spatial design”
    • Responsive
      • All interactive elements should be highlighted.  Hover effect is subtle.
      • You should add this for any custom elements
    • Intentional
      • You can use intention (long hoover) to provide tooltips or tab bar expansion
      • Focus on microphone gliph will enable voice search mode.
      • Can also provide information for assistant technology.

Hands

  • Pinch is select, pinch and drag is scroll, and pinch with two hands enable zoom and rotate
  • The above slide is the one we’ve all been looking for
  • Use these same patterns in your own app, to make it easier for users to get up to speed.
  • To create a custom gesture you should:
    • Easy to explain and perform
    • Avoid gesture conflicts
    • Comfortable and reliable
    • Accessible to everyone
    • Look at “Create Accessible spatial experiences”
    • Consider fallback
  • Using eye direction combined with hand gesture shows intent, I.e. picture zoom
  • Direct Touch – reach out and user your finger tips
    • You can use a virtual keyboard
    • Consider that direct interaction can cause fatigue- holding your hand in the air for a long time 
    • Lack of tactile response should be considered
      • Include things like UI changes and sound feedback
    • Check out Explore Immersive Sound design