**Name String**

`XR_ANDROID_trackables_object`

**Extension Type**

Instance extension

**Registered Extension Number**

467

**Revision**

1

**Extension and Version Dependencies**

[`XR_ANDROID_trackables`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables)

**Last Modified Date**

2024-11-01

**IP Status**

No known IP claims.

**Contributors**

Diego Tipaldi, Google

David Joseph Tan, Google

Christopher Doer, Google

Spencer Quin, Google

Jared Finder, Google

Levana Chen, Google

Kenny Vercaemer, Google

## Overview

This extension enables physical object tracking. For example, keyboards, mice, and other objects in the environment.
| **Caution:**   
|
| **Permissions**   
|
| Android applications**must** have the`android.permission.SCENE_UNDERSTANDING_COARSE`permission listed in their manifest as this extension depends on[`XR_ANDROID_trackables`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables)and exposes the geometry of the environment. The`android.permission.SCENE_UNDERSTANDING_COARSE`permission is considered a dangerous permission.  
| (protection level: dangerous)

## Track objects

This extension adds`XR_TRACKABLE_TYPE_OBJECT_ANDROID`to[`XrTrackableTypeANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTypeANDROID).

The application**may** create an[`XrTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerANDROID)by calling[`xrCreateTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#xrCreateTrackableTrackerANDROID)and specifying`XR_TRACKABLE_TYPE_OBJECT_ANDROID`as the trackable type in[`XrTrackableTrackerCreateInfoANDROID::trackableType`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerCreateInfoANDROID)to track objects.

The[`XrTrackableObjectConfigurationANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectconfigurationandroid)structure is defined as:  

    typedef struct XrTrackableObjectConfigurationANDROID {
        XrStructureType                type;
        void*                          next;
        uint32_t                       labelCount;
        const XrObjectLabelANDROID*    activeLabels;
    } XrTrackableObjectConfigurationANDROID;

### Member Descriptions

- `type`is the[`XrStructureType`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrStructureType)of this structure.
- `next`is`NULL`or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
- `labelCount`is the count of`activeLabels`.
- `activeLabels`is a pointer to the array of[`XRObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid)indicating the active objects in tracking.

The application**may** set an additional configuration by adding a[`XrTrackableObjectConfigurationANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectconfigurationandroid)to the next chain of[`XrTrackableTrackerCreateInfoANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerCreateInfoANDROID). The output from[`xrGetAllTrackablesANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#xrGetAllTrackablesANDROID)will be filtered to match the`activeLabels`.

If the application does not set[`XrTrackableObjectConfigurationANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectconfigurationandroid), then all objects that the system has identified will be tracked.

### Valid usage (implicit)

- The[`XR_ANDROID_trackables_object`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables)extension**must** be enabled prior to using[`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid)
- `type`**must** be`XR_TYPE_TRACKABLE_OBJECT_ANDROID`
- `next`**must** be`NULL`or a valid pointer to the[next structure in a structure chain](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#fundamentals-valid-usage-for-structure-pointer-chains)
- `activeLabels`**must** be a pointer to an array of`labelCount`valid[`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid)values
- The`labelCount`parameter**must** be greater than`0`

The[`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid)enum is a label for a[`XrTrackableANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableANDROID)object.  

    typedef enum XrObjectLabelANDROID {
        XR_OBJECT_LABEL_UNKNOWN_ANDROID = 0,
        XR_OBJECT_LABEL_KEYBOARD_ANDROID = 1,
        XR_OBJECT_LABEL_MOUSE_ANDROID = 2,
        XR_OBJECT_LABEL_LAPTOP_ANDROID = 3,
        XR_OBJECT_LABEL_MAX_ENUM_ANDROID = 0x7FFFFFFF
    } XrObjectLabelANDROID;

## Get trackable object

The[`xrGetTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrgettrackableobjectandroid)function is defined as:  

    XrResult xrGetTrackableObjectANDROID(
        XrTrackableTrackerANDROID                   tracker,
        const XrTrackableGetInfoANDROID*            getInfo,
        XrTrackableObjectANDROID*                   objectOutput);

### Parameter Descriptions

- `tracker`is the[`XrTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerANDROID)to query.
- `getInfo`is the[`XrTrackableGetInfoANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableGetInfoANDROID)with the information used to get the trackable object.
- `objectOutput`is a pointer to the[`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid)structure in which the trackable object is returned.

`XR_ERROR_MISMATCHING_TRACKABLE_TYPE_ANDROID`will be returned if the trackable type of the[`XrTrackableANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableANDROID)is not`XR_TRACKABLE_TYPE_OBJECT_ANDROID`, or if the trackable type of the[`XrTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerANDROID)is not`XR_TRACKABLE_TYPE_OBJECT_ANDROID`.

### Valid usage (implicit)

- The[`XR_ANDROID_trackables_object`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables)extension**must** be enabled prior to calling[`xrGetTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrgettrackableobjectandroid)
- `tracker`**must** be a valid[`XrTrackableTrackerANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTrackerANDROID)handle
- `getInfo`**must** be a pointer to a valid[`XrTrackableGetInfoANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableGetInfoANDROID)structure
- `objectOutput`**must** be a pointer to an[`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid)structure

### Return Codes

[**Success**](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#fundamentals-successcodes)

- `XR_SUCCESS`
- `XR_SESSION_LOSS_PENDING`

[**Failure**](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#fundamentals-errorcodes)

- `XR_ERROR_FUNCTION_UNSUPPORTED`
- `XR_ERROR_VALIDATION_FAILURE`
- `XR_ERROR_RUNTIME_FAILURE`
- `XR_ERROR_HANDLE_INVALID`
- `XR_ERROR_INSTANCE_LOST`
- `XR_ERROR_SESSION_LOST`
- `XR_ERROR_SESSION_NOT_RUNNING`
- `XR_ERROR_TIME_INVALID`
- `XR_ERROR_MISMATCHING_TRACKABLE_TYPE_ANDROID`

The[`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid)structure is defined as:  

    typedef struct XrTrackableObjectANDROID {
        XrStructureType           type;
        void*                     next;
        XrTrackingStateANDROID    trackingState;
        XrPosef                   centerPose;
        XrExtent3DfEXT            extents;
        XrObjectLabelANDROID      objectLabel;
        XrTime                    lastUpdatedTime;
    } XrTrackableObjectANDROID;

### Member Descriptions

- `type`is the[`XrStructureType`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrStructureType)of this structure.
- `next`is`NULL`or a pointer to the next structure in a structure chain. No such structures are defined in core OpenXR or this extension.
- `trackingState`is the[`XrTrackingStateANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackingStateANDROID)of the object.
- `centerPose`is the[`XrPosef`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrPosef)of the object located in[`XrTrackableGetInfoANDROID::baseSpace`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableGetInfoANDROID).
- `extents`is the[`XrExtent3DfEXT`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrExtent3DfEXT)dimension of the object.
- `objectLabel`is the[`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid)that the runtime has determined for this object.
- `lastUpdatedTime`is the[`XrTime`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrTime)of the last update of the object.

### Valid usage (implicit)

- The[`XR_ANDROID_trackables_object`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables)extension**must** be enabled prior to using[`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid)
- `type`**must** be`XR_TYPE_TRACKABLE_OBJECT_ANDROID`
- `next`**must** be`NULL`or a valid pointer to the next structure in a structure chain
- `trackingState`**must** be a valid[`XrTrackingStateANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackingStateANDROID)value
- `objectLabel`**must** be a valid[`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid)value

## Example code for getting trackable objects

The following example code demonstrates how to get trackable objects.  

    XrSession session; // previously initialized

    // The function pointers are previously initialized using xrGetInstanceProcAddr.
    PFN_xrCreateTrackableTrackerANDROID xrCreateTrackableTrackerANDROID; // previously initialized
    PFN_xrGetAllTrackablesANDROID xrGetAllTrackablesANDROID; // previously initialized
    PFN_xrGetTrackableObjectANDROID xrGetTrackableObjectANDROID; // previously initialized
    PFN_xrDestroyTrackableTrackerANDROID xrDestroyTrackableTrackerANDROID; // previously initialized

    XrTime updateTime; // Time used for the current frame's simulation update.
    XrSpace appSpace; // Space created for XR_REFERENCE_SPACE_TYPE_LOCAL.

    XrTrackableTrackerCreateInfoANDROID
      createInfo{XR_TYPE_TRACKABLE_TRACKER_CREATE_INFO_ANDROID};
    createInfo.trackableType = XR_TRACKABLE_TYPE_OBJECT_ANDROID;
    XrTrackableTrackerANDROID objectTrackableTracker;
    XrResult result = xrCreateTrackableTrackerANDROID(
      session,
      &createInfo,
      &objectTrackableTracker);
    if (result != XR_SUCCESS) { /* Handle failures. */ }

    uint32_t trackableCountOutput = 0;
    std::vector<XrTrackableANDROID> allObjectTrackables;

    // Query the number of trackables available.
    result = xrGetAllTrackablesANDROID(
      objectTrackableTracker,
      0,
      &trackableCountOutput,
      nullptr
    );

    if (result == XR_SUCCESS) {
      allObjectTrackables.resize(trackableCountOutput, XR_NULL_TRACKABLE_ANDROID);

      // Fetch the actual trackable handles in the appropriately resized array.
      result = xrGetAllTrackablesANDROID(
        objectTrackableTracker,
        trackableCountOutput,
        &trackableCountOutput,
        allObjectTrackables.data());

      if (result == XR_SUCCESS) {
        for (XrTrackableANDROID trackable : allObjectTrackables) {

          // Object trackable query information
          XrTrackableGetInfoANDROID objectGetInfo;
          objectGetInfo.type = XR_TYPE_TRACKABLE_GET_INFO_ANDROID;
          objectGetInfo.next = nullptr;
          objectGetInfo.trackable = trackable;
          objectGetInfo.baseSpace = appSpace;
          objectGetInfo.time = updateTime;

          // Get the object trackable. Note that the tracker only returns object types.
          XrTrackableObjectANDROID object = { XR_TYPE_TRACKABLE_OBJECT_ANDROID };
          result = xrGetTrackableObjectANDROID(
            objectTrackableTracker,
            &objectGetInfo,
            &object
          );

          if (result == XR_SUCCESS) {
            /** Do Stuff with the object */
          }
        }
      }
    }

    // Release trackable tracker.
    result = xrDestroyTrackableTrackerANDROID(objectTrackableTracker);

**New Enum Constants**

[`XrStructureType`](https://registry.khronos.org/OpenXR/specs/1.1/html/xrspec.html#XrStructureType)enumeration is extended with:

- `XR_TYPE_TRACKABLE_OBJECT_ANDROID`
- `XR_TYPE_TRACKABLE_OBJECT_CONFIGURATION_ANDROID`

[`XrTrackableTypeANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables#XrTrackableTypeANDROID)enumeration is extended with:

- `XR_TRACKABLE_TYPE_OBJECT_ANDROID`

**New Enums**

- [`XrObjectLabelANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrobjectlabelandroid)

**New Structures**

- [`XrTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectandroid)
- [`XrTrackableObjectConfigurationANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrtrackableobjectconfigurationandroid)

**New Functions**

- [`xrGetTrackableObjectANDROID`](https://developer.android.com/develop/xr/openxr/extensions/XR_ANDROID_trackables_object#xrgettrackableobjectandroid)

**Issues**

**Version History**

- Revision 1, 2024-10-03 (Kenny Vercaemer)
  - Initial extension description.

*** ** * ** ***

OpenXR™ and the OpenXR logo are trademarks owned by The Khronos Group Inc. and are registered as a trademark in China, the European Union, Japan and the United Kingdom.