Thanks for tuning in to Google I/O! Watch AR content on-demand.

Frame


          public     class
          Frame

Captures the state and changes to the AR system from a call to Session.update().

Public Methods

`Image`	`acquireCameraImage()` Attempts to acquire an image from the camera that corresponds to the current frame.
`Image`	`acquireDepthImage()` This method is deprecated. Please use `acquireDepthImage16Bits()` instead, which expands the depth range from 8191mm to 65535mm. This deprecated version may be slower than `acquireDepthImage16Bits()` due to the clearing of the top 3 bits per pixel.
`Image`	`acquireDepthImage16Bits()` Attempts to acquire a depth Android Image object that corresponds to the current frame.
`PointCloud`	`acquirePointCloud()` Acquires the current set of estimated 3d points attached to real-world geometry.
`Image`	`acquireRawDepthConfidenceImage()` Attempts to acquire the confidence Android Image object corresponding to the raw depth image of the current frame.
`Image`	`acquireRawDepthImage()` This method is deprecated. Please use `acquireRawDepthImage16Bits()` instead, which expands the depth range from 8191mm to 65535mm. This deprecated version may be slower than `acquireRawDepthImage16Bits()` due to the clearing of the top 3 bits per pixel.
`Image`	`acquireRawDepthImage16Bits()` Attempts to acquire a "raw", mostly unfiltered, depth Android Image object that corresponds to the current frame.
`Image`	`acquireSemanticConfidenceImage()` Attempts to acquire the semantic confidence image corresponding to the current frame.
`Image`	`acquireSemanticImage()` Attempts to acquire the semantic image corresponding to the current frame.
`long`	`getAndroidCameraTimestamp()` Returns the (Android Camera timestamp) of the image.
`Pose`	`getAndroidSensorPose()` Returns the pose of the Android Sensor Coordinate System in the world coordinate space for this frame.
`Camera`	`getCamera()` Returns the `Camera` object for the session.
`int`	`getCameraTextureName()` Returns the OpenGL ES camera texture name (id) associated with this frame.
`HardwareBuffer`	`getHardwareBuffer()` Gets the HardwareBuffer for this frame.
`ImageMetadata`	`getImageMetadata()` Returns the camera metadata for the current camera image, if available.
`LightEstimate`	`getLightEstimate()` Returns the current ambient light estimate, if light estimation was enabled.
`float`	`getSemanticLabelFraction(SemanticLabel queryLabel)` Retrieves the percentage of pixels in the most recent semantics image that are `queryLabel`.
`long`	`getTimestamp()` Returns the timestamp in nanoseconds when this image was captured.
`Collection<Anchor>`	`getUpdatedAnchors()` Returns the anchors that were changed by the `Session.update()` that returned this Frame.
`Collection<TrackData>`	`getUpdatedTrackData(UUID trackUuid)` Retrieve all track data that was written to the specified track during the current frame.
`<T extends Trackable> Collection<T>`	`getUpdatedTrackables(Class<T> filterType)` Returns the trackables of a particular type that were changed by the `Session.update()` that returned this Frame.
`boolean`	`hasDisplayGeometryChanged()` Checks if the display rotation or viewport geometry changed since the previous `Frame`.
`List<HitResult>`	`hitTest(MotionEvent motionEvent)` Similar to `hitTest(float, float)`, but will take values from Android `MotionEvent`.
`List<HitResult>`	`hitTest(float[] origin3, int originOffset, float[] direction3, int directionOffset)` Similar to `hitTest(float, float)`, but takes an arbitrary ray in world space coordinates instead of a screen-space point.
`List<HitResult>`	`hitTest(float xPx, float yPx)` Performs a ray cast from the user's device in the direction of the given location in the camera view.
`List<HitResult>`	`hitTestInstantPlacement(float xPx, float yPx, float approximateDistanceMeters)` Performs a ray cast that can return a result before ARCore establishes full tracking.
`void`	`recordTrackData(UUID trackId, ByteBuffer sample)` Writes a data sample in the specified track.
`void`	`transformCoordinates2d(Coordinates2d inputCoordinates, FloatBuffer inputVertices2d, Coordinates2d outputCoordinates, FloatBuffer outputVertices2d)` Transforms a list of 2D coordinates from one 2D coordinate system to another 2D coordinate system.
`void`	`transformCoordinates2d(Coordinates2d inputCoordinates, float[] inputVertices2d, Coordinates2d outputCoordinates, float[] outputVertices2d)` Transforms a list of 2D coordinates from one 2D coordinate system to another 2D coordinate system.
`void`	`transformCoordinates3d(Coordinates2d inputCoordinates, FloatBuffer inputVertices2d, Coordinates3d outputCoordinates, FloatBuffer outputVertices3d)` Transforms a list of 2D coordinates from one 2D coordinate system to 3D coordinate space.
`void`	`transformCoordinates3d(Coordinates2d inputCoordinates, float[] inputVertices2d, Coordinates3d outputCoordinates, float[] outputVertices3d)` Transforms a list of 2D coordinates from one 2D coordinate system to another 3D coordinate system.
`void`	`transformDisplayUvCoords(FloatBuffer uvCoords, FloatBuffer outUvCoords)` This method is deprecated. Replaced by `frame.transformCoordinates2d(Coordinates2d.VIEW_NORMALIZED, .., Coordinates2d.TEXTURE_NORMALIZED, ..)`.

Inherited Methods

From

class
            java.lang.Object

`Object`	`clone()`
`boolean`	`equals(Object arg0)`
`void`	`finalize()`
`final Class<?>`	`getClass()`
`int`	`hashCode()`
`final void`	`notify()`
`final void`	`notifyAll()`
`String`	`toString()`
`final void`	`wait(long arg0, int arg1)`
`final void`	`wait(long arg0)`
`final void`	`wait()`

Public Methods

public Image acquireCameraImage ()

acquireCameraImage

public Image acquireCameraImage()

Attempts to acquire an image from the camera that corresponds to the current frame. Depending on device performance, can throw NotYetAvailableException for several frames after session start, and for a few frames at a time while the session is running.

Details

Returns an Android Image object that contains the image data from the camera. The returned image object format is AIMAGE_FORMAT_YUV_420_888.

Throws

`NullPointerException`	if `session` or `frame` is null.
`DeadlineExceededException`	if the input frame is not the current frame.
`ResourceExhaustedException`	if the caller app has exceeded maximum number of images that it can hold without releasing.
`NotYetAvailableException`	if the image with the timestamp of the input frame did not become available within a bounded amount of time, or if the camera failed to produce the image.

public Image acquireDepthImage ()

acquireDepthImage

public Image acquireDepthImage()

This method was deprecated.
Please use acquireDepthImage16Bits() instead, which expands the depth range from 8191mm to 65535mm. This deprecated version may be slower than acquireDepthImage16Bits() due to the clearing of the top 3 bits per pixel.

Attempts to acquire a depth Android Image object that corresponds to the current frame.

The depth image has a single 16-bit plane at index 0, stored in little-endian format. Each pixel contains the distance in millimeters to the camera plane. Currently, the three most significant bits are always set to 000. The remaining thirteen bits express values from 0 to 8191, representing depth in millimeters. To extract distance from a depth map, see the Depth API developer guide.

The actual size of the depth image depends on the device and its display aspect ratio. The size of the depth image is typically around 160x120 pixels, with higher resolutions up to 640x480 on some devices. These sizes may change in the future. The outputs of acquireDepthImage(), acquireRawDepthImage() and acquireRawDepthConfidenceImage() will all have the exact same size.

Optimal depth accuracy occurs between 500 millimeters (50 centimeters) and 5000 millimeters (5 meters) from the camera. Error increases quadratically as distance from the camera increases.

Depth is estimated using data from the world-facing cameras, user motion, and hardware depth sensors such as a time-of-flight sensor (or ToF sensor) if available. As the user moves their device through the environment, 3D depth data is collected and cached which improves the quality of subsequent depth images and reducing the error introduced by camera distance.

If an up-to-date depth image isn't ready for the current frame, the most recent depth image available from an earlier frame will be returned instead. This is expected only to occur on compute-constrained devices. An up-to-date depth image should typically become available again within a few frames.

The image must be released via Image.close() once it is no longer needed.

Details

Returns The depth image corresponding to the frame.

Throws

`NotYetAvailableException`	if the number of observed frames is not yet sufficient for depth estimation; or depth estimation was not possible due to poor lighting, camera occlusion, or insufficient motion observed.
`NotTrackingException`	if the `Session` is not in the `TrackingState.TRACKING` state, which is required to acquire depth images.
`IllegalStateException`	if a supported depth mode was not enabled in Session configuration.
`ResourceExhaustedException`	if the caller app has exceeded maximum number of depth images that it can hold without releasing.
`DeadlineExceededException`	if the method is called on not the current frame.

public Image acquireDepthImage16Bits ()

acquireDepthImage16Bits

public Image acquireDepthImage16Bits()

Attempts to acquire a depth Android Image object that corresponds to the current frame.

The depth image has format HardwareBuffer.D_16, which is a single 16-bit plane at index 0, stored in little-endian format. Each pixel contains the distance in millimeters to the camera plane, with the representable depth range between 0 millimeters and 65535 millimeters, or about 65 meters.

To extract distance from a depth map, see the Depth API developer guide.

The actual size of the depth image depends on the device and its display aspect ratio. The size of the depth image is typically around 160x120 pixels, with higher resolutions up to 640x480 on some devices. These sizes may change in the future. The outputs of acquireDepthImage16Bits(), acquireRawDepthImage16Bits() and acquireRawDepthConfidenceImage() will all have the exact same size.

Optimal depth accuracy occurs between 500 millimeters (50 centimeters) and 15000 millimeters (15 meters) from the camera, with depth reliably observed up to 25000 millimeters (25 meters). Error increases quadratically as distance from the camera increases.

When the Geospatial API and the Depth API are enabled, output images from the Depth API will include terrain and building geometry when in a location with VPS coverage. See the Geospatial Depth Developer Guide for more information.

The image must be released via Image.close() once it is no longer needed.

Details

Returns The depth image corresponding to the frame.

Throws

`NotYetAvailableException`	if the number of observed frames is not yet sufficient for depth estimation; or depth estimation was not possible due to poor lighting, camera occlusion, or insufficient motion observed.
`NotTrackingException`	if the `Session` is not in the `TrackingState.TRACKING` state, which is required to acquire depth images.
`IllegalStateException`	if a supported depth mode was not enabled in Session configuration.
`ResourceExhaustedException`	if the caller app has exceeded maximum number of depth images that it can hold without releasing.
`DeadlineExceededException`	if the method is called on not the current frame.

public PointCloud acquirePointCloud ()

acquirePointCloud

public PointCloud acquirePointCloud()

Acquires the current set of estimated 3d points attached to real-world geometry. PointCloud.release() must be called after application is done using the PointCloud object.

Note: This information is for visualization and debugging purposes only. Its characteristics and format are subject to change in subsequent versions of the API.

Details

Throws

`ResourceExhaustedException`	if too many point clouds are acquired without being released.
`DeadlineExceededException`	if this method was called on a frame that isn't the latest frame.

public Image acquireRawDepthConfidenceImage ()

acquireRawDepthConfidenceImage

public Image acquireRawDepthConfidenceImage()

Attempts to acquire the confidence Android Image object corresponding to the raw depth image of the current frame.

The image must be released via Image.close() once it is no longer needed.

Each pixel is an 8-bit unsigned integer representing the estimated confidence of the corresponding pixel in the raw depth image. The confidence value is between 0 and 255, inclusive, with 0 representing the lowest confidence and 255 representing the highest confidence in the measured depth value. Pixels without a valid depth estimate have a confidence value of 0 and a corresponding depth value of 0 (see acquireRawDepthImage()).

The scaling of confidence values is linear and continuous within this range. Expect to see confidence values represented across the full range of 0 to 255, with values increasing as better observations are made of each location. If an application requires filtering out low-confidence pixels, removing depth pixels below a confidence threshold of half confidence (128) tends to work well.

Details

Returns The confidence image corresponding to the raw depth of the frame.

Throws

`NotYetAvailableException`	if the number of observed frames is not yet sufficient for depth estimation; or depth estimation was not possible due to poor lighting, camera occlusion, or insufficient motion observed.
`NotTrackingException`	if the `Session` is not in the `TrackingState.TRACKING` state, which is required to acquire depth images.
`IllegalStateException`	if a supported `Config.DepthMode` was not enabled in Session configuration.
`ResourceExhaustedException`	if the caller app has exceeded maximum number of depth images that it can hold without releasing.
`DeadlineExceededException`	if the method is called on not the current frame.

public Image acquireRawDepthImage ()

acquireRawDepthImage

public Image acquireRawDepthImage()

This method was deprecated.
Please use acquireRawDepthImage16Bits() instead, which expands the depth range from 8191mm to 65535mm. This deprecated version may be slower than acquireRawDepthImage16Bits() due to the clearing of the top 3 bits per pixel.

Attempts to acquire a "raw", mostly unfiltered, depth Android Image object that corresponds to the current frame.

The raw depth image is sparse and does not provide valid depth for all pixels. Pixels without a valid depth estimate have a pixel value of 0 and a corresponding confidence value of 0 (see acquireRawDepthConfidenceImage()).

Optimal depth accuracy occurs between 500 millimeters (50 centimeters) and 5000 millimeters (5 meters) from the camera. Error increases quadratically as distance from the camera increases.

Depth is primarily estimated using data from the motion of world-facing cameras. As the user moves their device through the environment, 3D depth data is collected and cached, improving the quality of subsequent depth images and reducing the error introduced by camera distance. Depth accuracy and robustness improves if the device has a hardware depth sensor, such as a time-of-flight (ToF) camera.

Not every raw depth image contains a new depth estimate. Typically there are about 10 updates to the raw depth data per second. The depth images between those updates are a 3D reprojection which transforms each depth pixel into a 3D point in space and renders those 3D points into a new raw depth image based on the current camera pose. This effectively transforms raw depth image data from a previous frame to account for device movement since the depth data was calculated. For some applications it may be important to know whether the raw depth image contains new depth data or is a 3D reprojection (for example, to reduce the runtime cost of 3D reconstruction). To do that, compare the current raw depth image timestamp, obtained via Image.getTimestamp(), with the previously recorded raw depth image timestamp. If they are different, the depth image contains new information.

The image must be released via Image.close() once it is no longer needed.

Details

Returns The raw depth image corresponding to the frame.

Throws

`NotYetAvailableException`	if the number of observed frames is not yet sufficient for depth estimation; or depth estimation was not possible due to poor lighting, camera occlusion, or insufficient motion observed.
`NotTrackingException`	if the `Session` is not in the `TrackingState.TRACKING` state, which is required to acquire depth images.
`IllegalStateException`	if a supported `Config.DepthMode` was not enabled in Session configuration.
`ResourceExhaustedException`	if the caller app has exceeded maximum number of depth images that it can hold without releasing.
`DeadlineExceededException`	if the method is called on not the current frame.

public Image acquireRawDepthImage16Bits ()

acquireRawDepthImage16Bits

public Image acquireRawDepthImage16Bits()

Attempts to acquire a "raw", mostly unfiltered, depth Android Image object that corresponds to the current frame.

To extract distance from a depth map, see the Depth API developer guide.

The actual size of the depth image depends on the device and its display aspect ratio. The size of the depth image is typically around 160x120 pixels, with higher resolutions up to 640x480 on some devices. These sizes may change in the future. The outputs of acquireDepthImage16Bits(), acquireRawDepthImage16Bits() and acquireRawDepthConfidenceImage() will all have the exact same size.

Depth is primarily estimated using data from the motion of world-facing cameras. As the user moves their device through the environment, 3D depth data is collected and cached, improving the quality of subsequent depth images and reducing the error introduced by camera distance. The depth accuracy improves as the user moves more while looking at a desired location. Depth accuracy and robustness improves if the device has a hardware depth sensor, such as a time-of-flight (ToF) camera.

The image must be released via Image.close() once it is no longer needed.

Details

Returns The raw depth image corresponding to the frame.

Throws

`NotYetAvailableException`	if the number of observed frames is not yet sufficient for depth estimation; or depth estimation was not possible due to poor lighting, camera occlusion, or insufficient motion observed.
`NotTrackingException`	if the `Session` is not in the `TrackingState.TRACKING` state, which is required to acquire depth images.
`IllegalStateException`	if a supported `Config.DepthMode` was not enabled in Session configuration.
`ResourceExhaustedException`	if the caller app has exceeded maximum number of depth images that it can hold without releasing.
`DeadlineExceededException`	if the method is called on not the current frame.

public Image acquireSemanticConfidenceImage ()

acquireSemanticConfidenceImage

public Image acquireSemanticConfidenceImage()

Attempts to acquire the semantic confidence image corresponding to the current frame. Each pixel is an 8-bit integer representing the estimated confidence of the corresponding pixel in the semantic image. See the Scene Semantics Developer Guide for more information.

The confidence value is between 0 and 255, inclusive, with 0 representing the lowest confidence and 255 representing the highest confidence in the semantic class prediction from acquireSemanticImage().

The image must be released via Image.close() once it is no longer needed.

In order to obtain a valid result from this function, you must set the session's Config.SemanticMode to Config.SemanticMode.ENABLED. Use Session.isSemanticModeSupported(Config.SemanticMode) to query for support for Scene Semantics.

The size of the semantic confidence image is the same size as the image obtained by acquireSemanticImage().

Details

Throws

`NotYetAvailableException`	if no semantic image is available that corresponds to the frame.
`ResourceExhaustedException`	if the caller app has exceeded maximum number of images that it can hold without releasing.
`DeadlineExceededException`	if the method is called on not the current frame.

public Image acquireSemanticImage ()

acquireSemanticImage

public Image acquireSemanticImage()

Attempts to acquire the semantic image corresponding to the current frame. Each pixel in the image is an 8-bit unsigned integer representing a semantic class label: see SemanticLabel for a list of pixel labels and the Scene Semantics Developer Guide for more information.

The image must be released via Image.close() once it is no longer needed.

The width of the semantic image is currently 256 pixels. The height of the image depends on the device and will match its display aspect ratio.

Details

Throws

`NotYetAvailableException`	if no semantic image is available that corresponds to the frame.
`ResourceExhaustedException`	if the caller app has exceeded maximum number of images that it can hold without releasing.
`DeadlineExceededException`	if the method is called on not the current frame.

public long getAndroidCameraTimestamp ()

getAndroidCameraTimestamp

public long getAndroidCameraTimestamp()

Returns the (Android Camera timestamp) of the image.

public Pose getAndroidSensorPose ()

getAndroidSensorPose

public Pose getAndroidSensorPose()

Returns the pose of the Android Sensor Coordinate System in the world coordinate space for this frame. The orientation follows the device's "native" orientation (it is not affected by display rotation) with all axes corresponding to those of the Android sensor coordinates.

getCamera

public Camera getCamera()

Returns the Camera object for the session. Note that Camera instances are long-lived and may be kept across frames for the duration of the session. Repeated calls to this method will return distinct Camera objects that refer to the same underlying camera state and are equal to each other.

public int getCameraTextureName ()

getCameraTextureName

public int getCameraTextureName()

Returns the OpenGL ES camera texture name (id) associated with this frame. This is guaranteed to be one of the texture names previously set via Session.setCameraTextureNames(int[]) or Session.setCameraTextureName(int). Texture names (ids) are returned in a round robin fashion in sequential frames.

Details
Returns	the OpenGL ES texture name (id).

public HardwareBuffer getHardwareBuffer ()

getHardwareBuffer

public HardwareBuffer getHardwareBuffer()

Gets the HardwareBuffer for this frame. See Vulkan Rendering developer guide for more information.

Should only be called when a configuration is active that uses Config.TextureUpdateMode.EXPOSE_HARDWARE_BUFFER.

Details

Throws

`NotYetAvailableException`	when the camera failed to produce the image.
`DeadlineExceededException`	if this method was called on a frame that isn't the latest frame.
`IllegalStateException`

public ImageMetadata getImageMetadata ()

getImageMetadata

public ImageMetadata getImageMetadata()

Returns the camera metadata for the current camera image, if available. Throws NotYetAvailableException when metadata is not yet available due to sensors data not yet being available.

If the AR session was created for shared camera access, this method will throw IllegalStateException. To retrieve image metadata in shared camera mode, use SharedCamera.setCaptureCallback(CameraCaptureSession.CaptureCallback, Handler), then use getAndroidCameraTimestamp() to correlate the frame to metadata retrieved from CameraCaptureSession.CaptureCallback.

Details

Throws

`NotYetAvailableException`	when metadata is not available because the sensors are not ready.
`DeadlineExceededException`	if this method was called on a frame that isn't the latest frame.
`ResourceExhaustedException`	if too many metadata objects are currently held.

public LightEstimate getLightEstimate ()

getLightEstimate

public LightEstimate getLightEstimate()

Returns the current ambient light estimate, if light estimation was enabled.

If lighting estimation is not enabled in the session configuration, the returned LightingEstimate will always return LightEstimate.State.NOT_VALID from LightEstimate.getState().

public float getSemanticLabelFraction (SemanticLabel queryLabel)

getSemanticLabelFraction

public float getSemanticLabelFraction(
  SemanticLabel queryLabel
)

Retrieves the percentage of pixels in the most recent semantics image that are queryLabel.

Queries the semantic image provided by acquireSemanticImage() for pixels labelled by queryLabel. This call is more efficient than retrieving the Image and performing a pixel-wise search for the detected label.

Details

Parameters

queryLabel

Throws

NotYetAvailableException if no semantic image has been detected since the start of the session.

public long getTimestamp ()

getTimestamp

public long getTimestamp()

Returns the timestamp in nanoseconds when this image was captured. This can be used to detect dropped frames or measure the camera frame rate. The time base of this value is specifically not defined, but it is likely similar to System.nanoTime().

public Collection<Anchor> getUpdatedAnchors ()

getUpdatedAnchors

public Collection<Anchor> getUpdatedAnchors()

Returns the anchors that were changed by the Session.update() that returned this Frame.

public Collection<TrackData> getUpdatedTrackData (UUID trackUuid)

getUpdatedTrackData

public Collection<TrackData> getUpdatedTrackData(
  UUID trackUuid
)

Retrieve all track data that was written to the specified track during the current frame. If frames are skipped during playback, which can happen when the device is under load, played back track data will be attached to a later frame in order.

Each call to recordTrackData(UUID, ByteBuffer) at recording time will be returned as a separate TrackData entry in the collection.

Details

Parameters

trackUuid

Throws

DeadlineExceededException if this method is called on a frame that is not the current frame (most recent produced by Session.update()).

public Collection<T> getUpdatedTrackables (Class<T> filterType)

getUpdatedTrackables

public Collection<T> getUpdatedTrackables(
  Class<T> filterType
)

Returns the trackables of a particular type that were changed by the Session.update() that returned this Frame. filterType may be Plane.class or Point.class, or Trackable.class to retrieve all changed trackables.

Details

Parameters

filterType

public boolean hasDisplayGeometryChanged ()

hasDisplayGeometryChanged

public boolean hasDisplayGeometryChanged()

Checks if the display rotation or viewport geometry changed since the previous Frame. The application should re-query Camera.getProjectionMatrix(float[], int, float, float) and transformCoordinates2d(Coordinates2d, float[], Coordinates2d, float[]) whenever this is true.

public List<HitResult> hitTest (MotionEvent motionEvent)

hitTest

public List<HitResult> hitTest(
  MotionEvent motionEvent
)

Similar to hitTest(float, float), but will take values from Android MotionEvent. It is assumed that the MotionEvent is received from the same view that was used as the size for Session.setDisplayGeometry(int, int, int).

Note: this method does not consider the action of the MotionEvent. The caller must check for appropriate action, if needed, before calling this method.

Note: When using Session.Feature.FRONT_CAMERA, the returned hit result list will always be empty, as the camera is not TrackingState.TRACKING. Hit testing against tracked faces is not currently supported.

Details

Parameters

motionEvent an event containing the x,y coordinates to hit test

public List<HitResult> hitTest (float[] origin3, int originOffset, float[] direction3, int directionOffset)

hitTest

public List<HitResult> hitTest(
  float[] origin3,
  int originOffset,
  float[] direction3,
  int directionOffset
)

Similar to hitTest(float, float), but takes an arbitrary ray in world space coordinates instead of a screen-space point.

Details

Parameters

`origin3`	an array of 3 floats containing ray origin in world space coordinates.
`originOffset`	the offset into `origin3` array.
`direction3`	an array of 3 floats containing ray direction in world space coordinates. Does not have to be normalized.
`directionOffset`	the offset into `direction3` array.

Returns an ordered list of intersections with scene geometry, nearest hit first.

public List<HitResult> hitTest (float xPx, float yPx)

hitTest

public List<HitResult> hitTest(
  float xPx,
  float yPx
)

Performs a ray cast from the user's device in the direction of the given location in the camera view. Intersections with detected scene geometry are returned, sorted by distance from the device; the nearest intersection is returned first.

Note: Significant geometric leeway is given when returning hit results. For example, a plane hit may be generated if the ray came close, but did not actually hit within the plane extents or plane bounds (Plane.isPoseInExtents(Pose) and Plane.isPoseInPolygon(Pose) can be used to determine these cases). A point (point cloud) hit is generated when a point is roughly within one finger-width of the provided screen coordinates.

Note: In ARCore 1.24.0 or later on supported devices, if depth is enabled by calling Config.setDepthMode(Config.DepthMode) with the value Config.DepthMode.AUTOMATIC, the returned list includes DepthPoint values sampled from the latest computed depth image.

Details

Parameters

`xPx`	x coordinate in pixels
`yPx`	y coordinate in pixels

Returns an ordered list of intersections with scene geometry, nearest hit first

public List<HitResult> hitTestInstantPlacement (float xPx, float yPx, float approximateDistanceMeters)

hitTestInstantPlacement

public List<HitResult> hitTestInstantPlacement(
  float xPx,
  float yPx,
  float approximateDistanceMeters
)

Performs a ray cast that can return a result before ARCore establishes full tracking.

The pose and apparent scale of attached objects depends on the InstantPlacementPoint tracking method and the provided approximateDistanceMeters. A discussion of the different tracking methods and the effects of apparent object scale are described in InstantPlacementPoint.

This function will succeed only if Config.InstantPlacementMode is Config.InstantPlacementMode.LOCAL_Y_UP in the ARCore session configuration, the ARCore session tracking state is TrackingState.TRACKING, and there are sufficient feature points to track the point in screen space.

Details

Parameters

`xPx`	x screen coordinate in pixels
`yPx`	y screen coordinate in pixels
`approximateDistanceMeters`	the distance at which to create an `InstantPlacementPoint`. This is only used while the tracking method for the returned point is `InstantPlacementPoint.TrackingMethod.SCREENSPACE_WITH_APPROXIMATE_DISTANCE`.

Returns if successful a list containing a single HitResult, otherwise an empty list. The HitResult will have a trackable of type InstantPlacementPoint.

public void recordTrackData (UUID trackId, ByteBuffer sample)

recordTrackData

public void recordTrackData(
  UUID trackId,
  ByteBuffer sample
)

Writes a data sample in the specified track. The samples recorded using this API are muxed into the recorded MP4 dataset as an additional MP4 stream.

Multiple samples can be recorded to the same frame and will be played back together.

For smooth playback of the MP4 on video players and for future compatibility of the MP4 datasets with ARCore's playback of data tracks it is recommended that the samples are recorded at a frequency no higher than 90kHz.

Additionally, if the samples are recorded at a frequency lower than 1Hz, empty (zero byte) padding samples will be automatically recorded at approximately one second intervals to fill in the gaps.

Recording samples introduces additional CPU and/or I/O overhead and may affect app performance.

Details

Parameters

`trackId`	The `UUID` of the track.
`sample`	The `ByteBuffer` representation of the sample to be written.

Throws

`IllegalStateException`	when either `Session.getRecordingStatus()` is not currently `RecordingStatus.OK` or the system is currently under excess load for images to be produced. The system should not be under such excess load for more than a few frames and an app should try to record the data again during the next frame.
`IllegalArgumentException`	when any argument is invalid, e.g. null.
`DeadlineExceededException`	when the frame is not the current frame.

public void transformCoordinates2d (Coordinates2d inputCoordinates, FloatBuffer inputVertices2d, Coordinates2d outputCoordinates, FloatBuffer outputVertices2d)

transformCoordinates2d

public void transformCoordinates2d(
  Coordinates2d inputCoordinates,
  FloatBuffer inputVertices2d,
  Coordinates2d outputCoordinates,
  FloatBuffer outputVertices2d
)

Transforms a list of 2D coordinates from one 2D coordinate system to another 2D coordinate system.

For Android view coordinates (Coordinates2d.VIEW, Coordinates2d.VIEW_NORMALIZED), the view information is taken from the most recent call to Session.setDisplayGeometry(int, int, int).

Must be called on the most recently obtained Frame object. If this function is called on an older frame, a log message will be printed and outputVertices2d will remain unchanged.

Some examples of useful conversions:

To transform from [0,1] range to screen-quad coordinates for rendering: Coordinates2d.VIEW_NORMALIZED -> Coordinates2d.TEXTURE_NORMALIZED
To transform from [-1,1] range to screen-quad coordinates for rendering: Coordinates2d.OPENGL_NORMALIZED_DEVICE_COORDINATES -> Coordinates2d.TEXTURE_NORMALIZED
To transform a point found by a computer vision algorithm in a CPU image into a point on the screen that can be used to place an Android View (e.g. Button) at that location: Coordinates2d.IMAGE_PIXELS -> Coordinates2d.VIEW
To transform a point found by a computer vision algorithm in a CPU image into a point to be rendered using GL in clip-space ([-1,1] range): Coordinates2d.IMAGE_PIXELS -> Coordinates2d.OPENGL_NORMALIZED_DEVICE_COORDINATES

Read-only array-backed buffers are not supported by inputVertices2d for performance reasons.

If inputCoordinates is same as outputCoordinates, the input vertices will be copied to the output vertices unmodified.

Details

Parameters

`inputCoordinates`	The coordinate system used by `inputVertices2d`.
`inputVertices2d`	Input 2D vertices to transform.
`outputCoordinates`	The coordinate system to convert to.
`outputVertices2d`	Buffer to put the transformed 2D vertices into.

Throws

`IllegalArgumentException`	If the buffer sizes don't match, or the input/output buffers have odd size.
`ReadOnlyBufferException`	If this buffer is a read-only array backed buffer.

public void transformCoordinates2d (Coordinates2d inputCoordinates, float[] inputVertices2d, Coordinates2d outputCoordinates, float[] outputVertices2d)

transformCoordinates2d

public void transformCoordinates2d(
  Coordinates2d inputCoordinates,
  float[] inputVertices2d,
  Coordinates2d outputCoordinates,
  float[] outputVertices2d
)

Transforms a list of 2D coordinates from one 2D coordinate system to another 2D coordinate system.

Same as transformCoordinates2d(Coordinates2d, FloatBuffer, Coordinates2d, FloatBuffer), but taking float arrays.

Details

Parameters

`inputCoordinates`	The coordinate system used by `inputVertices2d`.
`inputVertices2d`	Input 2D vertices to transform.
`outputCoordinates`	The coordinate system to convert to.
`outputVertices2d`	Buffer to put the transformed 2D vertices into.

public void transformCoordinates3d (Coordinates2d inputCoordinates, FloatBuffer inputVertices2d, Coordinates3d outputCoordinates, FloatBuffer outputVertices3d)

transformCoordinates3d

public void transformCoordinates3d(
  Coordinates2d inputCoordinates,
  FloatBuffer inputVertices2d,
  Coordinates3d outputCoordinates,
  FloatBuffer outputVertices3d
)

Transforms a list of 2D coordinates from one 2D coordinate system to 3D coordinate space. See the Electronic Image Stabilization developer guide for more information.

The view information is taken from the most recent call to Session.setDisplayGeometry(int, int, int).

If Electronic Image Stabilization is off, the device coordinates return (-1, -1, 0) -> (1, 1, 0) and texture coordinates return the same coordinates as transformCoordinates2d(Coordinates2d, float[], Coordinates2d, float[])} with the Z component set to 1.0f.

In order to use EIS, your app should use EIS compensated screen coordinates and camera texture coordinates to pass on to shaders. Use the 2D NDC space coordinates as input to obtain EIS compensated 3D screen coordinates and matching camera texture coordinates.

Details

Parameters

`inputCoordinates`	The coordinate system used by `inputVertices2d`.
`inputVertices2d`	Input 2D vertices to transform.
`outputCoordinates`	The 3D coordinate system to output in `outputVertices3d`.
`outputVertices3d`	Buffer to put the transformed 3D vertices into.

public void transformCoordinates3d (Coordinates2d inputCoordinates, float[] inputVertices2d, Coordinates3d outputCoordinates, float[] outputVertices3d)

transformCoordinates3d

public void transformCoordinates3d(
  Coordinates2d inputCoordinates,
  float[] inputVertices2d,
  Coordinates3d outputCoordinates,
  float[] outputVertices3d
)

Transforms a list of 2D coordinates from one 2D coordinate system to another 3D coordinate system.

Same as transformCoordinates3d(Coordinates2d, FloatBuffer, Coordinates3d, FloatBuffer), but taking float arrays.

Details

Parameters

`inputCoordinates`	The coordinate system used by `inputVertices2d`.
`inputVertices2d`	Input 2D vertices to transform.
`outputCoordinates`	The 3D coordinate system to output in `outputVertices3d`.
`outputVertices3d`	Buffer to put the transformed 3D vertices into.

public void transformDisplayUvCoords (FloatBuffer uvCoords, FloatBuffer outUvCoords)

transformDisplayUvCoords

public void transformDisplayUvCoords(
  FloatBuffer uvCoords,
  FloatBuffer outUvCoords
)

This method was deprecated.
Replaced by frame.transformCoordinates2d(Coordinates2d.VIEW_NORMALIZED, .., Coordinates2d.TEXTURE_NORMALIZED, ..).

Transform the given texture coordinates to correctly show the background image. This will account for the display rotation, and any additional required adjustment. For performance, this function should be called only if hasDisplayGeometryChanged() returns true.

Usage Notes / Bugs:

Both input and output buffers must be direct and native byte order.
Position and limit of buffers is ignored.
Capacity of both buffers must be identical.
Capacity of both buffers must be a multiple of 2.

Note: both buffer positions will remain unmodified after this call.

Details

Parameters

`uvCoords`	The uv coordinates to transform.
`outUvCoords`	The buffer to hold the transformed uv coordinates. Must have enough remaining elements to fit the input uvCoords.