Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/content/docs/a-depth-frame.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -59,12 +59,13 @@ const buffer = depth.getDepthData()

#### Interpreting Depth Data

The Depth Data's layout depends on the [`Depth`](/api/react-native-vision-camera/hybrid-objects/Depth)'s [`pixelFormat`](/api/react-native-vision-camera/hybrid-objects/Depth#pixelformat). For example, in [`'depth-16-bit'`](/api/react-native-vision-camera/type-aliases/DepthPixelFormat), pixels are laid out in 16-bit floats - one float per "pixel".
The Depth Data's layout depends on the [`Depth`](/api/react-native-vision-camera/hybrid-objects/Depth)'s [`pixelFormat`](/api/react-native-vision-camera/hybrid-objects/Depth#pixelformat). On iOS, [`'depth-16-bit'`](/api/react-native-vision-camera/type-aliases/DepthPixelFormat) stores one 16-bit float depth sample per pixel. On Android, [`'depth-16-bit'`](/api/react-native-vision-camera/type-aliases/DepthPixelFormat) maps to `ImageFormat.DEPTH16`, where each 16-bit sample stores range and confidence bits.

```ts
const depth = ...
const buffer = depth.getDepthData()
if (depth.pixelFormat === 'depth-16') {
if (depth.pixelFormat === 'depth-16-bit') {
// iOS depth-16-bit data is Float16. Android DEPTH16 uses packed Uint16 samples.
const pixels = new Float16Array(buffer)
// [!code ++]
const distanceToFirstPixel = pixels[0]
Expand All @@ -73,7 +74,8 @@ if (depth.pixelFormat === 'depth-16') {
```

> [!TIP]
> Typically, a single 16-bit float in [`'depth-16-bit'`](/api/react-native-vision-camera/type-aliases/DepthPixelFormat) data represents a distance in meters.
> On iOS, a single 16-bit float in [`'depth-16-bit'`](/api/react-native-vision-camera/type-aliases/DepthPixelFormat) data represents a distance in meters.
> On Android, read `ImageFormat.DEPTH16` as unsigned 16-bit samples: the low 13 bits are range in millimeters and the high 3 bits are confidence.

#### Converting between Depth Formats

Expand Down
4 changes: 2 additions & 2 deletions docs/content/docs/a-frame.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ console.log(frame.pixelFormat) // 'yuv-420-8-bit-full'
```

While the most commonly known [`PixelFormat`](/api/react-native-vision-camera/type-aliases/PixelFormat) is RGB (often [`'rgb-bgra-8-bit'`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat)), it is not natively produced by the Camera, which means it requires expensive conversion causing higher latency, more bandwidth, and overall slower performance.
The Camera's native pixel format is typically YUV (often [`'yuv-420-8-bit-full'`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat)) or a vendor-specific variation of it ([`'private'`](/api/react-native-vision-camera/type-aliases/PixelFormat)), which uses ~50% less memory than RGB and requires little to no conversion overhead.
The Camera's native pixel format is typically YUV (for example [`'yuv-420-8-bit-video'`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) on iOS native capture or [`'yuv-420-8-bit-full'`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) for common CPU-readable YUV paths), or a vendor-specific variation of it ([`'private'`](/api/react-native-vision-camera/type-aliases/PixelFormat)), which uses ~50% less memory than RGB and requires little to no conversion overhead.

> [!NOTE]
> See ["Frame Output: Choosing a Pixel Format"](frame-output#choosing-a-pixel-format) for more information about configuring the Pixel Format a [`CameraFrameOutput`](/api/react-native-vision-camera/hybrid-objects/CameraFrameOutput) streams in.
Expand Down Expand Up @@ -132,7 +132,7 @@ if (frame.isPlanar) {
const uvBuffer = planes[1].getPixelBuffer()
const yPixels = new Uint8Array(yBuffer)
const uvPixels = new Uint8Array(uvBuffer)
const firstPixel = { y: yPixels[0], u: uvPixels[0], v: uvPixels[0] }
const firstPixel = { y: yPixels[0], u: uvPixels[0], v: uvPixels[1] }
}
} else {
// regular pixel buffer access
Expand Down
9 changes: 6 additions & 3 deletions docs/content/docs/pixel-formats-map.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,13 @@ In a [`CameraFrameOutput`](/api/react-native-vision-camera/hybrid-objects/Camera

When you receive a [`Frame`](/api/react-native-vision-camera/hybrid-objects/Frame), you can inspect its [`pixelFormat`](/api/react-native-vision-camera/hybrid-objects/Frame#pixelformat) to find out the precise Pixel Format it uses - here are some examples that map to native pixel formats:

| VisionCamera [`VideoPixelFormat`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | iOS [`CVPixelFormatType`](https://developer.apple.com/documentation/corevideo/cvpixelformattype) | Android [`ImageFormat`](https://developer.android.com/reference/android/graphics/ImageFormat) |
| VisionCamera [`PixelFormat`](/api/react-native-vision-camera/type-aliases/PixelFormat) | iOS [`CVPixelFormatType`](https://developer.apple.com/documentation/corevideo/cvpixelformattype) | Android format |
|---------------|--------|---------------------|
| [`yuv-420-8-bit-video`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | [`420YpCbCr8BiPlanarVideoRange`](https://developer.apple.com/documentation/corevideo/kcvpixelformattype_420ypcbcr8biplanarvideorange) | [`YUV_420_888`](https://developer.android.com/reference/android/graphics/ImageFormat#YUV_420_888) + limited-range `DataSpace` |
| [`yuv-420-8-bit-full`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | [`420YpCbCr8BiPlanarFullRange`](https://developer.apple.com/documentation/CoreVideo/kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) | [`YUV_420_888`](https://developer.android.com/reference/android/graphics/ImageFormat#YUV_420_888) |
| [`yuv-420-10-bit-full`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | [`420YpCbCr10BiPlanarFullRange`](https://developer.apple.com/documentation/corevideo/kcvpixelformattype_420ypcbcr10biplanarfullrange) | [`YCBCR_P010`](https://developer.android.com/reference/android/graphics/ImageFormat#YCBCR_P010) |
| [`yuv-444-10-bit-video`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | [`444YpCbCr10BiPlanarVideoRange`](https://developer.apple.com/documentation/corevideo/kcvpixelformattype_444ypcbcr10biplanarvideorange) | / |
| [`yuv-444-10-bit-full`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | [`444YpCbCr10BiPlanarFullRange`](https://developer.apple.com/documentation/corevideo/kcvpixelformattype_444ypcbcr10biplanarfullrange) | / |
| [`rgb-bgra-8-bit`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | [`32BGRA`](https://developer.apple.com/documentation/CoreVideo/kCVPixelFormatType_32BGRA) | / |
| [`rgb-rgba-8-bit`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | / | [`FLEX_RGBA_888`](https://developer.android.com/reference/android/graphics/ImageFormat#FLEX_RGBA_888) |
| [`private`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | / | [`PRIVATE`](https://developer.android.com/reference/android/graphics/ImageFormat#PRIVATE) |
| [`rgb-rgba-8-bit`](/api/react-native-vision-camera/type-aliases/VideoPixelFormat) | [`32RGBA`](https://developer.apple.com/documentation/corevideo/kcvpixelformattype_32rgba) | [`PixelFormat.RGBA_8888`](https://developer.android.com/reference/android/graphics/PixelFormat#RGBA_8888) or [`FLEX_RGBA_8888`](https://developer.android.com/reference/android/graphics/ImageFormat#FLEX_RGBA_8888) |
| [`private`](/api/react-native-vision-camera/type-aliases/PixelFormat) | / | [`PRIVATE`](https://developer.android.com/reference/android/graphics/ImageFormat#PRIVATE) |
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package com.margelo.nitro.camera.extensions.converters
import android.graphics.ImageFormat
import com.margelo.nitro.camera.PixelFormat
import com.margelo.nitro.camera.utils.PixelRange
import android.graphics.PixelFormat as AndroidPixelFormat

fun PixelFormat.Companion.fromImageFormat(
imageFormat: Int,
Expand All @@ -11,6 +12,8 @@ fun PixelFormat.Companion.fromImageFormat(
return when (imageFormat) {
ImageFormat.FLEX_RGB_888 -> PixelFormat.RGB_RGB_8_BIT
ImageFormat.FLEX_RGBA_8888 -> PixelFormat.RGB_RGBA_8_BIT
AndroidPixelFormat.RGB_888 -> PixelFormat.RGB_RGB_8_BIT
AndroidPixelFormat.RGBA_8888 -> PixelFormat.RGB_RGBA_8_BIT
ImageFormat.DEPTH16 -> PixelFormat.DEPTH_16_BIT
ImageFormat.DEPTH_POINT_CLOUD -> PixelFormat.DEPTH_POINT_CLOUD_32_BIT
ImageFormat.YUV_420_888 -> {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package com.margelo.nitro.camera.utils

import android.graphics.ImageFormat
import android.graphics.PixelFormat as AndroidPixelFormat

object ImageFormatUtils {
val allVideoFormats =
Expand All @@ -15,6 +16,8 @@ object ImageFormatUtils {
// RGB
ImageFormat.FLEX_RGB_888,
ImageFormat.FLEX_RGBA_8888,
AndroidPixelFormat.RGB_888,
AndroidPixelFormat.RGBA_8888,
)
val allDepthFormats =
arrayOf(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ extension AVCaptureDevice.Format {
var bitDepth: DynamicRangeBitDepth {
let pixelFormat = self.formatDescription.mediaSubType
switch pixelFormat {
case .yuv4208BitVideo, .yuv4208BitFull, .yuv4228BitVideo, .yuv4228BitFull:
case .yuv4208BitVideo, .yuv4208BitFull, .yuv4228BitVideo, .yuv4228BitFull,
.yuv4448BitVideo, .yuv4448BitFull:
return .sdr8Bit
case .yuv42010BitVideo, .yuv42010BitFull, .yuv42210BitVideo, .yuv42210BitFull:
case .yuv42010BitVideo, .yuv42010BitFull, .yuv42210BitVideo, .yuv42210BitFull,
.yuv44410BitVideo, .yuv44410BitFull:
return .hdr10Bit
default:
return .unknown
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ extension AVCaptureDevice.Format {
var colorRange: ColorRange {
let pixelFormat = self.formatDescription.mediaSubType
switch pixelFormat {
case .yuv4208BitVideo, .yuv42010BitVideo, .yuv4228BitVideo, .yuv42210BitVideo:
case .yuv4208BitVideo, .yuv42010BitVideo, .yuv4228BitVideo, .yuv42210BitVideo,
.yuv4448BitVideo, .yuv44410BitVideo:
return .video
case .yuv4208BitFull, .yuv42010BitFull, .yuv4228BitFull, .yuv42210BitFull:
case .yuv4208BitFull, .yuv42010BitFull, .yuv4228BitFull, .yuv42210BitFull,
.yuv4448BitFull, .yuv44410BitFull:
return .full
default:
return .unknown
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ extension PixelFormat {
self = .yuv4448BitVideo
case .yuv4448BitFull:
self = .yuv4448BitFull
case .yuv44410BitVideo:
self = .yuv44410BitVideo
case .yuv44410BitFull:
self = .yuv44410BitFull
case .rgbBgra8Bit:
self = .rgbBgra8Bit
case .rgbRgba8Bit:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,11 @@ extension CMFormatDescription.MediaSubType {
rawValue: kCVPixelFormatType_444YpCbCr8BiPlanarVideoRange)
static let yuv4448BitFull = CMFormatDescription.MediaSubType(
rawValue: kCVPixelFormatType_444YpCbCr8BiPlanarFullRange)
// YUV 4:4:4 10-Bit
static let yuv44410BitVideo = CMFormatDescription.MediaSubType(
rawValue: kCVPixelFormatType_444YpCbCr10BiPlanarVideoRange)
static let yuv44410BitFull = CMFormatDescription.MediaSubType(
rawValue: kCVPixelFormatType_444YpCbCr10BiPlanarFullRange)

// BGRA
static let rgbBgra8Bit = CMFormatDescription.MediaSubType(rawValue: kCVPixelFormatType_32BGRA)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,11 @@ extension TargetColorSpace {
extension CMFormatDescription.MediaSubType {
fileprivate var colorRange: ColorRange {
switch self {
case .yuv4208BitVideo, .yuv42010BitVideo, .yuv4228BitVideo, .yuv42210BitVideo:
case .yuv4208BitVideo, .yuv42010BitVideo, .yuv4228BitVideo, .yuv42210BitVideo,
.yuv4448BitVideo, .yuv44410BitVideo:
return .video
case .yuv4208BitFull, .yuv42010BitFull, .yuv4228BitFull, .yuv42210BitFull:
case .yuv4208BitFull, .yuv42010BitFull, .yuv4228BitFull, .yuv42210BitFull,
.yuv4448BitFull, .yuv44410BitFull:
return .full
default:
return .unknown
Expand Down Expand Up @@ -134,9 +136,11 @@ extension ColorRange {
extension CMFormatDescription.MediaSubType {
var bitDepth: DynamicRangeBitDepth {
switch self {
case .yuv4208BitVideo, .yuv4208BitFull, .yuv4228BitVideo, .yuv4228BitFull:
case .yuv4208BitVideo, .yuv4208BitFull, .yuv4228BitVideo, .yuv4228BitFull,
.yuv4448BitVideo, .yuv4448BitFull:
return .sdr8Bit
case .yuv42010BitVideo, .yuv42010BitFull, .yuv42210BitVideo, .yuv42210BitFull:
case .yuv42010BitVideo, .yuv42010BitFull, .yuv42210BitVideo, .yuv42210BitFull,
.yuv44410BitVideo, .yuv44410BitFull:
return .hdr10Bit
default:
return .unknown
Expand Down
Loading
Loading