Accessing JavaScript binary data in C++

[mateogianolio]

Today I thought I'd share a method on how to efficiently work with data structures when making Node.js addons. Specifically, I will show you how to represent a struct in JavaScript and how to access and manipulate this data from v8. Because there is no official documentation for v8 except the reference material generated from v8.h, this post is meant to bridge the language gap between JavaScript and C/C++.

Data storage

Typed arrays were introduced a few years ago and they provide us with a way of accessing and manipulating raw binary data. The different types of typed arrays and their C equivalents are shown in the below table (derived from the MDN page):

Type	Size in bytes	Equivalent C type
Int8Array	1	int8_t
Uint8Array	1	uint8_t
Uint8ClampedArray	1	uint8_t
Int16Array	2	int16_t
Uint16Array	2	uint16_t
Int32Array	4	int32_t
Uint32Array	4	uint32_t
Float32Array	4	float
Float64Array	8	double

It's good to know that the above are only different views on the underlying ArrayBuffer storage. We can initialize typed arrays using regular arrays, like this:

var view = new Int8Array([1, 2, 3]);

We can also initialize them by allocating memory using an ArrayBuffer:

var buffer = new ArrayBuffer(3),
    view = new Int8Array(buffer);

view.set([1, 2, 3]);

More importantly, we can use the DataView interface to read and write arbitrary data to an ArrayBuffer:

var buffer = new ArrayBuffer(3),
    view = new DataView(buffer);

view.setInt8(0, 1, true);
view.setInt16(1, 2, true);

console.log(view.getInt8(0, true)); // 1
console.log(view.getInt16(1, true)); // 2

Note: the last argument of setInt8 specifies the endianness of the system. A value of true means little endian.

In C++, the Int8Array in the first two examples can be represented with an int8_t array. The same applies to the other types. Just look up the C equivalent in the table above.

int8_t data[] = {1, 2, 3};

The DataView in the last example can effectively be represented as a struct:

typedef struct {
    int8_t a;
    int16_t b;
} dataView;

So we have established some common ground between JavaScript and C++. The remaining question is how do we read this data into a Node.js addon?

Data access

Let's set up a simple example. If you have no idea what a Node.js addon is I suggest you read this guide.

#include <node.h>

typedef struct {
    int8_t a;
    int16_t b;
} dataView;

void accessInt8Array(const v8::FunctionCallbackInfo<v8::Value>& info) {
    // make sure the first argument is an Int8Array
    assert(info[0]->IsInt8Array());

    // read first argument as an Int8Array.
    v8::Local<v8::Int8Array> view = info[0].As<v8::Int8Array>();

    // get contents as a void pointer
    void *data = view->Buffer()->GetContents().Data();

    // create a pointer to int8_t and typecast
    int8_t *contents = static_cast<int8_t*>(data);

    // multiply all elements by 2
    for (int i = 0; i < view->Length(); i++)
        contents[i] *= 2;
}

void accessDataView(const v8::FunctionCallbackInfo<v8::Value>& info) {
    assert(info[0]->IsDataView());
    v8::Local<v8::DataView> view = info[0].As<v8::DataView>();

    // check size to make sure the data is compatible with our struct
    assert(view->ByteLength() == sizeof(dataView));

    void *data = view->Buffer()->GetContents().Data();
    dataView *contents = static_cast<dataView*>(data);

    // multiply both integers by 2
    contents->a *= 2;
    contents->b *= 2;
}


void init(v8::Local<v8::Object> exports) {
    NODE_SET_METHOD(exports, "accessInt8Array", accessInt8Array);
    NODE_SET_METHOD(exports, "accessDataView", accessDataView);
}

NODE_MODULE(addon, init);

Note: If you want to have full control over memory and avoid unexpected garbage collection, you should replace GetContents() with Externalize(). You will then lose the ability to manipulate the contents directly and the responsibility to free() the memory lies on you.

Finally, we create a binding.gyp file, compile with node-gyp configure rebuild and try it out!

var addon = require('./build/Release/addon'),
    buffer,
    view;

view = new Int8Array([1, 2, 3]);
addon.accessInt8Array(view);

console.log(view);
// Int8Array [ 2, 4, 6 ]

buffer = new ArrayBuffer(3);
view = new DataView(buffer);
view.setInt8(0, 1, true);
view.setInt16(1, 2, true);

addon.accessDataView(view);

console.log(view.getInt8(0, true)); // 2
console.log(view.getInt16(1, true)); // 4

Now we can work with practically any kind of binary data in both JavaScript and C++.

Success!

[yaakovyitzchak]

how do u do the other way around -- create an arraybuffer in C++ from image data or something and pass it back to javascript?

[mateogianolio]

Example (see https://stackoverflow.com/questions/44189618/how-to-read-write-data-to-arraybuffer-from-c-node-js-addon):

void test(const v8::FunctionCallbackInfo<v8::Value>& info) {
  int n = 3;
  double* data = (double*) malloc(sizeof(double) * n);

  int i;
  for (i = 0; i < n; i++) {
    data[i] = i;
  }
  
  v8::Isolate* isolate = info.GetIsolate();
  v8::Local<v8::ArrayBuffer> buffer = v8::ArrayBuffer::New(isolate, data, n * sizeof(double), v8::ArrayBufferCreationMode::kInternalized);

  return info.GetReturnValue().Set(buffer);
}

Then, you can access the data in JavaScript like this:

// ...

const buffer = test();
const view = new Float64Array(buffer);
console.log(view);
// Float64Array(3) [ 0, 1, 2 ]

[mateogianolio]

Notice how I didn’t use free. Setting the last argument to v8::ArrayBufferCreationMode::kInternalized hands over responsibility of freeing the memory to the GC.