Using Resources

Resources are C ABI objects which you declare should be managed by the BEAM reference-counted garbage collector. If you are passing data between function calls, it is best practice to pass them as a resource. Note that Zigler does not currently support passing references between multiple Zigler modules.

Important Note

The examples in this guide build upon each other and are tested in an integration test; so ~Z blocks present build upon code that is declared in previous ~Z blocks.

Basics: Definition

A resource is declared by using the /// resource: <name> definition directive. The name should match a zig type declared in the next line.

Examples

In the following example, the resource i64_res is bound to the builtin i64 type. This would allow you to store a mutable 64-bit integer and pass its pointer between function calls in the BEAM. NOTE: although it's mutable it is NOT threadsafe.

~Z"""
/// resource: i64_res definition
const i64_res = i64;
"""

In the next example, resource struct_res is bound to a custom struct type. This is a more typical situation, where structured data, possibly requiring internal memory allocation, needs to be passed between function calls.

~Z"""
/// resource: struct_res definition
const struct_res = struct {
    first_name: []u8,
    last_name: []u8,
};
"""

Creation

In order to create a resource, a __resource__ struct is autogenerated which makes manipulating BEAM resources much simpler. When returned from the nif, the resource will appear as a reference/0 term. As this term is passed around, potentially between processes, it will only be garbage collected when all references have been to the term have been lost (more on that in cleanup).

The create function takes three parameters. First, the type that is being encapsulated, second, the BEAM environment of the calling function, and third, the initial value for those parameters.

Important notes

  1. You zig code will only have access to the __resource__ struct in the context of NIF code associated with your module in which it resides, within the Zig.sigil_z/2 blocks; you cannot use the __resource__ struct in other modules or zig code which has been @included.
  2. Erlang/Elixir code will have no direct visibility of the content of these resources: you MUST use nif functions to access or modify those contents.
  3. Resource references must remain local. Unlike other erlang terms, the content will not be automatically serialized, transferred to another node, and deserialized on the other side.
  4. Resource creation is failable; you must catch the resource error and handle it before returning the resource term. In a future release it will be possible to pass it upwards with the try function with a full error return trace.
  5. For datastructures (structs, slices) you have the choice of storing it directly as the resource, or storing the pointer to the datastructure as the resource.

Example: Resource Creation

~Z"""
/// nif: create_i64/0
fn create_i64(env: beam.env) beam.term {
  return __resource__.create(i64_res, env, 47)
    catch return beam.raise_resource_error(env);
}

/// nif: create_struct/2
fn create_struct(env: beam.env, first_name: []u8, last_name: []u8) beam.term {
  var resource_fn = beam.allocator.alloc(u8, first_name.len)
    catch return beam.raise_enomem(env);
  errdefer beam.allocator.free(resource_fn);

  var resource_ln = beam.allocator.alloc(u8, last_name.len)
    catch return beam.raise_enomem(env);
  errdefer beam.allocator.free(resource_ln);

  std.mem.copy(u8, resource_fn, first_name);
  std.mem.copy(u8, resource_ln, last_name);

  return __resource__.create(struct_res, env, .{
    .first_name = resource_fn,
    .last_name = resource_ln,
  }) catch return beam.raise_resource_error(env);
}
"""

test "resource creation generates references" do
  i64_res_example = create_i64()
  assert is_reference(i64_res_example)

  struct_res_example = create_struct("john", "doe")
  assert is_reference(struct_res_example)
end

Example: Resource Pointer Creation

~Z"""
/// resource: struct_ptr_res definition
const struct_ptr_res = *struct_res;

/// nif: create_struct_ptr/2
fn create_struct_ptr(env: beam.env, first_name: []u8, last_name: []u8) beam.term {
  var resource = beam.allocator.create(struct_res)
    catch return beam.raise_enomem(env);
  errdefer beam.allocator.destroy(resource);

  resource.first_name = beam.allocator.alloc(u8, first_name.len)
    catch return beam.raise_enomem(env);
  errdefer beam.allocator.free(resource.first_name);

  resource.last_name = beam.allocator.alloc(u8, last_name.len)
    catch return beam.raise_enomem(env);
  errdefer beam.allocator.free(resource.last_name);

  std.mem.copy(u8, resource.first_name, first_name);
  std.mem.copy(u8, resource.last_name, last_name);

  // note the signature        vv here vv
  return __resource__.create(struct_ptr_res, env, resource)
    catch return beam.raise_resource_error(env);
}
"""

test "resource pointer generation creates a reference" do
  struct_ptr_example = create_struct_ptr("john", "doe")
  assert is_reference(struct_ptr_example)
end

Fetching and Updating

The __resource__ struct has fetch and update functions which allow you to retrieve or modify the contents in the struct, respectively. Note that although you can potentially access these data from different processes, it is not threadsafe by default without implementing mutexes.

The fetch function takes three parameters, the internal type of the resource, the calling function environment, and the resource beam.term value.

The update function takes four parameters, the internal type of the resource, the calling function environment, the resource beam.term value, and the replacement value.

Important considerations

  1. Resource fetching is failable, and you should handle it by returning a beam raise term. This may not necessarily be the same type as the contents of the resource itself! So you might have to marshall it into a term manually. Note: this code will likely become simpler in the future.
  2. The data encapsulated by the Resource term are mutable! If you update the term, that change will persist within the same call scope, across function call scopes, etc! Also importantly, if you pass the data to another process you must take care to guard changes around mutexes and be aware that change or could be subject to unexpected race conditions without additional coordination.
  3. For structs, you must replace the entire contents of the struct; depending on your needs, you may want to consider using a pointer, which will let you change the contents of the struct WITHOUT performing an update. Note that extra care must be taken in this case to avoid race conditions resulting in use-after-free segfaults.

Example: Fetching

~Z"""
/// nif: fetch_i64/1
fn fetch_i64(env: beam.env, res: beam.term) beam.term {
  var result = __resource__.fetch(i64_res, env, res)
    catch return beam.raise_resource_error(env);
  return beam.make_i64(env, result);
}

/// nif: fetch_struct_fn/1
fn fetch_struct_fn(env: beam.env, res: beam.term) beam.term {
  var result = __resource__.fetch(struct_res, env, res)
    catch return beam.raise_resource_error(env);
  return beam.make_slice(env, result.first_name);
}

/// nif: fetch_struct_ln/1
fn fetch_struct_ln(env: beam.env, res: beam.term) beam.term {
  var result = __resource__.fetch(struct_res, env, res)
    catch return beam.raise_resource_error(env);
  return beam.make_slice(env, result.last_name);
}

// note that the function bodies for the struct pointer form look NEARLY
// identical to that of the direct struct resource form.

/// nif: fetch_struct_ptr_fn/1
fn fetch_struct_ptr_fn(env: beam.env, res: beam.term) beam.term {
  var result = __resource__.fetch(struct_ptr_res, env, res)
    catch return beam.raise_resource_error(env);
  return beam.make_slice(env, result.first_name);
}

/// nif: fetch_struct_ptr_ln/1
fn fetch_struct_ptr_ln(env: beam.env, res: beam.term) beam.term {
  var result = __resource__.fetch(struct_ptr_res, env, res)
    catch return beam.raise_resource_error(env);
  return beam.make_slice(env, result.last_name);
}

"""

test "resources can be fetched" do
  i64_res_example = create_i64()
  assert 47 == fetch_i64(i64_res_example)

  struct_res_example = create_struct("john", "doe")
  assert "john" == fetch_struct_fn(struct_res_example)
  assert "doe" == fetch_struct_ln(struct_res_example)

  struct_ptr_res_example = create_struct_ptr("john", "doe")
  assert "john" == fetch_struct_ptr_fn(struct_ptr_res_example)
  assert "doe" == fetch_struct_ptr_ln(struct_ptr_res_example)
end

Example: Updating

~Z"""
/// nif: update_i64/2
fn update_i64(env: beam.env, res: beam.term, update_value: i64) beam.term {
  __resource__.update(i64_res, env, res, update_value)
    catch return beam.raise_resource_error(env);
  return beam.make_ok(env);
}

/// nif: update_struct_fn/2
fn update_struct_fn(env: beam.env, res: beam.term, update_value: []u8) beam.term {
  var old_value = __resource__.fetch(struct_res, env, res)
    catch return beam.raise_resource_error(env);

  var new_first_name = beam.allocator.alloc(u8, update_value.len)
    catch return beam.raise_enomem(env);
  errdefer beam.allocator.free(new_first_name);

  std.mem.copy(u8, new_first_name, update_value);
  defer beam.allocator.free(old_value.first_name);

  var new_struct = .{
    .first_name = new_first_name,
    .last_name = old_value.last_name,
  };

  __resource__.update(struct_res, env, res, new_struct)
    catch return beam.raise_resource_error(env);
  return beam.make_ok(env);
}

/// nif: update_struct_ln/2
fn update_struct_ln(env: beam.env, res: beam.term, update_value: []u8) beam.term {
  var old_value = __resource__.fetch(struct_res, env, res)
    catch return beam.raise_resource_error(env);

  var new_last_name = beam.allocator.alloc(u8, update_value.len)
    catch return beam.raise_enomem(env);
  errdefer beam.allocator.free(new_last_name);

  std.mem.copy(u8, new_last_name, update_value);
  defer beam.allocator.free(old_value.last_name);

  var new_struct = .{
    .first_name = old_value.first_name,
    .last_name = new_last_name,
  };

  __resource__.update(struct_res, env, res, new_struct)
    catch return beam.raise_resource_error(env);
  return beam.make_ok(env);
}

// note that updating POINTER structs does not require calling update.

/// nif: update_struct_ptr_fn/2
fn update_struct_ptr_fn(env: beam.env, res: beam.term, update_value: []u8) beam.term {
  var struct_ptr = __resource__.fetch(struct_ptr_res, env, res)
    catch return beam.raise_resource_error(env);

  beam.allocator.free(struct_ptr.first_name);
  struct_ptr.first_name = beam.allocator.alloc(u8, update_value.len)
    catch return beam.raise_enomem(env);
  errdefer beam.allocator.free(struct_ptr.first_name);

  std.mem.copy(u8, struct_ptr.first_name, update_value);

  return beam.make_ok(env);
}

/// nif: update_struct_ptr_ln/2
fn update_struct_ptr_ln(env: beam.env, res: beam.term, update_value: []u8) beam.term {
  var struct_ptr = __resource__.fetch(struct_ptr_res, env, res)
    catch return beam.raise_resource_error(env);

  beam.allocator.free(struct_ptr.last_name);
  struct_ptr.last_name = beam.allocator.alloc(u8, update_value.len)
    catch return beam.raise_enomem(env);
  errdefer beam.allocator.free(struct_ptr.last_name);

  std.mem.copy(u8, struct_ptr.last_name, update_value);

  return beam.make_ok(env);
}
"""

test "resources can be updated" do
  i64_res_example = create_i64()
  assert :ok == update_i64(i64_res_example, 50)
  assert 50 == fetch_i64(i64_res_example)

  struct_res_example = create_struct("john", "doe")
  assert :ok == update_struct_fn(struct_res_example, "jane")
  assert "jane" == fetch_struct_fn(struct_res_example)
  assert "doe" == fetch_struct_ln(struct_res_example)

  assert :ok == update_struct_ln(struct_res_example, "smith")
  assert "jane" == fetch_struct_fn(struct_res_example)
  assert "smith" == fetch_struct_ln(struct_res_example)

  struct_res_example = create_struct_ptr("john", "doe")
  assert :ok == update_struct_ptr_fn(struct_res_example, "jane")
  assert "jane" == fetch_struct_ptr_fn(struct_res_example)
  assert "doe" == fetch_struct_ptr_ln(struct_res_example)

  assert :ok == update_struct_ptr_ln(struct_res_example, "smith")
  assert "jane" == fetch_struct_ptr_fn(struct_res_example)
  assert "smith" == fetch_struct_ptr_ln(struct_res_example)
end

Keep and Release

If you are using resources in threaded or yielding long-running nifs, you should perform keep and release operations on these resources. This will prevent the garbage collector from triggering cleanup on your resource in the event that the calling process has been terminated. If you don't perform keep/release on the resource, it will at some point result in a segfault due to use-after-free and it may be difficult to detect because triggering segfault may depend on race conditions, VM computational load, and memory page layout.

keep is failable and takes out a gc reference to your resource; it should always be matched by a release call (typically with a defer statement). Release is not failable.

Both functions take three parameters, the type of the internal data structure, the calling function environment, and the resource term.

Example: Keep and Release

~Z"""
/// nif: long_running_i64/1 threaded
fn long_running_i64(env: beam.env, res: beam.term) beam.term {
  __resource__.keep(i64_res, env, res)
    catch return beam.raise_resource_error(env);
  defer __resource__.release(i64_res, env, res);

  var int_value = __resource__.fetch(i64_res, env, res)
    catch return beam.raise_resource_error(env);

  return beam.make_i64(env, int_value);
}
"""

test "keep and release" do
  i64_res_example = create_i64()
  assert 47 == long_running_i64(i64_res_example)
end

Cleanup

Most resources should define a cleanup function. This function will be invoked when the last reference to that resource has been garbage collected by the virtual machine. The cleanup function is declared by using the /// resource: <name> cleanup, this should be followed by a void function taking a pointer to the resource. If there are any internal data that need freeing, you should do it here.

Important Note

  • The pointer for the resource itself will be cleaned up by Zigler, so don't free that.
  • If your resource is a struct pointer, DO clean that up.

Example: Cleanup

~Z"""
/// resource: struct_res cleanup
fn struct_res_cleanup(_: beam.env, resource: *struct_res) void {
  beam.allocator.free(resource.first_name);
  beam.allocator.free(resource.last_name);
  // don't clean up the resource pointer itself.
}

/// resource: struct_ptr_res cleanup
fn struct_res_ptr_cleanup(_: beam.env, resource: *struct_ptr_res) void {
  beam.allocator.free(resource.first_name);
  beam.allocator.free(resource.last_name);
  // DO clean up the pointer in the resource.
  beam.allocator.destroy(resource.*);

  // don't clean up the resource pointer (pointer).
}
"""