Edifice.Vision.GaussianSplat (Edifice v0.2.0)

Copy Markdown View Source

3D Gaussian Splatting for real-time radiance field rendering.

Represents 3D scenes as collections of Gaussian primitives that can be differentiably rendered from arbitrary viewpoints. Achieves 100x faster rendering than NeRF while maintaining or improving quality.

Key Insight

Instead of querying a neural network per ray sample (NeRF), represent the scene explicitly as a set of 3D Gaussians. Each Gaussian has position, shape (covariance), color (spherical harmonics), and opacity. Rendering is a simple alpha-compositing of projected 2D Gaussians:

Scene Representation:
+------------------------------------------+
| N Gaussians, each with:                  |
|   μ: position [x, y, z]                  |
|   Σ: covariance (via R, S)               |  Σ = R·S·S^T·R^T
|   c: color (SH coefficients)             |  48 values for degree 3
|   α: opacity [0, 1]                      |
+------------------------------------------+

Rendering Pipeline:
3D Gaussians → Project to 2D → Sort by depth → Alpha composite
     |              |              |               |
     v              v              v               v
  [μ,Σ,c,α]    2D splats      front-to-back    final image

Differentiable Rendering

For each pixel, accumulate color from overlapping Gaussians:

C(p) = Σᵢ cᵢ · αᵢ · G(p; μ'ᵢ, Σ'ᵢ) · Πⱼ<ᵢ (1 - αⱼ · G(p; μ'ⱼ, Σ'ⱼ))

Where:

  • G(p; μ', Σ') is the 2D Gaussian evaluated at pixel p
  • μ', Σ' are the projected mean and covariance
  • Products are over Gaussians in front (sorted by depth)

Adaptive Density Control

During training, Gaussians are dynamically adjusted:

  • Clone: Split high-gradient small Gaussians
  • Split: Divide large high-gradient Gaussians into two
  • Prune: Remove nearly-transparent Gaussians (α < threshold)

Usage

# Build the Gaussian splatting model
model = GaussianSplat.build(
  num_gaussians: 10000,
  sh_degree: 3
)

# Initialize from point cloud
gaussians = GaussianSplat.initialize_gaussians(point_cloud)

# Render a view
image = GaussianSplat.render(gaussians, camera, {height, width})

# During training: adaptive density control
gaussians = GaussianSplat.densification_step(gaussians, gradients,
  clone_threshold: 0.0002,
  split_threshold: 0.0002,
  prune_threshold: 0.005
)

References

Summary

Types

camera()

Camera parameters for rendering.

gaussians()

Gaussian representation.

splat_opt()

Options for GaussianSplat functions.

Functions

build(opts \\ [])

Build an Axon model for 3D Gaussian Splatting scene representation.

densification_step(gaussians, gradients, opts \\ [])

Perform adaptive density control based on gradient statistics.

initialize_gaussians(point_cloud, opts \\ [])

Initialize Gaussians from a point cloud.

output_size(opts \\ [])

Get output size for rendered images.

param_count(opts \\ [])

Calculate the number of parameters in a Gaussian splatting model.

project_gaussians(gaussians, camera)

Project 3D Gaussians to 2D screen space.

recommended_defaults()

Get recommended defaults for Gaussian splatting.

render(gaussians, camera, image_size, opts \\ [])

Render Gaussians to an image via differentiable alpha compositing.

Types

camera()

@type camera() :: %{
  view_matrix: Nx.Tensor.t(),
  proj_matrix: Nx.Tensor.t(),
  position: Nx.Tensor.t()
}

Camera parameters for rendering.

gaussians()

@type gaussians() :: %{
  positions: Nx.Tensor.t(),
  rotations: Nx.Tensor.t(),
  scales: Nx.Tensor.t(),
  opacities: Nx.Tensor.t(),
  sh_coeffs: Nx.Tensor.t()
}

Gaussian representation.

splat_opt()

@type splat_opt() ::
  {:num_gaussians, pos_integer()}
  | {:sh_degree, 0..3}
  | {:name, String.t()}
  | {:image_size, pos_integer()}

Options for GaussianSplat functions.

Functions

build(opts \\ [])

@spec build([splat_opt()]) :: Axon.t()

Build an Axon model for 3D Gaussian Splatting scene representation.

The model takes viewing parameters and outputs a rendered image. Gaussian parameters are learned during training.

Options

  • :num_gaussians - Number of Gaussian primitives (default: 10000)
  • :sh_degree - Spherical harmonics degree for color (0-3, default: 3)
  • :name - Layer name prefix (default: "gaussian_splat")

Inputs

  • "camera_position" — Camera world position [batch, 3]
  • "view_matrix" — World-to-camera transform [batch, 4, 4]
  • "proj_matrix" — Camera-to-clip projection [batch, 4, 4]
  • "image_size" — Output resolution [batch, 2] (height, width)

Returns

An Axon model that renders the scene from the given viewpoint.

densification_step(gaussians, gradients, opts \\ [])

@spec densification_step(gaussians(), Nx.Tensor.t(), keyword()) :: gaussians()

Perform adaptive density control based on gradient statistics.

Implements the clone/split/prune operations from the original paper:

  • Clone: Duplicate small Gaussians with high positional gradients
  • Split: Divide large Gaussians with high gradients into two
  • Prune: Remove Gaussians with low opacity

Parameters

  • gaussians - Current Gaussian parameters
  • gradients - Position gradients from training [N, 3]
  • opts - Control options:
    • :clone_threshold - Gradient threshold for cloning (default: 0.0002)
    • :split_threshold - Gradient threshold for splitting (default: 0.0002)
    • :prune_threshold - Opacity threshold for pruning (default: 0.005)
    • :scale_threshold - Scale threshold for clone vs split (default: 0.01)

Returns

Updated Gaussian parameters with modified count.

initialize_gaussians(point_cloud, opts \\ [])

@spec initialize_gaussians(
  Nx.Tensor.t(),
  keyword()
) :: gaussians()

Initialize Gaussians from a point cloud.

Parameters

  • point_cloud - Tensor of 3D points [N, 3] or [N, 6] (with colors)
  • opts - Options:
    • :sh_degree - Spherical harmonics degree (default: 3)
    • :initial_scale - Initial Gaussian scale (default: 0.01)

Returns

A map of Gaussian parameters ready for optimization.

Example

points = Nx.tensor([[0, 0, 0], [1, 0, 0], [0, 1, 0]], type: :f32)
gaussians = GaussianSplat.initialize_gaussians(points)

output_size(opts \\ [])

@spec output_size(keyword()) :: {pos_integer(), pos_integer(), pos_integer()}

Get output size for rendered images.

param_count(opts \\ [])

@spec param_count(keyword()) :: pos_integer()

Calculate the number of parameters in a Gaussian splatting model.

project_gaussians(gaussians, camera)

@spec project_gaussians(gaussians(), camera()) :: map()

Project 3D Gaussians to 2D screen space.

Transforms Gaussian means and covariances from world space to screen space using the camera's view and projection matrices.

Parameters

  • gaussians - Map of Gaussian parameters
  • camera - Camera parameters (view_matrix, proj_matrix)

Returns

Map with projected means, covariances, and depths:

  • :means_2d - Screen-space positions [N, 2]
  • :covs_2d - Screen-space covariances [N, 2, 2]
  • :depths - View-space depths for sorting [N]

render(gaussians, camera, image_size, opts \\ [])

@spec render(gaussians(), camera(), {pos_integer(), pos_integer()}, keyword()) ::
  Nx.Tensor.t()

Render Gaussians to an image via differentiable alpha compositing.

Parameters

  • gaussians - Map of Gaussian parameters
  • camera - Camera parameters
  • image_size - Output resolution {height, width}
  • opts - Rendering options:
    • :sh_degree - SH degree for color evaluation (default: 3)
    • :background - Background color (default: [0, 0, 0])

Returns

Rendered image tensor of shape [height, width, 3].

Example

image = GaussianSplat.render(gaussians, camera, {256, 256})