# `Edifice.Vision.GaussianSplat` [🔗](https://github.com/blasphemetheus/edifice/blob/main/lib/edifice/vision/gaussian_splat.ex#L1) 3D Gaussian Splatting for real-time radiance field rendering. Represents 3D scenes as collections of Gaussian primitives that can be differentiably rendered from arbitrary viewpoints. Achieves 100x faster rendering than NeRF while maintaining or improving quality. ## Key Insight Instead of querying a neural network per ray sample (NeRF), represent the scene explicitly as a set of 3D Gaussians. Each Gaussian has position, shape (covariance), color (spherical harmonics), and opacity. Rendering is a simple alpha-compositing of projected 2D Gaussians: ``` Scene Representation: +------------------------------------------+ | N Gaussians, each with: | | μ: position [x, y, z] | | Σ: covariance (via R, S) | Σ = R·S·S^T·R^T | c: color (SH coefficients) | 48 values for degree 3 | α: opacity [0, 1] | +------------------------------------------+ Rendering Pipeline: 3D Gaussians → Project to 2D → Sort by depth → Alpha composite | | | | v v v v [μ,Σ,c,α] 2D splats front-to-back final image ``` ## Differentiable Rendering For each pixel, accumulate color from overlapping Gaussians: ``` C(p) = Σᵢ cᵢ · αᵢ · G(p; μ'ᵢ, Σ'ᵢ) · Πⱼ<ᵢ (1 - αⱼ · G(p; μ'ⱼ, Σ'ⱼ)) ``` Where: - G(p; μ', Σ') is the 2D Gaussian evaluated at pixel p - μ', Σ' are the projected mean and covariance - Products are over Gaussians in front (sorted by depth) ## Adaptive Density Control During training, Gaussians are dynamically adjusted: - **Clone**: Split high-gradient small Gaussians - **Split**: Divide large high-gradient Gaussians into two - **Prune**: Remove nearly-transparent Gaussians (α < threshold) ## Usage # Build the Gaussian splatting model model = GaussianSplat.build( num_gaussians: 10000, sh_degree: 3 ) # Initialize from point cloud gaussians = GaussianSplat.initialize_gaussians(point_cloud) # Render a view image = GaussianSplat.render(gaussians, camera, {height, width}) # During training: adaptive density control gaussians = GaussianSplat.densification_step(gaussians, gradients, clone_threshold: 0.0002, split_threshold: 0.0002, prune_threshold: 0.005 ) ## References - "3D Gaussian Splatting for Real-Time Radiance Field Rendering" (Kerbl et al., SIGGRAPH 2023) — https://arxiv.org/abs/2308.04079 # `camera` ```elixir @type camera() :: %{ view_matrix: Nx.Tensor.t(), proj_matrix: Nx.Tensor.t(), position: Nx.Tensor.t() } ``` Camera parameters for rendering. # `gaussians` ```elixir @type gaussians() :: %{ positions: Nx.Tensor.t(), rotations: Nx.Tensor.t(), scales: Nx.Tensor.t(), opacities: Nx.Tensor.t(), sh_coeffs: Nx.Tensor.t() } ``` Gaussian representation. # `splat_opt` ```elixir @type splat_opt() :: {:num_gaussians, pos_integer()} | {:sh_degree, 0..3} | {:name, String.t()} | {:image_size, pos_integer()} ``` Options for GaussianSplat functions. # `build` ```elixir @spec build([splat_opt()]) :: Axon.t() ``` Build an Axon model for 3D Gaussian Splatting scene representation. The model takes viewing parameters and outputs a rendered image. Gaussian parameters are learned during training. ## Options - `:num_gaussians` - Number of Gaussian primitives (default: 10000) - `:sh_degree` - Spherical harmonics degree for color (0-3, default: 3) - `:name` - Layer name prefix (default: "gaussian_splat") ## Inputs - "camera_position" — Camera world position [batch, 3] - "view_matrix" — World-to-camera transform [batch, 4, 4] - "proj_matrix" — Camera-to-clip projection [batch, 4, 4] - "image_size" — Output resolution [batch, 2] (height, width) ## Returns An Axon model that renders the scene from the given viewpoint. # `densification_step` ```elixir @spec densification_step(gaussians(), Nx.Tensor.t(), keyword()) :: gaussians() ``` Perform adaptive density control based on gradient statistics. Implements the clone/split/prune operations from the original paper: - Clone: Duplicate small Gaussians with high positional gradients - Split: Divide large Gaussians with high gradients into two - Prune: Remove Gaussians with low opacity ## Parameters - `gaussians` - Current Gaussian parameters - `gradients` - Position gradients from training [N, 3] - `opts` - Control options: - `:clone_threshold` - Gradient threshold for cloning (default: 0.0002) - `:split_threshold` - Gradient threshold for splitting (default: 0.0002) - `:prune_threshold` - Opacity threshold for pruning (default: 0.005) - `:scale_threshold` - Scale threshold for clone vs split (default: 0.01) ## Returns Updated Gaussian parameters with modified count. # `initialize_gaussians` ```elixir @spec initialize_gaussians( Nx.Tensor.t(), keyword() ) :: gaussians() ``` Initialize Gaussians from a point cloud. ## Parameters - `point_cloud` - Tensor of 3D points [N, 3] or [N, 6] (with colors) - `opts` - Options: - `:sh_degree` - Spherical harmonics degree (default: 3) - `:initial_scale` - Initial Gaussian scale (default: 0.01) ## Returns A map of Gaussian parameters ready for optimization. ## Example points = Nx.tensor([[0, 0, 0], [1, 0, 0], [0, 1, 0]], type: :f32) gaussians = GaussianSplat.initialize_gaussians(points) # `output_size` ```elixir @spec output_size(keyword()) :: {pos_integer(), pos_integer(), pos_integer()} ``` Get output size for rendered images. # `param_count` ```elixir @spec param_count(keyword()) :: pos_integer() ``` Calculate the number of parameters in a Gaussian splatting model. # `project_gaussians` ```elixir @spec project_gaussians(gaussians(), camera()) :: map() ``` Project 3D Gaussians to 2D screen space. Transforms Gaussian means and covariances from world space to screen space using the camera's view and projection matrices. ## Parameters - `gaussians` - Map of Gaussian parameters - `camera` - Camera parameters (view_matrix, proj_matrix) ## Returns Map with projected means, covariances, and depths: - `:means_2d` - Screen-space positions [N, 2] - `:covs_2d` - Screen-space covariances [N, 2, 2] - `:depths` - View-space depths for sorting [N] # `recommended_defaults` ```elixir @spec recommended_defaults() :: keyword() ``` Get recommended defaults for Gaussian splatting. # `render` ```elixir @spec render(gaussians(), camera(), {pos_integer(), pos_integer()}, keyword()) :: Nx.Tensor.t() ``` Render Gaussians to an image via differentiable alpha compositing. ## Parameters - `gaussians` - Map of Gaussian parameters - `camera` - Camera parameters - `image_size` - Output resolution `{height, width}` - `opts` - Rendering options: - `:sh_degree` - SH degree for color evaluation (default: 3) - `:background` - Background color (default: [0, 0, 0]) ## Returns Rendered image tensor of shape `[height, width, 3]`. ## Example image = GaussianSplat.render(gaussians, camera, {256, 256}) --- *Consult [api-reference.md](api-reference.md) for complete listing*