TensorFrame is a representation of Explorer.DataFrame that is designed to work inside Nx's defn expressions.

For example, imagine the following defn:

defn add_columns(tf) do
  tf[:a] + tf[:b]
end

We can now pass a DataFrame as argument:

iex> add_columns(Explorer.DataFrame.new(a: [11, 12], b: [21, 22]))
#Nx.Tensor<
  s64[2]
  [32, 34]
>

Passing an Explorer.DataFrame to a defn will automatically convert it to a TensorFrame. The TensorFrame will lazily build tensors out of the used dataframe fields.

Stack and concatenating

Due to the integration with Nx, you can also pass dataframes into Nx.stack/2 and Nx.concatenate and they will be automatically converted to tensors. This makes it easy to pass dataframes into neural networks and other computationally intensive algorithms:

iex> Nx.concatenate(Explorer.DataFrame.new(a: [11, 12], b: [21, 22]))
#Nx.Tensor<
  s64[4]
  [11, 12, 21, 22]
>

iex> Nx.stack(Explorer.DataFrame.new(a: [11, 12], b: [21, 22]))
#Nx.Tensor<
  s64[2][2]
  [
    [11, 12],
    [21, 22]
  ]
>

iex> Nx.stack(Explorer.DataFrame.new(a: [11, 12], b: [21, 22]), axis: -1)
#Nx.Tensor<
  s64[2][2]
  [
    [11, 21],
    [12, 22]
  ]
>

Warning: returning TensorFrames

It is not recommended to return a TensorFrame from a defn, as that would force all columns to be sent to the CPU/GPU and then copied back. Return only the columns that have been modified during the computation. For example, in the example above we used Nx to add two columns, if you want to put the result of the computation back into a DataFrame, you can use Explorer.DataFrame.put/4, which also accepts tensors:

iex> df = Explorer.DataFrame.new(a: [11, 12], b: [21, 22])
iex> Explorer.DataFrame.put(df, "result", add_columns(df))
#Explorer.DataFrame<
  Polars[2 x 3]
  a s64 [11, 12]
  b s64 [21, 22]
  result s64 [32, 34]
>

One benefit of using Explorer.DataFrame.put/4 is that it will preserve the type of the column if one already exists. Alternatively, use Explorer.Series.from_tensor/1 to explicitly convert a tensor back to a series.

Supported dtypes

The following dtypes can be converted to tensors:

• :integer
• {:f, 32}
• {:f, 64}
• :boolean
• :date
• {:naive_datetime, :millisecond}
• {:naive_datetime, :microsecond}
• {:naive_datetime, :nanosecond}

See Explorer.Series.to_iovec/1 and Explorer.Series.to_tensor/1 to learn more about their internal representation.