Temporal Convolutional Network (TCN) for sequence modeling.
TCNs use dilated causal convolutions to achieve large receptive fields efficiently. Each layer doubles the dilation rate (1, 2, 4, 8, ...), allowing the receptive field to grow exponentially with depth while maintaining linear parameter growth.
Key properties:
- Causal: output at time t depends only on inputs at times <= t
- Flexible receptive field: receptive field = num_layers kernel_size max_dilation
- Parallelizable: unlike RNNs, all timesteps can be computed in parallel
- Residual connections: each temporal block has a skip connection
Architecture
Input [batch, seq_len, features]
|
+-----v-----------+
| Temporal Block 1 | dilation = 1
| dilated conv |
| -> BN -> act |
| -> dilated conv |
| -> BN -> act |
| + residual skip |
+-----------------+
|
+-----v-----------+
| Temporal Block 2 | dilation = 2
| (same pattern) |
+-----------------+
|
+-----v-----------+
| Temporal Block 3 | dilation = 4
| (same pattern) |
+-----------------+
|
v
Output [batch, seq_len, channels]Receptive Field
For n layers with kernel size k:
receptive_field = 1 + 2 * (k - 1) * (2^n - 1)Examples:
- 4 layers, k=3: receptive field = 31
- 6 layers, k=3: receptive field = 127
- 8 layers, k=3: receptive field = 511
- 4 layers, k=7: receptive field = 181
Usage
# TCN for sequence classification
model = TCN.build(
input_size: 64,
channels: [128, 128, 128, 128],
kernel_size: 3,
dropout: 0.1
)
# Calculate required layers for a target receptive field
layers = TCN.layers_for_receptive_field(256, kernel_size: 3)Summary
Functions
Build a Temporal Convolutional Network.
Calculate the minimum number of layers needed for a target receptive field.
Get the output channel count for a TCN model.
Calculate the receptive field for a TCN configuration.
Build a single TCN temporal block with dilated causal convolution and residual.
Types
@type build_opt() :: {:activation, atom()} | {:channels, pos_integer()} | {:dropout, float()} | {:input_size, pos_integer()} | {:kernel_size, pos_integer()} | {:seq_len, pos_integer()}
Options for build/1.
Functions
Build a Temporal Convolutional Network.
Options
:input_size- Number of input features per timestep (required):channels- List of channel counts per temporal block (default: [64, 64, 64, 64]):kernel_size- Convolution kernel size, must be odd for symmetric padding (default: 3):dropout- Dropout rate (default: 0.1):seq_len- Expected sequence length, or nil for dynamic (default: nil):activation- Activation function (default: :relu)
Returns
An Axon model taking [batch, seq_len, input_size] and outputting
[batch, seq_len, last_channel_count].
@spec layers_for_receptive_field( pos_integer(), keyword() ) :: pos_integer()
Calculate the minimum number of layers needed for a target receptive field.
Parameters
target- Desired receptive field in timesteps
Options
:kernel_size- Convolution kernel size (default: 3)
Returns
Number of layers required.
@spec output_size(keyword()) :: pos_integer()
Get the output channel count for a TCN model.
@spec receptive_field(keyword()) :: pos_integer()
Calculate the receptive field for a TCN configuration.
Options
:num_layers- Number of temporal blocks (default: 4):kernel_size- Convolution kernel size (default: 3)
Returns
The receptive field size in timesteps.
@spec temporal_block(Axon.t(), pos_integer(), keyword()) :: Axon.t()
Build a single TCN temporal block with dilated causal convolution and residual.
Each temporal block contains two dilated causal convolution layers with batch normalization, activation, and dropout, plus a residual skip connection.
Parameters
input- Input Axon node[batch, seq_len, channels]out_channels- Number of output channels
Options
:kernel_size- Convolution kernel size (default: 3):dilation- Dilation rate for this block (default: 1):dropout- Dropout rate (default: 0.1):activation- Activation function (default: :relu):name- Layer name prefix (default: "tcn_block")
Returns
An Axon node with shape [batch, seq_len, out_channels].