View Source Trunk behaviour (trunk v1.5.1)
A Trunk
is a module used to handle the transformation and storage of files with multiple versions.
Usage:
To generate a Trunk
call use Trunk
and supply options to configure the specific behaviour you need.
defmodule MyTrunk do
use Trunk, versions: [:original],
storage: Trunk.Storage.Filesystem,
storage_opts: [path: "/tmp"]
end
Options
Options can be set at multiple levels
- Global, use
config :trunk, key: :value
- Per otp app, use
config :my_app, trunk: [key: :value]
- Per module, supply options to
use Trunk
. Useuse Trunk, otp_app: :my_app
to read in per otp app options. - Per function, you can supply additional or override options to every function call.
Available options
:versions
a list of versions as atoms,[:original, :thumb]
:async
(boolean) defaulttrue
, whether to process each version in parallel or in sequence.:timeout
default5_000
, how long to wait for each versions transformation and storage to complete when processing in parallel. Note: if storing files non-locally (i.e. Amazon S3), then include both expected time for transformation and expected time for storage in this timeout.:storage
defaultTrunk.Storage.Filesystem
, the storage module to use when storing files and versions:storage_opts
default:[path: ""]
, the options for the storage module. See each storage module's documentation for available options.:otp_app
, only used at module level to read options specific to the otp app.
Additional options
Any extra options passed in at the otp app, module or function level are made available in the opts
field of Trunk.State
. This means that you can pass in any options you might need to use in any of the processing steps.
File storage
When files are stored, they are passed through a transformation pipeline which allows you to generate different versions of a file. You might get a photo uploaded which you want to save along with a thumbnail. You might have a video which also needs a thumbnail extracted. You could convert a spreadsheet to CSV or a document to PDF.
At each point in the pipeline you have access to a Trunk.State
or Trunk.VersionState
struct which contains information collected at each step. Each state struct has an assigns map (similar to Plug.Conn
) in which you can store information to be used later in the pipeline process or once storage is complete.
The storage pipeline goes through the following steps:
preprocess/1
- Here you have access to the information about the file including the path to the file on disk. This is where you can validate the file or extract information that can be useful for later processing. This callback is called only once for the pipeline.transform/2
- This is where you determine how each version should be transformed. You can return a transformation instruction or a function that will do the actual transformation. This callback is called once per version.postprocess/3
- Here you have access to information about the version after its transformation. At this point you have access to the state about the version including the path to the temporary file on disk. You can extract information about the version file like file size or hash its contents. This callback is called once per version.storage_dir/2
- This is where you determine at which path the version should be saved. This callback is called once per version.filename/2
- This is where you determine which filename the version should be saved as. This callback is called once per version.storage_opts/2
- This allows you to set additional storage options like file permissions:acl
or add things like S3 headers (Check with the storage module as to what options are avialble to be set here). This callback is called once per version.- The file is saved using the configured storage module.
Example:
defmodule MyTrunk do
use Trunk, versions: [:original, :thumb],
storage: Trunk.Storage.Filesystem,
storage_opts: [path: "/tmp"]
# Ensure the file is a JPG or PNG file
def preprocess(%Trunk.State{} = state) do
if String.downcase(extname) in [".png", ".jpg", ".jpeg"] do
{:ok, state}
else
{:error, "Invalid file"}
end
end
# Do not transform the original file
def transform(%Trunk.State{}, :original), do: nil
# Resize the file to a thumbnail of maximum 200x200px
def transform(%Trunk.State{}, :thumb),
do: {:convert, "-strip -thumbnail 200x200> -limit area 10MB -limit disk 100MB"}
# Store the file size of each version
def postprocess(%Trunk.VersionState{temp_path: temp_path} = version_state, _version, %Trunk.State{} = _state) do
%File.State{size: file_size} = File.stat(temp_path)
{:ok, Trunk.VersionState.assign(version_state, :file_size, file_size)}
end
# Store all versions in a directory based on a model id
def storage_dir(%Trunk.State{scope: %{id: model_id}}, _version),
do: to_string(model_id)
# Store the original file with its original filename
def filename(%Trunk.State{filename: filename}, :original),
do: filename
# Store other versions with the version in its filename
def filename(%Trunk.State{rootname: rootname, extname: extname}, version),
do: "#{rootname}_#{version}#{extname}"
def storage_opts(%Trunk.State{}, :original),
do: [acl: 0o600]
def storage_opts(%Trunk.State{}, :thumb),
do: [acl: 0o644]
end
> {:ok, %Trunk.State{filename: filename, versions: versions}} = MyTrunk.store("/path/to/photo.jpg")
> filename
"photo.jpg"
> versions |> Enum.map(fn({version, %Trunk.VersionState{assigns: %{file_size: file_size}}}) -> {version, file_size} end)
[original: 34567, thumb: 456]
Summary
Callbacks
Copies the stored file(s) from the storage_dir/2
and filename/2
of the original scope to the same of the new scope. This function is generated by use Trunk
Deletes the identified file and all its versions. This function is generated by use Trunk
Deletes the identified file and all its versions. This function is generated by use Trunk
Deletes the identified file and all its versions. This function is generated by use Trunk
A callback that should be used to generate the filename specific to each version.
Return nil
to prevent storing a specific version
A callback that can be used to do additional processing on each version file before it gets saved.
A callback that can be used to do any preprocessing on the state before transformation and storage begins.
Reprocesses the original file for all versions. This function is generated by use Trunk
Reprocesses the original file for all or the specified version(s). This function is generated by use Trunk
Reprocesses the original file for all or the specified version(s). This function is generated by use Trunk
Reprocesses the original file for the specified version(s). This function is generated by use Trunk
Retrieves the original for the identified file. This function is generated by use Trunk
Retrieves the original, or specified version, for the identified file. This function is generated by use Trunk
Retrieves the original, or specified version, for the identified file. This function is generated by use Trunk
Retrieves the version for the identified file. This function is generated by use Trunk
A callback that should be used to determine the storage directory specific to each version.
A callback that should be used to set version specific storage options.
Stores the supplied file. This function is generated by use Trunk
Stores the supplied file. This function is generated by use Trunk
Stores the supplied file after running it through the processing pipeline. This function is generated by use Trunk
A callback that can be used to generate a transform specific to each version
Generates a URL for the given information. This function is generated by use Trunk
Will return nil if the filename/2
callback returns nil
Generates a URL for the given information. This function is generated by use Trunk
Will return nil if the filename/2
callback returns nil
Generates a URL for the given information. This function is generated by use Trunk
Will return nil if the filename/2
callback returns nil
Generates a URL for the given information. This function is generated by use Trunk
Will return nil if the filename/2
callback returns nil
Functions
A convenience macro for generating a preprocess/1
function that validates the file being saved against a list of approved extensions (case-insensitive).
Types
Callbacks
@callback copy(file_info(), scope(), new_scope(), opts()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Copies the stored file(s) from the storage_dir/2
and filename/2
of the original scope to the same of the new scope. This function is generated by use Trunk
It returns an {:ok, %Trunk.State{}}
tuple of the new state on success and {:error, %Trunk.State{}}
tuple of the new state if an error occurred anywhere in the processing pipeline.
file_info
- The file name or base info needed to identify the file. Minimum is%{filename: "file.ext"}
scope
- a map or struct that will help when generating the filename and storage directory for retrieving the file(s)new_scope
- a map or struct that will help when generating the filename and storage directory for saving the file(s)opts
- (optional) options to override module, app, or global options. See "Options" in the module documentations for all options.
@callback delete(file_info()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Deletes the identified file and all its versions. This function is generated by use Trunk
Calls delete/3
with delete(file_info, nil, [])
@callback delete(file_info(), scope() | opts()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Deletes the identified file and all its versions. This function is generated by use Trunk
@callback delete(file_info(), scope(), opts()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Deletes the identified file and all its versions. This function is generated by use Trunk
It returns an {:ok, %Trunk.State{}}
tuple on success and {:error, %Trunk.State{}}
tuple if an error occurred.
file_info
- The file name or base info needed to identify the file. Minimum is%{filename: "file.ext"}
scope
- (optional) a map or struct that will help when generating the filename and storage directory for saving the fileopts
- (optional) options to override module, app, or global options. See "Options" in the module documentations for all options.
@callback filename(state :: Trunk.State.t(), version()) :: String.t() | nil
A callback that should be used to generate the filename specific to each version.
Return nil
to prevent storing a specific version
state
- The trunk stateversion
- An atom representing the version
Example:
# For the :original version return the original file name
def filename(%{filename: filename}, :original), do: filename
# For any other version append the version to the root name
# mypic.jpg with :thumb version becomes mypic_thumb.jpg
def filename(%{rootname: rootname, extname: extname}, version),
do: "#{rootname}_#{version}#{extname}"
@callback postprocess( version_state :: Trunk.VersionState.t(), version(), state :: Trunk.State.t() ) :: {:ok, Trunk.VersionState.t()} | {:error, any()}
A callback that can be used to do additional processing on each version file before it gets saved.
This callback can update the version state information which can then be used in storage_dir/2
and filename/2
as additional information for storage.
Note: If information is stored in the version state and used for file naming, then that information needs to be stored in order to retrieve that file or generate a URL etc.
Example:
To generate and store the md5 hash of each version.
def postprocess(%Trunk.VersionState{temp_path: temp_path} = version_state, _version, _state) do
hash = :crypto.hash(:md5, File.read!(temp_path)) |> Base.encode16(case: :lower)
{:ok, Trunk.VersionState.assign(version_state, :hash, hash)}
end
Example:
To calculate and store the file size of each version.
def postprocess(%Trunk.VersionState{temp_path: temp_path} = version_state, _version, _state) do
{:ok, %File.Stat{size: file_size}} = File.stat(temp_path)
{:ok, Trunk.VersionState.assign(version_state, :file_size, file_size)}
end
@callback preprocess(state :: Trunk.State.t()) :: {:ok, Trunk.State.t()} | {:error, any()}
A callback that can be used to do any preprocessing on the state before transformation and storage begins.
This is a good place to do file validation (Note this example is wrapped in a convenience macro validate_file_extensions/1
.
Example
def preprocess(%Trunk.State{lower_extname: extname} = state) do
if extname in [".jpg", ".png"] do
{:ok, state}
else
{:error, "Invalid file"}
end
end
This is also a place to do any processing that might be needed when transforming versions.
@callback reprocess(file_info()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Reprocesses the original file for all versions. This function is generated by use Trunk
Calls reprocess/4
with reprocess(file_info, nil, :all, [])
@callback reprocess(file_info(), scope() | opts() | version() | versions()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Reprocesses the original file for all or the specified version(s). This function is generated by use Trunk
- With a
scope
, callsreprocess/4
withreprocess(file_info, scope, :all, [])
- With
opts
, callsreprocess/4
withreprocess(file_info, nil, :all, opts)
- With
version
, callsreprocess/4
withreprocess(file_info, nil, version, [])
@callback reprocess( file_info(), scope() | version() | versions(), opts() | version() | versions() ) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Reprocesses the original file for all or the specified version(s). This function is generated by use Trunk
- With version(s) and opts, calls
reprocess/4
withreprocess(file_info, nil, version, opts)
- With a scope and version(s), calls
reprocess/4
withreprocess(file_info, scope, version, [])
- With a scope and opts, calls
reprocess/4
withreprocess(file_info, scope, :all, opts)
@callback reprocess(file_info(), scope(), version() | versions(), opts()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Reprocesses the original file for the specified version(s). This function is generated by use Trunk
It returns an {:ok, path}
tuple on success and {:error, error}
tuple if an error occurred.
file_info
- The file name or base info needed to identify the file. Minimum is%{filename: "file.ext"}
scope
- (optional) a map or struct that will help when generating the filename and storage directory for saving the fileversion
- (optional) an atom representing the versionopts
- (optional) options to override module, app, or global options. See "Options" in the module documentations for all options.
@callback retrieve(file_info()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Retrieves the original for the identified file. This function is generated by use Trunk
Calls retrieve/4
with retrieve(file_info, nil, :original, [])
@callback retrieve(file_info(), scope() | opts() | version()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Retrieves the original, or specified version, for the identified file. This function is generated by use Trunk
- With a
scope
, callsretrieve/4
withretrieve(file_info, scope, :original, [])
- With
opts
, callsretrieve/4
withretrieve(file_info, nil, :original, opts)
- With
version
, callsretrieve/4
withretrieve(file_info, nil, version, [])
@callback retrieve(file_info(), scope() | version(), opts() | version()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Retrieves the original, or specified version, for the identified file. This function is generated by use Trunk
- With a version and opts, calls
retrieve/4
withretrieve(file_info, nil, version, opts)
- With a scope and version, calls
retrieve/4
withretrieve(file_info, scope, version, [])
- With a scope and opts, calls
retrieve/4
withretrieve(file_info, scope, :original, opts)
@callback retrieve(file_info(), scope(), version(), opts()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Retrieves the version for the identified file. This function is generated by use Trunk
It returns an {:ok, path}
tuple on success and {:error, error}
tuple if an error occurred.
file_info
- The file name or base info needed to identify the file. Minimum is%{filename: "file.ext"}
scope
- (optional) a map or struct that will help when generating the filename and storage directory for saving the fileversion
- (optional) an atom representing the versionopts
- (optional) options to override module, app, or global options. See "Options" in the module documentations for all options.Additional options: - `:output_path` a specific path (including filename) to save the file to. Defaults to a temporary file path
@callback storage_dir(state :: Trunk.State.t(), version()) :: String.t()
A callback that should be used to determine the storage directory specific to each version.
Each storage system will have a base path in which to store files. The result of storage_dir/2
is then appended to this to determine where to save files.
state
- The trunk stateversion
- An atom representing the version
Example:
# Assuming a base path of /tmp files will be saved in /tmp
def storage_dir(_state, _version), do: ""
# Place the file in a directory named after a model id
# With a base path of /tmp and a scope of %{id: 42}, the file will be saved in /tmp/42/
def storage_dir(%Trunk.State{scope: %{id: model_id}}, _version),
do: to_string(model_id)
@callback storage_opts(state :: Trunk.State.t(), version()) :: storage_opts()
A callback that should be used to set version specific storage options.
This is the place to set access permissions, storage headers (s3) and other options provided by the storage module.
This callback should return a list of options.
Example:
# With Filesystem
def storage_opts(_state, _version), do: [acl: "0600"]
# With S3
def storage_opts(_state, _version), do: [acl: :public_read]
@callback store(file()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Stores the supplied file. This function is generated by use Trunk
Calls store/3
with store(file, nil, [])
@callback store(file(), opts()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Stores the supplied file. This function is generated by use Trunk
Calls store/3
with store(file, nil, opts)
@callback store(file(), scope(), opts()) :: {:ok, Trunk.State.t()} | {:error, Trunk.State.t()}
Stores the supplied file after running it through the processing pipeline. This function is generated by use Trunk
It returns an {:ok, %Trunk.State{}}
tuple on success and {:error, %Trunk.State{}}
tuple if an error occurred anywhere in the processing pipeline.
file
- the file to store.- This can be a full path to file, or
- a url to a file, or
- a map with
:filename
and:path
keys (e.g.%Plug.Upload{}
), or - a map with
:filename
and:binary
keys (e.g.%{filename: "myfile.jpg", binary: <<…>>}
)
scope
- (optional) a map or struct that will help when generating the filename and storage directory for saving the fileopts
- (optional) options to override module, app, or global options. See "Options" in the module documentations for all options.
@callback transform(state :: Trunk.State.t(), version()) :: nil | {command(), args()} | {command(), args(), ext()} | transform_func()
A callback that can be used to generate a transform specific to each version
The transformation instruction is a two element tuple {command, arguments}
, a three element tuple {command, arguments, extension}
, or a function that accepts a single argument (the source file path) and returns either {:ok, "/path/to/transformed/file"}
or {:error, "Reason"}
Example:
To generate a thumnail, you might have a version named :thumb
.
A transform that runs the uploaded file through ImageMagick’s convert
could be the following
def transform(%Trunk.State{}, :thumb),
do: {:convert, "-strip -thumbnail 200x200>"}
This would generate a call to System.cmd
as follows:
# input_file is the source (uploaded) file
# output_file is a temporary file with the same extension as the source file
System.cmd "convert", [input_file, "-strip", "-thumbnail", "200x200>", output_file], stderr_to_stdout: true
To generate a JPEG thumbnail whether the source is JPEG or PNG you would add the extension to the transform instruction.
def transform(_state, :thumb),
do: {:convert, "-strip -thumbnail 200x200>", :jpg}
In this case the temporary file generated for the transformation would have a .jpg extension.
In either form, the arguments can be generated by a function which receives the input and output file paths. This is useful for transforms where the input and output paths are not the first and last respectively.
def transform(_state, _version),
do: {:convert, fn(input, output) -> ["-density", "300", input, "-flatten", "-strip", "-thumbnail", "200x200>", output] end, :jpg}
Note: Returning an instruction with an extension will change the extension of the temporary file used in the transformation but will not affect the filename during save. That still needs to be done in filename/2
Finally a function can be returned which will be wholly responsible for doing the transformation and returning the location of the transformed file.
The function should return {:ok, file_path}
if successful, or {:error, reason}
if not.
This example uses soffice
to transform an XLS document to CSV. The problem is soffice
doesn’t allow the output file to be specified so the above {:soffice, [args]}
style can’t be used. To solve this, a transformation function is used to handle the transformation.
The function creates a temporary directory for the transformation, runs the soffice
command and then parses the output for the resulting file.
def transform(%Trunk.State{extname: extname}, :csv) when extname in [".xls", ".xlsx"],
do: fn(source_path) ->
{:ok, directory} = Briefly.create(directory: true)
# soffice must be in the PATH
case System.cmd("soffice", ["--headless", "--convert-to", "csv:Text - txt - csv (StarCalc):44,34,76,0", "--outdir", directory, source_path], stderr_to_stdout: true) do
{output, 0} ->
[_, path] = Regex.run(~r/-> (.+\.csv)/, output)
{:ok, path}
{result, _} -> {:error, result}
end
end
def transform(_state, _version), do: nil
Generates a URL for the given information. This function is generated by use Trunk
Will return nil if the filename/2
callback returns nil
Calls url/4
with url(file_info, nil, :original, [])
Generates a URL for the given information. This function is generated by use Trunk
Will return nil if the filename/2
callback returns nil
Generates a URL for the given information. This function is generated by use Trunk
Will return nil if the filename/2
callback returns nil
Generates a URL for the given information. This function is generated by use Trunk
Will return nil if the filename/2
callback returns nil
file_info
- The file name or base info needed to identify the file. Minimum is%{filename: "file.ext"}
scope
- The scope objectversion
- The file version to which the URL must pointopts
- Override options.
Functions
A convenience macro for generating a preprocess/1
function that validates the file being saved against a list of approved extensions (case-insensitive).
Example:
defmodule MyTrunk do
use Trunk, versions: [:original]
validate_file_extensions ~w[.jpg .jpeg .png]
end