sftp_toolkit

v1.0.0

sftp_toolkit v1.0.0 SFTPToolkit.Recursive

Module containing functions that allow to do recursive operations on the directories.

Link to this section Summary

Functions

del_dir_recursive(sftp_channel_pid, path, options \\ [])

Recursively deletes a given directory over existing SFTP channel

list_dir_recursive(sftp_channel_pid, path \\ "", options \\ [])

Recursively lists files in a given directory over existing SFTP channel

make_dir_recursive(sftp_channel_pid, path, options \\ [])

Recursively creates a directory over existing SFTP channel

Link to this section Functions

Link to this function

del_dir_recursive(sftp_channel_pid, path, options \\ []) 
del_dir_recursive(pid(), Path.t(), [{:operation_timeout, timeout()}]) ::
  {:ok, [] | [Path.t() | {Path.t(), :file.file_info()}]} | {:error, any()}

Recursively deletes a given directory over existing SFTP channel.

Arguments

Expects the following arguments:

  • sftp_channel_pid - PID of already opened SFTP channel,
  • path - path to delete,
  • options - additional options, see below.

Options

  • operation_timeout - SFTP operation timeout (it is a timeout per each SFTP operation, not total timeout), defaults to 5000 ms.

Limitations

It will ignore symbolic links. They will not be followed.

Return values

On success returns :ok.

On error returns {:error, reason}, where reason might be one of the following:

  • {:invalid_type, path, type} - given path is not a directory and it's actual type is specified as type,
  • {:invalid_access, path, access} - given path is a directory, but it's access is is invalid and it's actual access mode is specified as access,
  • {:delete, path, info} - failed to delete file at path,
  • {:del_dir, path, info} - failed to delete directory at path,
  • {:list_dir, path, info} - :ssh_sftp.list_dir/3 failed and info contains the underlying error returned from it,
  • {:file_info, path, info} - :ssh_sftp.read_file_info/3 failed and info contains the underlying error returned from it.

Notes

Timeouts

It was observed in the wild that underlying :ssh_sftp.list_dir/3 and :ssh_sftp.read_file_info/3 always returned {:error, :timeout} with some servers when SFTP version being used was greater than 3, at least with Elixir 1.7.4 and Erlang 21.0. If you encounter such issues try passing {:sftp_vsn, 3} option while creating a SFTP channel.

Link to this function

list_dir_recursive(sftp_channel_pid, path \\ "", options \\ []) 
list_dir_recursive(pid(), Path.t(),
  operation_timeout: timeout(),
  result_format: :path | :file_info,
  included_types: [
    :device | :directory | :other | :regular | :symlink | :undefined
  ],
  recurse_callback: nil | (Path.t() -> :skip | :skip_but_include | :ok),
  iterate_callback: nil | (Path.t() -> :skip | :skip_but_include | :ok)
) :: {:ok, [] | [Path.t() | {Path.t(), :file.file_info()}]} | {:error, any()}

Recursively lists files in a given directory over existing SFTP channel.

Arguments

Expects the following arguments:

  • sftp_channel_pid - PID of already opened SFTP channel,
  • path - path to list, defaults to empty string, which will map into SFTP server's default directory,
  • options - additional options, see below.

Options

  • operation_timeout - SFTP operation timeout (it is a timeout per each SFTP operation, not total timeout), defaults to 5000 ms,

  • included_types - which file types should be included in the result, defaults to [:regular]. See the :file.file_info typespec for list of all valid values,

  • result_format - can be one of :path or :file_info.

    If you pass :path, the result will be a list of strings containing file names.

    If you pass :file_info, the result will be a list of {path, file_info} tuples, where file_info will have have the same format as :file.file_info. Please note that if you return :skip_but_include from the iterate_callback the file_info will be :undefined.

    Defaults to :path.

  • recurse_callback - optional function that will be called before recursing to the each subdirectory that is found. It will get one argument that is a path currently being evaluated and should return one of :skip, :skip_but_include or :ok.

    If it will return :skip, the whole tree, including the path passed as an argument to the function will be skipped and they won't be included in the final result.

    If it will return :skip_but_include, the underlying tree, except the path passed as an argument to the function will be skipped won't be included in the final result but the path itself will, as long as it's type is within included_types.

    If it will return :ok, it will recurse, and this is also the default behaviour if function is not passed.

  • iterate_callback - optional function that will be called before evaluating each file that is found whether it is a directory. It will get one argument that is a path currently being evaluated and should return one of :skip, :skip_but_include or :ok.

    If it will return :skip, the file will not be evaluated for its type and it will not be included in the final result.

    If it will return :skip_but_include, the file will not be evaluated for its type but it will be always included in the final result.

    If it will return :ok, it will evaluate file's type and try recurse if it's directory, and this is also the default behaviour if function is not passed.

The recurse_callback and iterate_callback options are useful if you traverse a large tree and you can determine that only certain parts of it are meaningful solely from the paths or file names. For example if your directories are created programatically, and you know that files with the .pdf extension are always regular files and by no means they are directories you can instruct this function that it's pointless to read their file information. Thanks to this you can limit amount of calls to :ssh_sftp.read_file_info/3 just by checking if given path has an appropriate suffix and returning the appropriate value.

Limitations

It will ignore symbolic links. They will not be followed.

It will ignore directories without proper access and recurse only to these that provide at least read access.

Return values

On success returns {:ok, list_of_files}.

On error returns {:error, reason}, where reason might be one of the following:

  • {:invalid_type, path, type} - given path is not a directory and it's actual type is specified as type,
  • {:invalid_access, path, access} - given path is a directory, but it's access is is invalid and it's actual access mode is specified as access.
  • {:list_dir, path, info} - :ssh_sftp.list_dir/3 failed and info contains the underlying error returned from it,
  • {:file_info, path, info} - :ssh_sftp.read_file_info/3 failed and info contains the underlying error returned from it.

Notes

Timeouts

It was observed in the wild that underlying :ssh_sftp.list_dir/3 and :ssh_sftp.read_file_info/3 always returned {:error, :timeout} with some servers when SFTP version being used was greater than 3, at least with Elixir 1.7.4 and Erlang 21.0. If you encounter such issues try passing {:sftp_vsn, 3} option while creating a SFTP channel.

Link to this function

make_dir_recursive(sftp_channel_pid, path, options \\ []) 
make_dir_recursive(pid(), Path.t(), [{:operation_timeout, timeout()}]) ::
  :ok | {:error, any()}

Recursively creates a directory over existing SFTP channel.

Arguments

Expects the following arguments:

  • sftp_channel_pid - PID of the already opened SFTP channel,
  • path - path to create,
  • options - additional options, see below.

Options

  • operation_timeout - SFTP operation timeout (it is a timeout per each SFTP operation, not total timeout), defaults to 5000 ms.

Limitations

This function will not follow symbolic links. If it is going to encounter a symbolic link while evaluating existing path components, even if it points to a directory, it will return an error.

Return values

On success returns :ok.

On error returns {:error, reason}, where reason might be one of the following:

  • {:invalid_path, path} - given path is invalid,
  • {:make_dir, info} - :ssh_sftp.make_dir/3 failed and info contains the underlying error returned from it,
  • {:file_info, path, info} - :ssh_sftp.read_file_info/3 failed and info contains the underlying error returned from it,
  • {:invalid_type, path, type} - one of the components of the path to create, specified as path is not a directory, and it's actual type is specified as type,
  • {:invalid_access, path, access} - one of the components of the path to create, specified as path is a directory, but it's access is is invalid and it's actual access mode is specified as access.

Notes

Implementation details

If we're using SFTP version 3 we get just :failure when trying to create a directory that already exists, so we have no clear error code to distinguish a real error from a case where directory just exists and we can proceed.

Moreover, the path component may exist but it can be a regular file which will prevent us from creating a subdirectory.

Due to these limitations we're checking if directory exists prior to each creation of a directory as we can't rely on the returned error reasons, even if we use newer versions of SFTP they tend to return more fine-grained information.

Timeouts

It was observed in the wild that underlying :ssh_sftp.list_dir/3 and :ssh_sftp.read_file_info/3 always returned {:error, :timeout} with some servers when SFTP version being used was greater than 3, at least with Elixir 1.7.4 and Erlang 21.0. If you encounter such issues try passing {:sftp_vsn, 3} option while creating a SFTP channel.