librarian v0.2.0 SSH View Source

SSH streams and SSH and basic SCP functionality

The librarian SSH module provides SSH streams (see stream/3) and three protocols over the SSH stream:

  • run/3, which runs a command on the remote SSH host.
  • fetch/3, which uses the SCP protocol to obtain a remote file,
  • send/4, which uses the SCP protocol to send a file to the remote host.

Note that not all SSH hosts (for example, embedded shells), implement an SCP command, so you may not necessarily be able to perform SCP over your SSH stream.

Using SSH

The principles of this library are simple. You will first want to create an SSH connection using the connect/2 function. There you will provide credentials (or let the system figure out the default credentials). The returned conn term can then be passed to the multiple utilities.

{:ok, conn} = SSH.connect("some.other.server")
SSH.run!(conn, "echo hello ssh")  # ==> "hello ssh"

Using SCP

This library also provides send!/4 and fetch!/3 functions which let you perform SCP operations via the SSH streams.

{:ok, conn} = SSH.connect("some.other.server")
SSH.send!(conn, binary_or_filestream, "path/to/destination.file")
{:ok, conn} = SSH.connect("some.other.server")
SSH.fetch!(conn, "path/to/source.file")
|> Enum.map(&do_something_with_chunks/1)

Important

If you are performing a streaming SCP send, you may only pass a filestream or a stream-of-a-filestream into the send!/3 function. If you are streaming your filestream through other stream operators make sure that the total file size remains unchanged.

Bang vs non-bang functions

As a general rule, if you expect to run a single or series of tasks with transient (or no) supervision, for example in a worker task or elixir script you should use the bang function and let the task fail, designing your supervision accordingly. This will also potentially let you be lazy about system resources such as SSH connections.

If you expect your SSH task to run as a part of a long-running process (for example, checking in on a host and retrieving data), you should use the error tuple forms and also be careful about closing your ssh connections after use. Check the connection labels documentation for a strategy to organize your code around this neatly.

Mocking

There's a good chance you'll want to mock your SSH commands and responses. The SSH.Api behaviour module is provided for that purpose.

Logging

The SSH and related modules interface with Elixir (and Erlang's) logging facility. The default metadata tagged on the message is ssh: true; if you would like to set it otherwise you can set the :librarian, :ssh_metadata application environment variable.

Customization

If you would like to write your own SSH stream handlers that plug in to the SSH stream and provide either rudimentary interactivity or early stream token processing, you may want to consider implementing a module following the SSH.ModuleApi behaviour, and initiating your stream as desired.

Limitations

This library has largely been tested against Linux SSH clients. Not all SSH schemes are amenable to stream processing. In those cases you should implement an ssh client gen_server using erlang's ssh_client, though support for this in elixir is planned in the near-term.

Link to this section Summary

Types

channel reference for the SSH and SCP operations

connection reference for the SSH and SCP operations

erlang ip4 format, {byte, byte, byte, byte}

connect to a remote is specified using either a domain name or an ip address

unix-style return codes for ssh-executed functions

Functions

closes the ssh connection.

initiates an ssh connection with a remote server.

like connect/2 but raises with a ConnectionError instead of emitting an error tuple.

retrieves a binary file from the remote host.

like fetch/3 except raises instead of emitting an error tuple.

runs a command on the remote host. Typically returns {:ok, result, retval} where retval is the unix return value from the range 0..255.

like run/3 except raises on errors instead of returning an error tuple.

sends binary content to the remote host.

like send/4, except raises on errors, instead of returning an error tuple.

creates an SSH stream struct as an ok tuple or error tuple.

like stream/2, except raises on an error instead of an error tuple.

Link to this section Types

Specs

chan() :: :ssh.channel_id()

channel reference for the SSH and SCP operations

Specs

conn() :: :ssh.connection_ref()

connection reference for the SSH and SCP operations

Specs

connect_result() :: {:ok, conn()} | {:error, any()}

Specs

fetch_result() :: {:ok, binary()} | {:error, term()}

Specs

filestreams() ::
  %Stream{
    accs: term(),
    done: term(),
    enum: %File.Stream{
      line_or_bytes: term(),
      modes: term(),
      path: term(),
      raw: term()
    },
    funs: term()
  }
  | %File.Stream{
      line_or_bytes: term(),
      modes: term(),
      path: term(),
      raw: term()
    }

Specs

ip4() :: :inet.ip4_address()

erlang ip4 format, {byte, byte, byte, byte}

Specs

remote() :: String.t() | charlist() | ip4()

connect to a remote is specified using either a domain name or an ip address

Specs

retval() :: 0..255

unix-style return codes for ssh-executed functions

Specs

run_content() :: iodata() | {String.t(), String.t()}

Specs

run_result() :: {:ok, run_content(), retval()} | {:error, term()}

Specs

send_result() :: :ok | {:error, term()}

Link to this section Functions

Specs

close(conn() | term()) :: :ok | {:error, String.t()}

closes the ssh connection.

Typically you will pass the connection reference to this function. If your connection is contained to its own transient task process, you may not need to call this function as the ssh client library will detect that the process has ended and clean up after you.

In some cases, you may want to be able to close a connection out-of-band. In this case, you may label your connection and use the label to perform the close operation. See labels

Link to this function

connect(remote, options \\ [])

View Source

Specs

connect(remote(), keyword()) :: connect_result()

initiates an ssh connection with a remote server.

options:

  • :use_ssh_config see SSH.Config, defaults to false.
  • :global_config_path see SSH.Config.
  • :user_config_path see SSH.Config.
  • :user username to log in as.
  • :port port to use to ssh, defaults to 22.
  • :label see labels
  • :link if true, links the connection with the calling process. Note the calling process will not die if the SSH connection is closed using close/1.

and other SSH options. Some conversions between ssh options and SSH.connect options:

ssh commandline optionSSH library option
-o StrictHostKeyChecking=nosilently_accept_hosts: true
-qquiet_mode: true
-o ConnectTimeout=timeconnect_timeout: time_in_ms
-i pemfileidentity: file

also consult documentation on client options in the erlang docs

labels:

You can label your ssh connection to provide a side-channel for correctly closing the connection pid. This is most useful in the context of with/1 blocks. As an example, the following code works:

def run_ssh_tasks do
  with {:ok, conn} <- SSH.connect("some_host", label: :this_task),
       {:ok, _result1, 0} <- SSH.run(conn, "some_command"),
       {:ok, result2, 0} <- SSH.run(conn, "some other command") do
    {:ok, result1}
  end
after
  SSH.close(:this_task)
end

Some important points:

  • If you are wrangling multiple SSH sessions, please use unique connection labels.
  • The ssh connection label is stored in the process dictionary, so the label will not be valid across process boundaries.
  • If the ssh connection failed in the first place, the tagged close will return an error tuple. In the example, this will be silent.
Link to this function

connect!(remote, options \\ [])

View Source

Specs

connect!(remote(), keyword()) :: conn() | no_return()

like connect/2 but raises with a ConnectionError instead of emitting an error tuple.

Link to this function

fetch(conn, remote_file, options \\ [])

View Source

Specs

fetch(conn(), Path.t(), keyword()) :: fetch_result()

retrieves a binary file from the remote host.

Under the hood, this uses the scp protocol to transfer files.

The SCP protocol is as follows:

  • execute scp remotely in the undocumented -f <source> mode
  • send a single zero byte to initiate the conversation
  • wait for a control string "C0<perms> <size> <filename>"
  • send a single zero byte
  • wait for the binary data + terminating zero
  • send a single zero byte

The perms term should be in octal, and the filename should be rootless.

Example:

SSH.fetch(conn, "path/to/desired/file")
Link to this function

fetch!(conn, remote_file, options \\ [])

View Source

Specs

fetch!(conn(), Path.t(), keyword()) :: binary() | no_return()

like fetch/3 except raises instead of emitting an error tuple.

Link to this function

run(conn, cmd, options \\ [])

View Source

Specs

run(conn(), String.t(), keyword()) :: run_result()

runs a command on the remote host. Typically returns {:ok, result, retval} where retval is the unix return value from the range 0..255.

the result value is governed by the passed options, but defaults to a string. of the run value.

Options

  • {iostream, redirect}: iostream may be either :stdout or :stderr. redirect may be one of the following:
    • :stream sends the data to the stream.
    • :stdout sends the data to the group_leader stdout.
    • :stderr sends the data to the standard error io stream.
    • :silent deletes all of the data.
    • :raw sends the data to the stream tagged with source information as either {:stdout, data} or {:stderr, data}, as appropriate.
    • {:file, path} sends the data to a new or existing file at the provided path.
    • fun/1 processes the data via the function, with the output flat-mapped into the stream. this means that the results of fun/1 should be lists, with an empty list sending nothing into the stream.
    • fun/2 is like fun/1 except the stream struct is passed as the second parameter. The output of fun/2 should take the shape {flat_map_results, modified_stream}. You may use the :data field of the stream struct to store arbitrary data; and a value of nil indicates that it has been unused.
  • {:tty, true | <options>}: register the connection as a tty connection. Note this changes the default behavior to send the output to group leader stdout instead of to the result, but this is overridable with the iostream redirect above. For options, see :ssh_client_connection.ptty_alloc/4
  • {:env, <env list>}: a list of environment variables to be passed. NB: typically environment variables are filtered by the host environment.
  • {:dir, path}: changes directory to path and then runs the command
  • {:as, :binary} (default): outputs result as a binary
  • {:as, :iolist}: outputs result as an iolist
  • {:as, :tuple}: result takes the shape of the tuple {stdout_binary, stderr_binary} note that this mode will override any other redirection selected.

Example:

SSH.run(conn, "hostname")  # ==> {:ok, "hostname_of_remote\n", 0}

SSH.run(conn, "some_program", stderr: :silent) # ==> similar to running "some_program 2>/dev/null"

SSH.run(conn, "some_program", stderr: :stream) # ==> similar to running "some_program 2>&1"

SSH.run(conn, "some_program", stdout: :silent, stderr: :stream) # ==> only capture standard error
Link to this function

run!(conn, cmd, options \\ [])

View Source

Specs

run!(conn(), String.t(), keyword()) :: run_content() | no_return()

like run/3 except raises on errors instead of returning an error tuple.

Note that by default this raises in the case that the SSH connection fails AND in the case that the remote command returns non-zero.

Link to this function

send(conn, stream, remote_file, options \\ [])

View Source

Specs

send(conn(), iodata() | filestreams(), Path.t(), keyword()) :: send_result()

sends binary content to the remote host.

Under the hood, this uses the scp protocol to transfer files.

Protocol is as follows:

  • execute scp remotely in the undocumented -t <destination> mode
  • send a control string "C0<perms> <size> <filename>"
  • wait for single zero byte
  • send the binary data + terminating zero
  • wait for single zero byte
  • send EOF

The perms term should be in octal, and the filename should be rootless.

options:

  • :permissions - sets unix-style permissions on the file. Defaults to 0o644

Example:

SSH.send(conn, "foo", "path/to/desired/file")
Link to this function

send!(conn, content, remote_file, options \\ [])

View Source

Specs

send!(conn(), iodata(), Path.t(), keyword()) :: :ok | no_return()

like send/4, except raises on errors, instead of returning an error tuple.

Link to this function

stream(conn, cmd, options \\ [])

View Source

Specs

stream(conn(), String.t(), keyword()) ::
  {:ok, SSH.Stream.t()} | {:error, String.t()}

creates an SSH stream struct as an ok tuple or error tuple.

Options

  • {iostream, redirect}: iostream may be either :stdout or :stderr. redirect may be one of the following:
    • :stream sends the data to the stream.
    • :stdout sends the data to the group_leader stdout.
    • :stderr sends the data to the standard error io stream.
    • :silent deletes all of the data.
    • :raw sends the data to the stream tagged with source information as either {:stdout, data} or {:stderr, data}, as appropriate.
    • {:file, path} sends the data to a new or existing file at the provided path.
    • fun/1 processes the data via the function, with the output flat-mapped into the stream. this means that the results of fun/1 should be lists, with an empty list sending nothing into the stream.
    • fun/2 is like fun/1 except the stream struct is passed as the second parameter. The output of fun/2 should take the shape {flat_map_results, modified_stream}. You may use the :data field of the stream struct to store arbitrary data; and a value of nil indicates that it has been unused.
  • {:stream_control_messages, boolean}: should the stream control messages :eof, or {:retval, integer} be sent to the stream?
  • module: {mod, init}, The stream is operated using an module with behaviour SSH.ModuleApi
  • data_timeout: timeout, how long to wait between packets till we send a timeout event.
Link to this function

stream!(conn, cmd, options \\ [])

View Source

Specs

stream!(conn(), String.t(), keyword()) :: SSH.Stream.t() | no_return()

like stream/2, except raises on an error instead of an error tuple.