GenServer which wraps spawned external command.

Use Exile.stream!/1 over using this. Use this only if you are familiar with life-cycle and need more control of the IO streams and OS process.

Comparison with Port

• it is demand driven. User explicitly has to read the command output, and the progress of the external command is controlled using OS pipes. Exile never load more output than we can consume, so we should never experience memory issues

• it can close stdin while consuming output

• tries to handle zombie process by attempting to cleanup external process. Note that there is no middleware involved with exile so it is still possible to endup with zombie process.

• selectively consume stdout and stderr

Internally Exile uses non-blocking asynchronous system calls to interact with the external process. It does not use port's message based communication, instead uses raw stdio and NIF. Uses asynchronous system calls for IO. Most of the system calls are non-blocking, so it should not block the beam schedulers. Make use of dirty-schedulers for IO

Introduction

Exile.Process is a process based wrapper around the external process. It is similar to port as an entity but the interface is different. All communication with the external process must happen via Exile.Process interface.

Exile process life-cycle tied to external process and owners. All system resources such are open file-descriptors, external process are cleaned up when the Exile.Process dies.

Owner

Each Exile.Process has an owner. And it will be the process which created it (via Exile.Process.start_link/2). Process owner can not be changed.

Owner process will be linked to the Exile.Process. So when the exile process is dies abnormally the owner will be killed too or visa-versa. Owner process should avoid trapping the exit signal, if you want avoid the caller getting killed, create a separate process as owner to run the command and monitor that process.

Only owner can get the exit status of the command, using Exile.Process.await_exit/2. All exile processes MUST be awaited. Exit status or reason is ALWAYS sent to the owner. It is similar to Task. If the owner exit without await_exit, the exile process will be killed, but if the owner continue without await_exit then the exile process will linger around till the process exit.

iex> alias Exile.Process
iex> {:ok, p} = Process.start_link(~w(echo hello))
iex> Process.read(p, 100)
{:ok, "hello\n"}
iex> Process.read(p, 100) # read till we get :eof
:eof
iex> Process.await_exit(p)
{:ok, 0}

Pipe & Pipe Owner

Standard IO pipes/channels/streams of the external process such as STDIN, STDOUT, STDERR are called as Pipes. User can either write or read data from pipes.

Each pipe has an owner process and only that process can write or read from the exile process. By default the process who created the exile process is the owner of all the pipes. Pipe owner can be changed using Exile.Process.change_pipe_owner/3.

Pipe owner is monitored and the pipes are closed automatically when the pipe owner exit. Pipe Owner can close the pipe early using Exile.Process.close_stdin/1 etc.

Exile.Process.await_exit/2 closes all of the caller owned pipes by default.

iex> {:ok, p} = Process.start_link(~w(cat))
iex> writer = Task.async(fn ->
...>   :ok = Process.change_pipe_owner(p, :stdin, self())
...>   Process.write(p, "Hello World")
...> end)
iex> Task.await(writer)
:ok
iex> Process.read(p, 100)
{:ok, "Hello World"}
iex> Process.await_exit(p)
{:ok, 0}

Pipe Operations

Only Pipe owner can read or write date to the owned pipe. All Pipe operations (read/write) blocks the caller as a mechanism to put back-pressure, and this also makes the API simpler. This is same as how command-line programs works on the shell, along with pipes in-between, Example: cat larg-file | grep "foo". Internally Exile uses asynchronous IO APIs to avoid blocking VM (by default NIF calls blocks the VM scheduler), so you can open several pipes and do concurrent IO operations without blocking VM.

stderr

by default is :stderr is connected to console, data written to stderr will appear on the console.

You can change the behavior by setting :stderr:

1. :console - stderr output is redirected to console (Default)
2. :redirect_to_stdout - stderr output is redirected to stdout
3. :consume - stderr output read separately, allowing you to consume it separately from stdout. See below for more details
4. :disable - stderr output is redirected /dev/null suppressing all output. See below for more details.

Using redirect_to_stdout

stderr data will be redirected to stdout. When you read stdout you will see both stdout & stderr combined and you won't be able differentiate stdout and stderr separately. This is similar to :stderr_to_stdout option present in Ports.

Unexpected Behaviors

On many systems, stdout and stderr are separated. And between the source program to Exile, via the kernel, there are several places that may buffer data, even temporarily, before Exile is ready to read them. There is no enforced ordering of the readiness of these independent buffers for Exile to make use of.

This can result in unexpected behavior, including:

• mangled data, for example, UTF-8 characters may be incomplete until an additional buffered segment is released on the same source
• raw data, where binary data sent on one source, is incompatible with data sent on the other source.
• interleaved data, where what appears to be synchronous, is not

In short, the two streams might be combined at arbitrary byte position leading to above mentioned issue.

Most well-behaved command-line programs are unlikely to exhibit this, but you need to be aware of the risk.

A good example of this unexpected behavior is streaming JSON from an external tool to Exile, where normal JSON output is expected on stdout, and errors or warnings via stderr. In the case of an unexpected error, the stdout stream could be incomplete, or the stderr message might arrive before the closing data on the stdout stream.

Using consume

stderr data can be consumed separately using Exile.Process.read_stderr/2. Special function Exile.Process.read_any/2 can be used to read from either stdout or stderr whichever has the data available. See the examples for more details.

Unexpected Behaviors

When set, the stderr output MUST be consumed to avoid blocking the external program when stderr buffer is full.

Reading from stderr using read_stderr

# write "Hello" to stdout and "World" to stderr
iex> script = Enum.join(["echo Hello", "echo World >&2"], "\n")
iex> {:ok, p} = Process.start_link(["sh", "-c", script], stderr: :consume)
iex> Process.read(p, 100)
{:ok, "Hello\n"}
iex> Process.read_stderr(p, 100)
{:ok, "World\n"}
iex> Process.await_exit(p)
{:ok, 0}

Reading using read_any

# write "Hello" to stdout and "World" to stderr
iex> script = Enum.join(["echo Hello", "echo World >&2"], "\n")
iex> {:ok, p} = Process.start_link(["sh", "-c", script], stderr: :consume)
iex> Process.read_any(p)
{:ok, {:stdout, "Hello\n"}}
iex> Process.read_any(p)
{:ok, {:stderr, "World\n"}}
iex> Process.await_exit(p)
{:ok, 0}

Process Termination

When owner does (normally or abnormally) the Exile process always terminated irrespective of pipe status or process status. External process get a chance to terminate gracefully, if that fail it will be killed.

If owner calls await_exit then the owner owned pipes are closed and we wait for external process to terminate, if the process already terminated then call returns immediately with exit status. Else command will be attempted to stop gracefully following the exit sequence based on the timeout value (5s by default).

If owner calls await_exit with timeout as :infinity then Exile does not attempt to forcefully stop the external command and wait for command to exit on itself. The await_exit call can be blocked indefinitely waiting for external process to terminate.

If external process exit on its own, exit status is collected and Exile process will wait for owner to close pipes. Most commands exit with pipes are closed, so just ensuring to close pipes when works is done should be enough.

Example of process getting terminated by SIGTERM signal

# sleep command does not watch for stdin or stdout, so closing the
# pipe does not terminate the sleep command.
iex> {:ok, p} = Process.start_link(~w(sleep 100000000)) # sleep indefinitely
iex> Process.await_exit(p, 100) # ensure `await_exit` finish within `100ms`. By default it waits for 5s
{:ok, 143} # 143 is the exit status when command exit due to SIGTERM

Examples

Run a command without any input or output

iex> {:ok, p} = Process.start_link(["sh", "-c", "exit 1"])
iex> Process.await_exit(p)
{:ok, 1}

Single process reading and writing to the command

# bc is a calculator, which reads from stdin and writes output to stdout
iex> {:ok, p} = Process.start_link(~w(bc))
iex> Process.write(p, "1 + 1\n") # there must be new-line to indicate the end of the input line
:ok
iex> Process.read(p)
{:ok, "2\n"}
iex> Process.write(p, "2 * 10 + 1\n")
:ok
iex> Process.read(p)
{:ok, "21\n"}
# We must close stdin to signal the `bc` command that we are done.
# since `await_exit` implicitly closes the pipes, in this case we don't have to
iex> Process.await_exit(p)
{:ok, 0}

Running a command which flush the output on stdin close. This is not supported by Erlang/Elixir ports.

# `base64` command reads all input and writes encoded output when stdin is closed.
iex> {:ok, p} = Process.start_link(~w(base64))
iex> Process.write(p, "abcdef")
:ok
iex> Process.close_stdin(p) # we can selectively close stdin and read all output
:ok
iex> Process.read(p)
{:ok, "YWJjZGVm\n"}
iex> Process.read(p) # typically it is better to read till we receive :eof when we are not sure how big the output data size is
:eof
iex> Process.await_exit(p)
{:ok, 0}

Read and write to pipes in separate processes

iex> {:ok, p} = Process.start_link(~w(cat))
iex> writer = Task.async(fn ->
...>   :ok = Process.change_pipe_owner(p, :stdin, self())
...>   Process.write(p, "Hello World")
...>   # no need to close the pipe explicitly here. Pipe will be closed automatically when process exit
...> end)
iex> reader = Task.async(fn ->
...>   :ok = Process.change_pipe_owner(p, :stdout, self())
...>   Process.read(p)
...> end)
iex> :timer.sleep(500) # wait for the reader and writer to change pipe owner, otherwise `await_exit` will close the pipes before we change pipe owner
iex> Process.await_exit(p, :infinity) # let the reader and writer take indefinite time to finish
{:ok, 0}
iex> Task.await(writer)
:ok
iex> Task.await(reader)
{:ok, "Hello World"}