Combo.Endpoint behaviour (combo v0.8.0)

View Source

Defines an endpoint.

The endpoint is the boundary where all requests to a web application start. It is also the interface of the underlying web servers.

Overall, an endpoint has three responsibilities:

  • to define an initial plug pipeline for requests to pass through

  • to provide a wrapper for starting and stopping the endpoint as part of a supervision tree

  • to host configuration for your web application

Endpoints

An endpoint is a module defined with the help of Combo.Endpoint.

defmodule MyApp.Web.Endpoint do
  use Combo.Endpoint, otp_app: :my_app

  # plug ...
  # plug ...

  plug MyApp.Web.Router
end

Endpoints can be added to the supervision tree as following:

children = [
  MyApp.Web.Endpoint
]

Endpoint configuration

Endpoints are configured in your application environment. For example:

config :my_app, MyApp.Web.Endpoint,
  secret_key_base: "kjoy3o1zeidquwy1398juxzldjlksahdk3"

Endpoint configuration is split into two categories:

  • Compile-time configuration
  • Runtime configuretion

Compile-time configuration means the configuration is read during compilation and changing it at runtime has no effect.

Runtime configuration, instead, is read during or after your application is started and can be read through the config/2 function.

Compile-time configuration

The configuration below may be set on config/dev.exs, config/prod.exs and so on, but has no effect on config/runtime.exs.

  • :live_reloader - the configuration of Combo.LiveReloader.

  • :code_reloader - the configuration of Combo.CodeReloader.

  • :debug_errors - when true, uses Plug.Debugger functionality for debugging failures in the application. Recommended to be set to true only in development as it allows listing of the application source code during debugging. Defaults to false.

Runtime configuration

The configuration below may be set on config/dev.exs, config/prod.exs and so on, as well as on config/runtime.exs.

Typically, if you need to configure them with system environment variables, you set them in config/runtime.exs. These options may also be set when starting the endpoint in your supervision tree, such as {MyApp.Web.Endpoint, opts}.

  • :adapter - which web server adapter to use for serving web requests. See the "Adapter configuration" section below.

  • :static - the configuration of Combo.Static.

  • :check_origin - configure the :check_origin option for transports. See socket/3 for options. Defaults to true.

  • :secret_key_base - a secret key used as a base to generate secrets for encrypting and signing data. For example, cookies and tokens are signed by default, but they may also be encrypted if desired. Defaults to nil as it must be set per application.

  • :server - when true, starts the web server when the endpoint supervision tree starts. Defaults to false. The mix combo.serve task automatically sets this to true.

  • :url - a keyword list for generating URLs. Accepts the :scheme, :host, :path and :port options. All option except :path can be changed at runtime. Defaults to:

    [host: "localhost", path: "/"]

    The :scheme option accepts "http" and "https" values. Default value is inferred from top level :http or :https option. It is useful when hosting your web application behind a load balancer or reverse proxy and terminating SSL there.

    The :host option requires a string.

    The :port option requires either an integer or a string.

    The :path option requires a string. It can be used to override root path. It's useful when hosting your web application behind a reverse proxy with URL rewrite rules.

  • :static_url - a keyword list for generating URLs for static files. It will fallback to :url option if not set. Accepts the same value as :url option.

  • :watchers - a set of watchers to run alongside the server. It expects a list of tuples containing the executable and its arguments. Only when the server is enabled, or mix combo.serve runs, watchers will run. For example, the watcher below will run the "watch" mode of the webpack when the server starts. You can configure it to whatever build tool or command you want:

    [
      npx: [
        "webpack",
        "--mode",
        "development",
        "--watch",
        "--watch-options-stdin"
      ]
    ]

    The :cd and :env options can be given at the end of the list to customize the watcher:

    [node: [..., cd: "assets", env: [{"BUILD_MODE", "debug"}]]]

    A watcher can also be an MFA that will be invoked accordingly:

    [another: {Mod, :fun, [arg1, arg2]}]

    When false, watchers can be disabled.

  • :pubsub_server - the name of the pubsub server to use in channels and via the Endpoint broadcast functions. The pubsub server is typically started in your supervision tree.

  • :render_errors - responsible for rendering templates whenever there is a failure in your application. For example, if your application crashes with a 500 error during a HTML request, render("500.html", assigns) will be called in the view given to :render_errors. A :formats list can be provided to specify a module per format to handle error rendering. For example:

    [
      formats: [html: MyApp.Web.ErrorHTML, json: MyApp.Web.ErrorJSON],
      layout: false,
      log: :debug
    ]
  • :log_access_url - log the access url once the server boots. Default to true

Note that you can also store your own configurations in the Combo.Endpoint.

Adapter configuration

Combo allows you to choose which web server adapter to use.

Applications generated via combo_new uses Combo.Endpoint.BanditAdapter. If not otherwise specified via the :adapter option, Combo will fall back to use Combo.Endpoint.Cowboy2Adapter.

Both adapters can be configured in a similar manner using the following two top-level options:

  • :http - the configuration for the HTTP server. It accepts all options as defined by either Bandit or Plug.Cowboy depending on your choice of adapter. Defaults to false.

  • :https - the configuration for the HTTPS server. It accepts all options as defined by either Bandit or Plug.Cowboy depending on your choice of adapter. Defaults to false.

In addition, the connection draining can be configured for the Cowboy web server via the following top-level option (this is not required for Bandit as it has connection draining built-in):

  • :drainer - a drainer process waits for any on-going request to finish during application shutdown. It accepts the :shutdown and :check_interval options as defined by Plug.Cowboy.Drainer. Note the draining does not terminate any existing connection, it simply waits for them to finish. Socket connections run their own drainer before this one is invoked. That's because sockets are stateful and can be gracefully notified, which allows us to stagger them over a longer period of time. See the documentation for socket/3 for more information

Endpoint API

Here's a list of all the functions that are generated in your endpoint:

Summary

Callbacks

Broadcasts a msg as event in the given topic to all nodes.

Broadcasts a msg as event in the given topic to all nodes.

Broadcasts a msg from the given from as event in the given topic to all nodes.

Broadcasts a msg from the given from as event in the given topic to all nodes.

Returns the endpoint configuration for key.

Returns the host from the :url configuration.

Broadcasts a msg as event in the given topic within the current node.

Broadcasts a msg from the given from as event in the given topic within the current node.

Generates the path information when routing to this endpoint.

Returns the script name from the :url configuration.

Returns the address and port that the server is running on

Starts the endpoint supervision tree.

Generates an integrity hash to a static file in priv/static.

Generates a route to a static file in priv/static.

Generates the static URL without any path information.

Subscribes the caller to the given topic.

Unsubscribes the caller from the given topic.

Generates the endpoint base URL without any path information.

Generates the endpoint base URL, but as a URI struct.

Functions

Checks if an endpoint's web server has been configured to start.

Defines a websocket/longpoll mount-point for a socket.

Types

event()

@type event() :: String.t()

msg()

@type msg() :: map() | {:binary, binary()}

topic()

@type topic() :: String.t()

Callbacks

broadcast(topic, event, msg)

@callback broadcast(topic(), event(), msg()) :: :ok | {:error, term()}

Broadcasts a msg as event in the given topic to all nodes.

broadcast!(topic, event, msg)

@callback broadcast!(topic(), event(), msg()) :: :ok

Broadcasts a msg as event in the given topic to all nodes.

Raises in case of failures.

broadcast_from(from, topic, event, msg)

@callback broadcast_from(from :: pid(), topic(), event(), msg()) :: :ok | {:error, term()}

Broadcasts a msg from the given from as event in the given topic to all nodes.

broadcast_from!(from, topic, event, msg)

@callback broadcast_from!(from :: pid(), topic(), event(), msg()) :: :ok

Broadcasts a msg from the given from as event in the given topic to all nodes.

Raises in case of failures.

config key, default

@callback config(key :: atom(), default :: term()) :: term()

Returns the endpoint configuration for key.

host()

@callback host() :: String.t()

Returns the host from the :url configuration.

local_broadcast(topic, event, msg)

@callback local_broadcast(topic(), event(), msg()) :: :ok

Broadcasts a msg as event in the given topic within the current node.

local_broadcast_from(from, topic, event, msg)

@callback local_broadcast_from(from :: pid(), topic(), event(), msg()) :: :ok

Broadcasts a msg from the given from as event in the given topic within the current node.

path(path)

@callback path(path :: String.t()) :: String.t()

Generates the path information when routing to this endpoint.

script_name()

@callback script_name() :: [String.t()]

Returns the script name from the :url configuration.

server_info(scheme)

@callback server_info(Plug.Conn.scheme()) ::
  {:ok,
   {:inet.ip_address(), :inet.port_number()} | :inet.returned_non_ip_address()}
  | {:error, term()}

Returns the address and port that the server is running on

start_link(opts)

@callback start_link(opts :: keyword()) :: Supervisor.on_start()

Starts the endpoint supervision tree.

Starts endpoint's configuration cache and possibly the servers for handling requests.

static_integrity(path)

@callback static_integrity(path :: String.t()) :: String.t() | nil

Generates an integrity hash to a static file in priv/static.

static_path(path)

@callback static_path(path :: String.t()) :: String.t()

Generates a route to a static file in priv/static.

static_url()

@callback static_url() :: String.t()

Generates the static URL without any path information.

subscribe(topic, opts)

@callback subscribe(topic(), opts :: keyword()) :: :ok | {:error, term()}

Subscribes the caller to the given topic.

See Combo.PubSub.subscribe/3 for options.

unsubscribe(topic)

@callback unsubscribe(topic()) :: :ok | {:error, term()}

Unsubscribes the caller from the given topic.

url()

@callback url() :: String.t()

Generates the endpoint base URL without any path information.

url_struct()

@callback url_struct() :: URI.t()

Generates the endpoint base URL, but as a URI struct.

Functions

server?(otp_app, endpoint)

Checks if an endpoint's web server has been configured to start.

  • otp_app - The OTP application running the endpoint, such as :my_app.
  • endpoint - The endpoint module, such as MyApp.Web.Endpoint.

Examples

iex> Combo.Endpoint.server?(:my_app, MyApp.Web.Endpoint)
true

socket(path, module, opts \\ [])

(macro)

Defines a websocket/longpoll mount-point for a socket.

It expects a path, a socket module, and a set of options. The socket module is typically defined with Combo.Socket.

Both websocket and longpolling connections are supported out of the box.

Options

  • :websocket - controls the websocket configuration. May be a boolean or a keyword list of options. See "Common configuration" and "WebSocket configuration" for the whole list. Defaults to true.

  • :longpoll - controls the longpoll configuration. May be a boolean or a keyword list of options. of options. See "Common configuration" and "Longpoll configuration" for the whole list. Defaults to false.

  • :drainer - a keyword list or an MFA function returning a keyword list. For example, {MyApp.Web.Socket, :drainer_configuration, []} configuring how to drain sockets on application shutdown. The goal is to notify all channels clients to reconnect. The supported options are:

    • :batch_size - the amount of clients to notify at once in a given batch. Defaults to 10000.
    • :batch_interval - the amount of time in milliseconds given for a batch to terminate. Defaults to 2000ms.
    • :shutdown - the maximum amount of time in milliseconds allowed to drain all batches. Defaults to 30000ms.
    • :log - the log level for drain actions. Defaults the :log option passed to use Combo.Socket or :info. Set it to false to disable logging.

    For example, if you have 150k connections, the default values will split them into 15 batches of 10k connections. Each batch takes 2000ms before the next batch starts. In this case, we will do everything right under the maximum shutdown time of 30000ms. Therefore, as you increase the number of connections, remember to adjust the shutdown accordingly. Finally, after the socket drainer runs, the lower level HTTP/HTTPS connection drainer will still run, and apply to all connections. Set it to false to disable draining.

  • :auth_token - a boolean that enables the use of the channels client's auth_token option. The exact token exchange mechanism depends on the transport:

    • the websocket transport, this enables a token to be passed through the Sec-WebSocket-Protocol header.
    • the longpoll transport, this allows the token to be passed through the Authorization header.

    The token is available in the connect_info as :auth_token.

    Custom transports might implement their own mechanism.

You can also pass the options below on use Combo.Socket. The values specified here override the value in use Combo.Socket.

Examples

socket "/ws", MyApp.Web.UserSocket

socket "/ws/admin", MyApp.Web.AdminUserSocket,
  longpoll: true,
  websocket: [compress: true]

Path params

It is possible to include variables in the path, these will be available in the params that are passed to the socket.

socket "/ws/:user_id", MyApp.Web.UserSocket,
  websocket: [path: "/project/:project_id"]

Common configuration

The configuration below can be given to both :websocket and :longpoll options:

  • :path - the path to use for the transport. Default to the transport name ("/websocket" or "/longpoll").

  • :serializer - a list of serializers for messages. See Combo.Socket for more information.

  • :transport_log - if the transport layer itself should log and, if so, the level.

  • :check_origin - if the transport should check the origin of requests when the origin header is present. May be a boolean, a list of URIs that are allowed, or a function provided as an MFA. Defaults to :check_origin setting at endpoint configuration.

    If true, the header is checked against :host in MyApp.Web.Endpoint.config(:url)[:host].

    If false and you do not validate the session in your socket, your app is vulnerable to Cross-Site WebSocket Hijacking (CSWSH) attacks. Only use in development, when the host is truly unknown or when serving clients that do not send the origin header, such as mobile apps.

    You can also specify a list of explicitly allowed origins. Each origin may include scheme, host, and port. Wildcards are supported.

    check_origin: [
      "https://example.com",
      "//another.com:888",
      "//*.other.com"
    ]

    Or to accept any origin matching the request connection's host, port, and scheme:

    check_origin: :conn

    Or a function provided as an MFA:

    check_origin: {MyApp.Web.Auth, :check_origin?, []}

    The MFA is invoked with the request %URI{} as the first argument, followed by arguments in the MFA, and must return a boolean.

  • :check_csrf - if the transport should perform CSRF check. To avoid "Cross-Site WebSocket Hijacking", you must have at least one of check_origin and check_csrf enabled. If you set both to false, Combo will raise, but it is still possible to disable both by passing an MFA to check_origin. In such cases, it is your responsibility to ensure at least one of them is enabled. Defaults to true.

  • :code_reloader - enable or disable the code reloader. Defaults to your endpoint configuration.

  • :connect_info - a list of keys that represent data to be copied from the transport to be made available in the user socket connect/3 callback. See the "Connect info" subsection for valid keys.

Connect info

The valid keys are:

  • :peer_data - the result of Plug.Conn.get_peer_data/1.

  • :trace_context_headers - a list of all trace context headers. Supported headers are defined by the W3C Trace Context Specification. These headers are necessary for libraries such as OpenTelemetry to extract trace propagation information to know this request is part of a larger trace in progress.

  • :x_headers - all request headers that have an "x-" prefix.

  • :uri - a %URI{} with information from the conn.

  • :user_agent - the value of the "user-agent" request header.

  • {:session, session_config} - the session information from Plug.Conn. The session_config is typically an exact copy of the arguments given to Plug.Session. In order to validate the session, the "_csrf_token" must be given as request parameter when connecting the socket with the value of URI.encode_www_form(Plug.CSRFProtection.get_csrf_token()). The CSRF token request parameter can be modified via the :csrf_token_key option.

    Additionally, session_config may be an MFA, such as {MyApp.Web.Auth, :get_session_config, []}, to allow loading config in runtime.

Arbitrary keywords may also appear following the above valid keys, which is useful for passing custom connection information to the socket.

For example:

socket "/socket", MyApp.Web.UserSocket,
  websocket: [
    connect_info: [:peer_data, :trace_context_headers, :x_headers, :uri, session: [store: :cookie]]
  ]

With arbitrary keywords:

socket "/socket", MyApp.Web.UserSocket,
  websocket: [
    connect_info: [:uri, custom_value: "abcdef"]
  ]

Where are my headers?

Combo only gives you limited access to the connection headers for security reasons. WebSockets are cross-domain, which means that, when a user "John Doe" visits a malicious website, the malicious website can open up a WebSocket connection to your application, and the browser will gladly submit John Doe's authentication/cookie information. If you were to accept this information as is, the malicious website would have full control of a WebSocket connection to your application, authenticated on John Doe's behalf.

To safe-guard your application, Combo limits and validates the connection information your socket can access. This means your application is safe from these attacks, but you can't access cookies and other headers in your socket. You may access the session stored in the connection via the :connect_info option, provided you also pass a csrf token when connecting over WebSocket.

Websocket configuration

The following configuration applies only to :websocket.

  • :timeout - the timeout for keeping websocket connections open after it last received data. Defaults to 60_000ms.

  • :max_frame_size - the maximum allowed frame size in bytes, Defaults to "infinity".

  • :fullsweep_after - the maximum number of garbage collections before forcing a fullsweep for the socket process. You can set it to 0 to force more frequent cleanups of your websocket transport processes. Setting this option requires Erlang/OTP 24.

  • :compress - whether to enable per message compression on all data frames, Defaults to false.

  • :subprotocols - a list of supported websocket subprotocols. Used for handshake Sec-WebSocket-Protocol response header. Defaults to nil.

    For example:

    subprotocols: ["sip", "mqtt"]
  • :error_handler - custom error handler for connection errors. If Combo.Socket.connect/3 returns an {:error, reason} tuple, the error handler will be called with the error reason. For WebSockets, the error handler must be an MFA tuple that receives a Plug.Conn, the error reason, and returns a Plug.Conn with a response. For example:

    socket "/socket", MyApp.Web.Socket,
      websocket: [
        error_handler: {MyApp.Web.Socket, :handle_error, []}
      ]

    and a {:error, :rate_limit} return may be handled on MyApp.Web.Socket as:

    def handle_error(conn, :rate_limit),
      do: Plug.Conn.send_resp(conn, 429, "Too many requests")

Longpoll configuration

The following configuration applies only to :longpoll:

  • :window_ms - how long the client can wait for new messages in its poll request in milliseconds (ms). Defaults to 10_000.

  • :pubsub_timeout_ms - how long a request can wait for the pubsub layer to respond in milliseconds (ms). Defaults to 2000.

  • :crypto - options for verifying and signing the token, accepted by Combo.Token. By default tokens are valid for 2 weeks.