Localize.Utils.Http (Localize v0.32.0)

Copy Markdown View Source

Supports securely downloading HTTPS content.

This module provides HTTP GET functionality using the built-in :httpc client with certificate verification enabled by default. It follows the erlef security guidelines for secure TLS connections.

The primary public API consists of:

  • get/2 - download content from a URL, returning the body on success.

  • get_with_headers/2 - download content from a URL, returning both headers and body on success.

  • certificate_locations/0 - return the list of possible certificate store locations.

Summary

Functions

Return all possible locations of a certificate file.

Return the dynamically discovered certificate file locations.

Securely download HTTPS content from a URL.

Securely download HTTPS content from a URL, returning headers and body.

Functions

certificate_locations()

@spec certificate_locations() :: [String.t()]

Return all possible locations of a certificate file.

Returns dynamically discovered locations followed by well-known static locations on common operating systems.

Returns

  • A list of file path strings for all candidate certificate locations.

dynamic_certificate_locations()

@spec dynamic_certificate_locations() :: [String.t()]

Return the dynamically discovered certificate file locations.

These include application configuration and optional hex packages such as CAStore and certifi.

Returns

  • A list of file path strings for discovered certificate locations.

get(url, options \\ [])

@spec get(String.t() | {String.t(), list()}, options :: keyword()) ::
  {:ok, binary()} | {:not_modified, any()} | {:error, any()}

Securely download HTTPS content from a URL.

This function uses the built-in :httpc client but enables certificate verification which is not enabled by :httpc by default.

See also https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/ssl

Arguments

  • url is a binary URL or a {url, list_of_headers} tuple. If provided the headers are a list of {'header_name', 'header_value'} tuples. Note that the name and value are both charlists, not strings.

  • options is a keyword list of options.

Options

  • :verify_peer is a boolean value indicating if peer verification should be done for this request. The default is true in which case the default :ssl options follow the erlef guidelines.

  • :timeout is the number of milliseconds available for the request to complete. The default is "120000". This option may also be set with the LOCALIZE_HTTP_TIMEOUT environment variable.

  • :connection_timeout is the number of milliseconds available for a connection to be established to the remote host. The default is "60000". This option may also be set with the LOCALIZE_HTTP_CONNECTION_TIMEOUT environment variable.

Returns

  • {:ok, body} if the return is successful.

  • {:not_modified, headers} if the request would result in returning the same results as one matching an etag.

  • {:error, error} if the download is unsuccessful. An error will also be logged in these cases.

Unsafe HTTPS

If the environment variable LOCALIZE_UNSAFE_HTTPS is set to anything other than "FALSE", "false", "nil" or "NIL" then no peer verification of certificates is performed. Setting this variable is not recommended but may be required where peer verification fails for unidentified reasons.

Certificate stores

In order to keep dependencies to a minimum, get/2 attempts to locate an already installed certificate store. It will try to locate a store in the following order which is intended to satisfy most host systems. The certificate store is expected to be a path name on the host system.

# A certificate store configured by the developer
Application.get_env(:localize, :cacertfile)

# Populated if hex package `CAStore` is configured
CAStore.file_path()

# Populated if hex package `certifi` is configured
:certifi.cacertfile()

# Debian/Ubuntu/Gentoo etc.
"/etc/ssl/certs/ca-certificates.crt"

# Fedora/RHEL 6
"/etc/pki/tls/certs/ca-bundle.crt"

# OpenSUSE
"/etc/ssl/ca-bundle.pem"

# OpenELEC
"/etc/pki/tls/cacert.pem"

# CentOS/RHEL 7
"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem"

# OpenSSL on MacOS
"/usr/local/etc/openssl/cert.pem"

# MacOS & Alpine Linux
"/etc/ssl/cert.pem"

get_with_headers(request, options \\ [])

@spec get_with_headers(String.t() | {String.t(), list()}, options :: keyword()) ::
  {:ok, list(), binary()} | {:not_modified, any()} | {:error, any()}

Securely download HTTPS content from a URL, returning headers and body.

This function uses the built-in :httpc client but enables certificate verification which is not enabled by :httpc by default.

See also https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/ssl

Arguments

  • url is a binary URL or a {url, list_of_headers} tuple. If provided the headers are a list of {'header_name', 'header_value'} tuples. Note that the name and value are both charlists, not strings.

  • options is a keyword list of options.

Options

  • :verify_peer is a boolean value indicating if peer verification should be done for this request. The default is true in which case the default :ssl options follow the erlef guidelines.

  • :timeout is the number of milliseconds available for the request to complete. The default is "120000". This option may also be set with the LOCALIZE_HTTP_TIMEOUT environment variable.

  • :connection_timeout is the number of milliseconds available for a connection to be established to the remote host. The default is "60000". This option may also be set with the LOCALIZE_HTTP_CONNECTION_TIMEOUT environment variable.

  • :https_proxy is the URL of an HTTPS proxy to be used. The default is nil.

Returns

  • {:ok, headers, body} if the return is successful.

  • {:not_modified, headers} if the request would result in returning the same results as one matching an etag.

  • {:error, error} if the download is unsuccessful. An error will also be logged in these cases.

HTTPS Proxy

Localize.Utils.Http.get_with_headers/2 will look for a proxy URL in the following locations in the order presented:

  • options[:https_proxy]

  • Localize compile-time configuration under the key :localize[:https_proxy].

  • The environment variable HTTPS_PROXY.

  • The environment variable https_proxy.