scrapy_cloud_ex v0.1.2 ScrapyCloudEx.HttpAdapter.RequestConfig View Source

Struct containing the configuration for an API call.

Link to this section Summary

Types

t()

Contains the configuration relevant for an API request

Link to this section Types

t() :: %ScrapyCloudEx.HttpAdapter.RequestConfig{
  api_key: term(),
  body: term(),
  headers: term(),
  method: term(),
  opts: term(),
  url: term()
}

Contains the configuration relevant for an API request:

:api_key - the API key as can be obtained here. This must be provided to the API either by using HTTP Basic authentication (which is the approach used by ScrapyCloudEx.HttpAdapters.Default), or within the URL as a query parameter: https://storage.scrapinghub.com/foo?apikey=APIKEY. See docs for more info.
:url - the API URL to send the request to. May contain query parameters.
:method - HTTP request method to use. Supported values are: :get, :post, :put, :delete.
:headers - headers to add to the request. By default a {:"Accept-Encoding", "gzip"} is always present.
:body - request body.
:opts - any options provided to an endpoint method will be copied here. Always contains a :decoder value which is either a module implementing the ScrapyCloudEx.Decoder behaviour, or a function of type ScrapyCloudEx.Decoder.decoder_function/0. Adding values here can be particularly useful to work around certain API quirks, such as the ScrapyCloudEx.Endpoints.App.Jobs.list/4 endpoint which will return a “text/plain” encoding value when requesting the :jl format. By adding (for example) the requested format in the :opts parameter of the endpoint call, the implementation of ScrapyCloudEx.HttpAdapter.handle_response/2 can process the body as appropriate.