View Source GoogleApi.BigQuery.V2.Model.SparkOptions (google_api_big_query v0.79.0)

Options for a user-defined Spark routine.

Attributes

archiveUris (type: list(String.t), default: nil) - Archive files to be extracted into the working directory of each executor. For more information about Apache Spark, see Apache Spark.
connection (type: String.t, default: nil) - Fully qualified name of the user-provided Spark connection object. Format: "projects/{project_id}/locations/{location_id}/connections/{connection_id}"
containerImage (type: String.t, default: nil) - Custom container image for the runtime environment.
fileUris (type: list(String.t), default: nil) - Files to be placed in the working directory of each executor. For more information about Apache Spark, see Apache Spark.
jarUris (type: list(String.t), default: nil) - JARs to include on the driver and executor CLASSPATH. For more information about Apache Spark, see Apache Spark.
mainClass (type: String.t, default: nil) - The fully qualified name of a class in jar_uris, for example, com.example.wordcount. Exactly one of main_class and main_jar_uri field should be set for Java/Scala language type.
mainFileUri (type: String.t, default: nil) - The main file/jar URI of the Spark application. Exactly one of the definition_body field and the main_file_uri field must be set for Python. Exactly one of main_class and main_file_uri field should be set for Java/Scala language type.
properties (type: map(), default: nil) - Configuration properties as a set of key/value pairs, which will be passed on to the Spark application. For more information, see Apache Spark and the procedure option list.
pyFileUris (type: list(String.t), default: nil) - Python files to be placed on the PYTHONPATH for PySpark application. Supported file types: .py, .egg, and .zip. For more information about Apache Spark, see Apache Spark.
runtimeVersion (type: String.t, default: nil) - Runtime version. If not specified, the default runtime version is used.

Summary

Types

t()

Functions

decode(value, options)

Unwrap a decoded JSON object into its complex fields.

Types

t()

@type t() :: %GoogleApi.BigQuery.V2.Model.SparkOptions{
  archiveUris: [String.t()] | nil,
  connection: String.t() | nil,
  containerImage: String.t() | nil,
  fileUris: [String.t()] | nil,
  jarUris: [String.t()] | nil,
  mainClass: String.t() | nil,
  mainFileUri: String.t() | nil,
  properties: map() | nil,
  pyFileUris: [String.t()] | nil,
  runtimeVersion: String.t() | nil
}

Functions

decode(value, options)

@spec decode(struct(), keyword()) :: struct()

Unwrap a decoded JSON object into its complex fields.