Pipeline.Utils.FileUtils (pipeline v0.0.1)

View Source

File utility functions for pipeline file operations.

Provides comprehensive file manipulation capabilities including:

  • File operations (copy, move, delete)
  • File validation (existence, size, permissions)
  • Format conversion (CSV, JSON, YAML, XML)
  • Path resolution and workspace management
  • Streaming operations for large files (>100MB)
  • Memory-efficient file processing

Summary

Functions

convert_format(source, destination, format)

Convert file format from source to destination.

copy_file(source, destination)

Copy a file from source to destination with error handling.

delete_file(path)

Delete a file or directory with error handling.

get_file_size(path)

Get file size efficiently.

list_files(path, pattern \\ nil)

List files in a directory with optional pattern matching.

move_file(source, destination)

Move a file from source to destination with error handling.

resolve_path(path, workspace_dir)

Resolve a path relative to the workspace directory.

should_use_streaming?(path)

Check if a file should be processed using streaming based on size.

smart_process_file(path, processor_fn)

Process a file with automatic streaming decision.

stream_copy_file(source, destination)

Copy a large file using streaming to avoid memory issues.

stream_process_lines(source_path, processor_fn)

Process a large file line by line with a given function. Memory-efficient for large text files.

stream_read_file(path)

Read a file in chunks using streaming. Returns a stream that yields binary chunks.

stream_write_file(path, data_stream)

Write data to a file using streaming.

validate_file(path, criteria)

Validate file properties according to given criteria.

Functions

convert_format(source, destination, format)

@spec convert_format(String.t(), String.t(), String.t()) :: :ok | {:error, String.t()}

Convert file format from source to destination.

copy_file(source, destination)

@spec copy_file(String.t(), String.t()) :: :ok | {:error, String.t()}

Copy a file from source to destination with error handling.

delete_file(path)

@spec delete_file(String.t()) :: :ok | {:error, String.t()}

Delete a file or directory with error handling.

get_file_size(path)

@spec get_file_size(String.t()) :: {:ok, non_neg_integer()} | {:error, String.t()}

Get file size efficiently.

list_files(path, pattern \\ nil)

@spec list_files(String.t(), String.t() | nil) ::
  {:ok, [String.t()]} | {:error, String.t()}

List files in a directory with optional pattern matching.

move_file(source, destination)

@spec move_file(String.t(), String.t()) :: :ok | {:error, String.t()}

Move a file from source to destination with error handling.

resolve_path(path, workspace_dir)

@spec resolve_path(String.t(), String.t()) :: String.t()

Resolve a path relative to the workspace directory.

should_use_streaming?(path)

@spec should_use_streaming?(String.t()) :: boolean()

Check if a file should be processed using streaming based on size.

smart_process_file(path, processor_fn)

@spec smart_process_file(String.t(), (String.t() -> any())) ::
  {:ok, any()} | {:error, String.t()}

Process a file with automatic streaming decision.

stream_copy_file(source, destination)

@spec stream_copy_file(String.t(), String.t()) :: :ok | {:error, String.t()}

Copy a large file using streaming to avoid memory issues.

stream_process_lines(source_path, processor_fn)

@spec stream_process_lines(String.t(), (String.t() -> String.t())) ::
  {:ok, String.t()} | {:error, String.t()}

Process a large file line by line with a given function. Memory-efficient for large text files.

stream_read_file(path)

@spec stream_read_file(String.t()) :: File.Stream.t() | {:error, String.t()}

Read a file in chunks using streaming. Returns a stream that yields binary chunks.

stream_write_file(path, data_stream)

@spec stream_write_file(String.t(), Enumerable.t()) :: :ok | {:error, String.t()}

Write data to a file using streaming.

validate_file(path, criteria)

@spec validate_file(String.t(), map()) :: :ok | {:error, String.t()}

Validate file properties according to given criteria.