Xbase.Parser (Xbase v0.1.0)

View Source

Binary parser for DBF file format components.

This module provides functions to parse DBF file headers and field descriptors from binary data using Elixir's efficient binary pattern matching.

Summary

Functions

Appends a new record to the DBF file.

Appends multiple records to the DBF file in a single batch operation.

Deletes multiple records by their indices in a single operation.

Deletes records in a specified index range (inclusive).

Deletes records that match a given condition function.

Updates multiple records by their indices in a single batch operation.

Updates multiple records with write conflict detection.

Updates records that match a given condition function.

Calculates the byte offset for a specific record in the file.

Closes a DBF file handle.

Counts the number of active (non-deleted) records in the DBF file.

Counts the number of deleted records in the DBF file.

Creates a new DBF file with the specified field structure.

Ensures header consistency for all write operations. Call this after any operation that modifies the DBF structure.

Extracts the deletion flag from record data.

Validates if a record index is within the valid range for the file.

Marks a record as deleted in the DBF file.

Marks a record as deleted with write conflict detection.

Returns current memory usage statistics.

Opens a DBF file and parses its header and field descriptors.

Opens a DBF file and parses its header and field descriptors with specified file modes.

Packs a DBF file by removing all deleted records and creating a compacted file.

Packs a DBF file with write conflict detection.

Parses field descriptors from binary data until field terminator (0x0D).

Parses a 32-byte DBF header from binary data.

Parses record field data according to field descriptors.

Reads records in chunks of specified size.

Reads records in chunks with progress reporting.

Reads a complete record from the DBF file at the specified index.

Reads all records from a DBF file.

Provides comprehensive statistics about records in the DBF file.

Refreshes the DBF structure by re-reading the header and fields from file.

Creates a lazy stream of records from the DBF file.

Creates a stream with progress reporting.

Creates a filtered stream of records from the DBF file.

Executes a transaction function with rollback capability.

Undeletes a previously deleted record in the DBF file.

Updates an existing record in the DBF file.

Updates a record with write conflict detection.

Updates a record with automatic retry on conflict detection.

Validates header consistency after write operations.

Functions

append_record(dbf, record_data)

Appends a new record to the DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1 or create_dbf/2
  • record_data - Map of field name => value

Returns

  • {:ok, updated_dbf} - Successfully appended with updated DBF structure
  • {:error, reason} - Error appending record

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> {:ok, updated_dbf} = Xbase.Parser.append_record(dbf, %{"NAME" => "John", "AGE" => 30})

batch_append_records(dbf, records)

Appends multiple records to the DBF file in a single batch operation.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • records - List of record data maps to append

Returns

  • {:ok, updated_dbf} - Successfully appended all records
  • {:error, reason} - Error during batch append

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf", [:read, :write])
iex> records = [%{"NAME" => "John"}, %{"NAME" => "Jane"}]
iex> Xbase.Parser.batch_append_records(dbf, records)
{:ok, updated_dbf}

batch_delete(dbf, indices)

Deletes multiple records by their indices in a single operation.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • indices - List of record indices to delete

Returns

  • {:ok, updated_dbf} - Successfully deleted records
  • {:error, reason} - Error during deletion

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf", [:read, :write])
iex> Xbase.Parser.batch_delete(dbf, [1, 5, 10])
{:ok, updated_dbf}

batch_delete_range(dbf, start_index, end_index)

Deletes records in a specified index range (inclusive).

Parameters

  • dbf - DBF file structure from open_dbf/1
  • start_index - Starting index (inclusive)
  • end_index - Ending index (inclusive)

Returns

  • {:ok, updated_dbf} - Successfully deleted records in range
  • {:error, reason} - Error during deletion

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf", [:read, :write])
iex> Xbase.Parser.batch_delete_range(dbf, 10, 20)
{:ok, updated_dbf}

batch_delete_where(dbf, condition_fn)

Deletes records that match a given condition function.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • condition_fn - Function that takes record data and returns true to delete

Returns

  • {:ok, updated_dbf} - Successfully deleted matching records
  • {:error, reason} - Error during deletion

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf", [:read, :write])
iex> condition = fn record -> record["STATUS"] == "inactive" end
iex> Xbase.Parser.batch_delete_where(dbf, condition)
{:ok, updated_dbf}

batch_update_records(dbf, updates)

Updates multiple records by their indices in a single batch operation.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • updates - List of {index, update_data} tuples

Returns

  • {:ok, updated_dbf} - Successfully updated all records
  • {:error, reason} - Error during batch update

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf", [:read, :write])
iex> updates = [{0, %{"NAME" => "John"}}, {2, %{"STATUS" => "active"}}]
iex> Xbase.Parser.batch_update_records(dbf, updates)
{:ok, updated_dbf}

batch_update_records_with_conflict_check(dbf, updates)

Updates multiple records with write conflict detection.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • updates - List of {index, update_data} tuples

Returns

  • {:ok, updated_dbf} - Successfully updated records
  • {:error, :write_conflict} - File was modified by another process
  • {:error, reason} - Other error during batch update

batch_update_where(dbf, condition_fn, update_data)

Updates records that match a given condition function.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • condition_fn - Function that takes record data and returns true to update
  • update_data - Map of field updates to apply

Returns

  • {:ok, updated_dbf} - Successfully updated matching records
  • {:error, reason} - Error during batch update

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf", [:read, :write])
iex> condition = fn record -> record["STATUS"] == "pending" end
iex> Xbase.Parser.batch_update_where(dbf, condition, %{"STATUS" => "active"})
{:ok, updated_dbf}

calculate_record_offset(header, record_index)

Calculates the byte offset for a specific record in the file.

Parameters

  • header - The DBF header containing file structure information
  • record_index - Zero-based record index

Returns

  • The byte offset where the record starts

Examples

iex> header = %Header{header_length: 97, record_length: 25}
iex> Xbase.Parser.calculate_record_offset(header, 0)
97
iex> Xbase.Parser.calculate_record_offset(header, 1)
122

close_dbf(map)

Closes a DBF file handle.

Parameters

  • dbf - DBF structure returned from open_dbf/1

Returns

  • :ok - File closed successfully

count_active_records(dbf)

Counts the number of active (non-deleted) records in the DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1

Returns

  • {:ok, count} - Number of active records
  • {:error, reason} - Error reading records

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> Xbase.Parser.count_active_records(dbf)
{:ok, 150}

count_deleted_records(dbf)

Counts the number of deleted records in the DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1

Returns

  • {:ok, count} - Number of deleted records
  • {:error, reason} - Error reading records

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> Xbase.Parser.count_deleted_records(dbf)
{:ok, 25}

create_dbf(path, fields, opts \\ [])

Creates a new DBF file with the specified field structure.

Parameters

  • path - Path for the new DBF file
  • fields - List of field descriptors defining the schema
  • opts - Options (version, overwrite)

Returns

  • {:ok, dbf} - Successfully created DBF file structure
  • {:error, reason} - Error creating file

Examples

iex> fields = [%FieldDescriptor{name: "NAME", type: "C", length: 20}]
iex> {:ok, dbf} = Xbase.Parser.create_dbf("new.dbf", fields)

ensure_header_consistency(dbf)

Ensures header consistency for all write operations. Call this after any operation that modifies the DBF structure.

Parameters

  • dbf - DBF file structure

Returns

  • {:ok, dbf} - Header is consistent
  • {:error, reason} - Header inconsistency detected

get_deletion_flag(arg)

Extracts the deletion flag from record data.

Parameters

  • record_binary - The raw record binary data

Returns

  • {:ok, boolean} - true if deleted (0x2A), false if active (0x20)
  • {:error, reason} - Error for invalid data

is_valid_record_index?(header, record_index)

Validates if a record index is within the valid range for the file.

Parameters

  • header - The DBF header containing record count
  • record_index - Zero-based record index to validate

Returns

  • true if the index is valid, false otherwise

mark_deleted(dbf, record_index)

Marks a record as deleted in the DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • record_index - Zero-based index of the record to mark as deleted

Returns

  • {:ok, updated_dbf} - Successfully marked as deleted with updated DBF structure
  • {:error, reason} - Error marking record as deleted

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> {:ok, updated_dbf} = Xbase.Parser.mark_deleted(dbf, 2)

mark_deleted_with_conflict_check(dbf, record_index)

Marks a record as deleted with write conflict detection.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • record_index - Index of record to delete

Returns

  • {:ok, updated_dbf} - Successfully marked record as deleted
  • {:error, :write_conflict} - File was modified by another process
  • {:error, reason} - Other error during deletion

memory_usage()

Returns current memory usage statistics.

Returns

  • %{total: integer, processes: integer, system: integer} - Memory usage in bytes

Examples

memory = Xbase.Parser.memory_usage()
# => %{total: 52428800, processes: 12345, system: 9876, ...}

open_dbf(path)

Opens a DBF file and parses its header and field descriptors.

Parameters

  • path - Path to the DBF file

Returns

  • {:ok, %{header: header, fields: fields, file: file}} - Successfully opened DBF
  • {:error, reason} - Error opening or parsing file

Examples

iex> Xbase.Parser.open_dbf("data.dbf")
{:ok, %{header: %Header{...}, fields: [...], file: #Port<...>}}

open_dbf(path, modes)

Opens a DBF file and parses its header and field descriptors with specified file modes.

Parameters

  • path - Path to the DBF file
  • modes - List of file open modes (e.g., [:read], [:read, :write])

Returns

  • {:ok, %{header: header, fields: fields, file: file}} - Successfully opened DBF
  • {:error, reason} - Error opening or parsing file

pack(dbf, output_path)

Packs a DBF file by removing all deleted records and creating a compacted file.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • output_path - Path for the packed output file

Returns

  • {:ok, packed_dbf} - Successfully packed DBF file structure
  • {:error, reason} - Error packing file

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> {:ok, packed_dbf} = Xbase.Parser.pack(dbf, "data_packed.dbf")

pack_with_conflict_check(dbf, output_path)

Packs a DBF file with write conflict detection.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • output_path - Path for the packed file

Returns

  • {:ok, packed_dbf} - Successfully packed file
  • {:error, :write_conflict} - File was modified by another process
  • {:error, reason} - Other error during packing

parse_fields(binary, offset \\ 0)

Parses field descriptors from binary data until field terminator (0x0D).

Parameters

  • binary - The binary data containing field descriptors
  • offset - Starting offset in the binary data

Returns

  • {:ok, [%FieldDescriptor{}]} - List of parsed field descriptors
  • {:error, reason} - Parse error with reason

parse_header(binary)

Parses a 32-byte DBF header from binary data.

Parameters

  • binary - The binary data containing the DBF header

Returns

  • {:ok, %Header{}} - Successfully parsed header
  • {:error, reason} - Parse error with reason

Examples

iex> header_data = <<0x03, 124, 12, 17, 100::little-32, 161::little-16, 50::little-16, 0::16, 0, 0, 0::12*8, 0, 0, 0::16>>
iex> Xbase.Parser.parse_header(header_data)
{:ok, %Xbase.Types.Header{version: 3, record_count: 100, ...}}

parse_record_data(record_data, fields)

Parses record field data according to field descriptors.

Parameters

  • record_data - Binary data for the record fields (without deletion flag)
  • fields - List of field descriptors

Returns

  • {:ok, %{field_name => parsed_value}} - Parsed field data
  • {:error, reason} - Parse error

read_in_chunks(dbf, chunk_size)

Reads records in chunks of specified size.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • chunk_size - Number of records per chunk

Returns

  • Stream.t() - Stream of record lists (chunks)

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> chunks = Xbase.Parser.read_in_chunks(dbf, 100)
iex> Enum.each(chunks, fn chunk -> process_chunk(chunk) end)
:ok

read_in_chunks_with_progress(dbf, chunk_size, progress_fn)

Reads records in chunks with progress reporting.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • chunk_size - Number of records per chunk
  • progress_fn - Function called with progress info: fn %{current: int, total: int, percentage: float} -> any end

Returns

  • Stream.t() - Stream of record lists (chunks) with progress reporting

Examples

progress_fn = fn prog -> IO.puts("Progress: " <> to_string(prog.percentage) <> "%") end
chunks = Xbase.Parser.read_in_chunks_with_progress(dbf, 100, progress_fn)
Enum.each(chunks, fn chunk -> process_chunk(chunk) end)

read_record(map, record_index)

Reads a complete record from the DBF file at the specified index.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • record_index - Zero-based record index

Returns

  • {:ok, %Record{}} - Successfully parsed record
  • {:error, reason} - Error reading or parsing record

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> {:ok, record} = Xbase.Parser.read_record(dbf, 0)
iex> record.data["NAME"]
"John Doe"

read_records(dbf)

Reads all records from a DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1

Returns

  • {:ok, [record_data]} - List of record data maps
  • {:error, reason} - Error reading records

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> {:ok, records} = Xbase.Parser.read_records(dbf)
iex> length(records)
10

record_statistics(dbf)

Provides comprehensive statistics about records in the DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1

Returns

  • {:ok, %{total_records: int, active_records: int, deleted_records: int, deletion_percentage: float}} - Statistics
  • {:error, reason} - Error reading records

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> Xbase.Parser.record_statistics(dbf)
{:ok, %{total_records: 100, active_records: 85, deleted_records: 15, deletion_percentage: 15.0}}

refresh_dbf_state(dbf)

Refreshes the DBF structure by re-reading the header and fields from file.

Parameters

  • dbf - DBF file structure from open_dbf/1

Returns

  • {:ok, refreshed_dbf} - DBF with updated header and fields
  • {:error, reason} - Error refreshing from file

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf", [:read, :write])
iex> Xbase.Parser.refresh_dbf_state(dbf)
{:ok, refreshed_dbf}

stream_records(dbf)

Creates a lazy stream of records from the DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1

Returns

  • Stream.t() - Stream of record data maps (excludes deleted records)

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> stream = Xbase.Parser.stream_records(dbf)
iex> stream |> Enum.take(10) |> length()
10

stream_records_with_progress(dbf, progress_fn)

Creates a stream with progress reporting.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • progress_fn - Function called with progress info

Returns

  • Stream.t() - Stream of records with progress reporting

Examples

progress_fn = fn prog -> send(self(), {:progress, prog}) end
records = Xbase.Parser.stream_records_with_progress(dbf, progress_fn) |> Enum.to_list()

stream_where(dbf, filter_fn)

Creates a filtered stream of records from the DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • filter_fn - Function that takes record data and returns true/false

Returns

  • Stream.t() - Stream of filtered record data maps

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> high_scores = fn record -> record["SCORE"] > 90 end
iex> stream = Xbase.Parser.stream_where(dbf, high_scores)
iex> Enum.to_list(stream)
[%{"SCORE" => 95, ...}, ...]

transaction(dbf, transaction_fn)

Executes a transaction function with rollback capability.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • transaction_fn - Function that takes a DBF and returns {:ok, updated_dbf} or {:error, reason}

Returns

  • {:ok, final_dbf} - Successfully committed transaction
  • {:error, reason} - Transaction failed and was rolled back

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> {:ok, final_dbf} = Xbase.Parser.transaction(dbf, fn dbf ->
...>   {:ok, dbf1} = Xbase.Parser.append_record(dbf, %{"NAME" => "John"})
...>   {:ok, dbf2} = Xbase.Parser.update_record(dbf1, 0, %{"STATUS" => "active"})
...>   {:ok, dbf2}
...> end)

undelete_record(dbf, record_index)

Undeletes a previously deleted record in the DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • record_index - Zero-based index of the record to undelete

Returns

  • {:ok, updated_dbf} - Successfully undeleted with updated DBF structure
  • {:error, reason} - Error undeleting record

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> {:ok, updated_dbf} = Xbase.Parser.undelete_record(dbf, 2)

update_record(dbf, record_index, update_data)

Updates an existing record in the DBF file.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • record_index - Zero-based index of the record to update
  • update_data - Map of field name => value for fields to update

Returns

  • {:ok, updated_dbf} - Successfully updated with updated DBF structure
  • {:error, reason} - Error updating record

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf")
iex> {:ok, updated_dbf} = Xbase.Parser.update_record(dbf, 0, %{"NAME" => "Updated", "AGE" => 40})

update_record_with_conflict_check(dbf, record_index, update_data)

Updates a record with write conflict detection.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • record_index - Index of record to update
  • update_data - Map of field updates

Returns

  • {:ok, updated_dbf} - Successfully updated record
  • {:error, :write_conflict} - File was modified by another process
  • {:error, reason} - Other error during update

Examples

iex> {:ok, dbf} = Xbase.Parser.open_dbf("data.dbf", [:read, :write])
iex> Xbase.Parser.update_record_with_conflict_check(dbf, 0, %{"NAME" => "John"})
{:ok, updated_dbf}

update_record_with_retry(dbf, record_index, update_data)

Updates a record with automatic retry on conflict detection.

This function automatically refreshes the DBF state if a write conflict is detected and retries the operation once.

Parameters

  • dbf - DBF file structure from open_dbf/1
  • record_index - Index of record to update
  • update_data - Map of field updates

Returns

  • {:ok, updated_dbf} - Successfully updated record
  • {:error, reason} - Error during update (after retry if conflict occurred)

validate_header_consistency(dbf)

Validates header consistency after write operations.

Parameters

  • dbf - DBF file structure

Returns

  • :ok - Header is consistent
  • {:error, reason} - Header inconsistency detected

write_header(file, header)