PBF Parser v0.1.2 PBFParser View Source

Elixir parser and decoder for OpenStreetMap PBF format described in PBF file specification. This library provides a collection of functions that one can use to build their own decoder flow of .pbf files, as seen in examples.

Examples

With Stream

PBFParser.stream("test.osm.pbf")
  |> Stream.drop(1)
  |> Stream.map(&PBFParser.decompress_block/1)
  |> Stream.map(&PBFParser.decode_block/1)
  |> Stream.each(&IO.inspect/1)
  |> Stream.run()

With Flow

PBFParser.stream("test.osm.pbf")
  |> Stream.drop(1)
  |> Stream.take(1_000)
  |> Flow.from_enumerable(max_demand: 50)
  |> Flow.partition(max_demand: 5, stages: 5)
  |> Flow.map(&PBFParser.decompress_block/1)
  |> Flow.partition(max_demand: 5, stages: 10)
  |> Flow.map(&PBFParser.decode_block/1)
  |> Flow.partition(window: Flow.Window.count(20))
  |> Flow.reduce(fn -> [] end, fn batch, total -> [batch | total] end)
  |> Flow.emit(:state)
  |> Flow.partition(max_demand: 5, stages: 1)
  |> Flow.each(fn item -> IO.inspect(length(item)) end)
  |> Flow.run()

Link to this section Summary

Functions

decode_block(block)

Decodes the raw PrimitiveBlock (as obtained from PBFParser.decompress_block/1) into a more usable format. Each block usually contains around 8000 densely packed node entities and a number of relation and way entities. Those are extracted along with accompanying metadata

decompress_block(data)

Decompresses zlib encoded block data (as obtained from PBFParser.stream/1)

decompress_header(data)

Decompresses zlib encoded header data (as obtained from PBFParser.stream/1)

stream(path)

Opens .pbf file specified by given path and return a Stream yielding zlib encoded data of consecutive Blobs. First emitted chunk of data should represent a HeaderBlock, all those coming after should be decoded as PrimitiveBlocks

Link to this section Functions

decode_block(block)

decode_block(PBFParser.Proto.OsmFormat.PrimitiveBlock.t()) :: [
  Data.Node.t() | Data.Relation.t() | Data.Way.t()
]

Returns a list containing PBFParser.Data.Node, PBFParser.Data.Relation and PBFParser.Data.Way structs.

Example

iex(1)> PBFParser.decode_decode_block(...)
[
    ...
    %PBFParser.Data.Node{
    id: 219219898,
    info: %PBFParser.Data.Info{
      changeset: 0,
      timestamp: #DateTime<2008-01-11 23:29:41.000Z>,
      uid: 0,
      user: "",
      version: 1,
      visible: nil
    },
    latitude: 14.860650000000001,
    longitude: -83.43016,
    tags: %{"created_by" => "JOSM"}
  },
  ...
]

decompress_block(data)

decompress_block(iodata()) :: PBFParser.Proto.OsmFormat.PrimitiveBlock.t()

Decompresses zlib encoded block data (as obtained from PBFParser.stream/1).

Returns PrimitiveBlock, a struct generated directly from PBF protobuf specification.

decompress_header(data)

decompress_header(iodata()) :: PBFParser.Proto.OsmFormat.HeaderBlock.t()

Decompresses zlib encoded header data (as obtained from PBFParser.stream/1).

Returns HeaderBlock, a struct generated directly from PBF protobuf specification.

stream(path)

stream(String.t()) :: Enumerable.t()