HTTP/2 Analysis Guide

Complete guide to analyzing HTTP/2 cleartext (h2c) traffic in PcapFileEx.

HTTP/2 Overview

PcapFileEx provides HTTP/2 stream reconstruction for cleartext (h2c) traffic:

• Cleartext only: No TLS-encrypted HTTP/2 (h2) support
• Prior-knowledge h2c: No HTTP/1.1 Upgrade flow support
• Analysis only: No playback server implementation

Quick Start

# Analyze PCAP file for HTTP/2 exchanges
{:ok, complete, incomplete} = PcapFileEx.HTTP2.analyze("capture.pcap")

# Print complete exchanges
Enum.each(complete, fn ex ->
  IO.puts("#{ex.request.method} #{ex.request.path} -> #{ex.response.status}")
end)

# Check incomplete exchanges
Enum.each(incomplete, fn ex ->
  IO.puts("Incomplete: #{PcapFileEx.HTTP2.IncompleteExchange.to_string(ex)}")
end)

Public API

analyze/2

Analyzes a PCAP file and returns HTTP/2 exchanges:

{:ok, complete, incomplete} = PcapFileEx.HTTP2.analyze("capture.pcap")

# With port filter
{:ok, complete, incomplete} = PcapFileEx.HTTP2.analyze("capture.pcap", port: 8080)

# Disable content decoding (raw binary bodies)
{:ok, complete, incomplete} = PcapFileEx.HTTP2.analyze("capture.pcap", decode_content: false)

Options:

• :port - Filter to specific TCP port (default: nil, all ports)
• :decode_content - Auto-decode bodies based on Content-Type (default: true)

Returns:

• complete - List of Exchange.t() with full request/response pairs
• incomplete - List of IncompleteExchange.t() for partial exchanges

analyze_segments/2

Analyzes directional TCP segments directly (skip PCAP parsing):

segments = [
  %{flow_key: {client, server}, direction: :a_to_b, data: preface, timestamp: ts1},
  %{flow_key: {client, server}, direction: :a_to_b, data: settings, timestamp: ts2},
  ...
]

{:ok, complete, incomplete} = PcapFileEx.HTTP2.analyze_segments(segments)

# With options
{:ok, complete, incomplete} = PcapFileEx.HTTP2.analyze_segments(segments, decode_content: false)

Options:

• :decode_content - Auto-decode bodies based on Content-Type (default: true)

http2?/1

Check if binary starts with HTTP/2 connection preface:

PcapFileEx.HTTP2.http2?(payload)  # => true/false

connection_preface/0

Returns the HTTP/2 connection preface string (24 bytes):

preface = PcapFileEx.HTTP2.connection_preface()
# => "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"

Exchange Structure

Complete Exchange

%PcapFileEx.HTTP2.Exchange{
  stream_id: 1,
  flow_key: {client_endpoint, server_endpoint},

  request: %PcapFileEx.HTTP2.Request{
    method: "GET",
    path: "/api/users",
    scheme: "http",
    authority: "localhost:8080",
    headers: %PcapFileEx.HTTP2.Headers{
      pseudo: %{":method" => "GET", ":path" => "/api/users", ...},
      regular: %{"content-type" => "application/json", ...}
    },
    body: "",
    decoded_body: nil,  # Auto-decoded based on Content-Type
    trailers: nil
  },

  response: %PcapFileEx.HTTP2.Response{
    status: 200,
    headers: %PcapFileEx.HTTP2.Headers{
      pseudo: %{":status" => "200"},
      regular: %{"content-type" => "application/json", ...}
    },
    body: "{\"users\": [...]}",
    decoded_body: {:json, %{"users" => [...]}},  # Auto-decoded JSON
    trailers: nil
  },

  request_timestamp: ~U[2024-01-01 12:00:00Z],
  response_timestamp: ~U[2024-01-01 12:00:01Z]
}

Incomplete Exchange

%PcapFileEx.HTTP2.IncompleteExchange{
  stream_id: 3,
  flow_key: {client_endpoint, server_endpoint},
  request: %PcapFileEx.HTTP2.Request{...},  # May be nil
  response: %PcapFileEx.HTTP2.Response{...},  # May be nil
  reason: :rst_stream | {:rst_stream, error_code} | {:goaway, last_stream_id} | :truncated_no_response
}

Understanding Incomplete Exchanges

Exchanges may be incomplete for several reasons:

RST_STREAM

Stream was reset by client or server:

case ex.reason do
  {:rst_stream, 0x08} -> IO.puts("Stream cancelled (CANCEL)")
  {:rst_stream, 0x07} -> IO.puts("Stream refused (REFUSED_STREAM)")
  {:rst_stream, code} -> IO.puts("RST_STREAM error: #{code}")
end

GOAWAY

Connection was terminated:

case ex.reason do
  {:goaway, last_stream_id} ->
    IO.puts("GOAWAY: streams > #{last_stream_id} were terminated")
end

Truncated

Capture ended before exchange completed:

case ex.reason do
  :truncated_no_response -> IO.puts("Request sent, no response captured")
  :truncated -> IO.puts("Exchange incomplete (capture ended)")
end

Content Decoding

HTTP/2 exchanges automatically decode request and response bodies based on Content-Type headers.

Decoded Content Types

Accessing Decoded Bodies

{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap")

Enum.each(complete, fn ex ->
  case ex.response.decoded_body do
    {:json, data} ->
      IO.inspect(data, label: "JSON response")

    {:text, text} ->
      IO.puts("Text response: #{text}")

    {:multipart, parts} ->
      Enum.each(parts, fn part ->
        IO.puts("Part: #{part.content_type}")
        IO.inspect(part.body)
      end)

    {:binary, bin} ->
      IO.puts("Binary response: #{byte_size(bin)} bytes")

    nil ->
      IO.puts("No body")
  end
end)

Multipart Response Handling

Multipart bodies are recursively decoded. Each part has:

• content_type - Part's Content-Type header
• content_id - Part's Content-Id header (or nil)
• headers - All part headers (lowercase keys)
• body - Recursively decoded body (tagged tuple)

{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap")

Enum.each(complete, fn ex ->
  case ex.response.decoded_body do
    {:multipart, parts} ->
      Enum.each(parts, fn part ->
        IO.puts("Part #{part.content_id}: #{part.content_type}")
        case part.body do
          {:json, json} -> IO.inspect(json)
          {:text, text} -> IO.puts(text)
          {:binary, bin} -> IO.puts("Binary: #{byte_size(bin)} bytes")
        end
      end)
    _ -> :skip
  end
end)

Disabling Content Decoding

For raw binary access without decoding overhead:

{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap", decode_content: false)

ex = hd(complete)
ex.response.body          # Raw binary
ex.response.decoded_body  # nil (not decoded)

Common Patterns

Pattern 1: Extract All API Calls

{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap")

api_calls = complete
|> Enum.filter(fn ex ->
  String.starts_with?(ex.request.path, "/api/")
end)
|> Enum.map(fn ex ->
  %{
    method: ex.request.method,
    path: ex.request.path,
    status: ex.response.status,
    request_time: ex.request_timestamp,
    response_time: ex.response_timestamp
  }
end)

Pattern 2: Find Error Responses

{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap")

errors = Enum.filter(complete, fn ex ->
  ex.response.status >= 400
end)

Enum.each(errors, fn ex ->
  IO.puts("#{ex.request.method} #{ex.request.path} -> #{ex.response.status}")
  IO.puts("Response: #{ex.response.body}")
end)

Pattern 3: Calculate Response Times

{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap")

response_times = Enum.map(complete, fn ex ->
  duration_ms = DateTime.diff(ex.response_timestamp, ex.request_timestamp, :millisecond)

  %{
    path: ex.request.path,
    method: ex.request.method,
    duration_ms: duration_ms
  }
end)

# Find slow requests
slow = Enum.filter(response_times, & &1.duration_ms > 1000)

Pattern 4: Analyze gRPC Traffic

HTTP/2 is the transport for gRPC. Use trailers to get gRPC status:

{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap", port: 50051)

grpc_calls = Enum.map(complete, fn ex ->
  grpc_status = ex.response.trailers && ex.response.trailers.regular["grpc-status"]
  grpc_message = ex.response.trailers && ex.response.trailers.regular["grpc-message"]

  %{
    service_method: ex.request.path,  # e.g., "/myservice.MyService/MyMethod"
    grpc_status: grpc_status,
    grpc_message: grpc_message,
    content_type: ex.request.headers.regular["content-type"]
  }
end)

Pattern 5: Group by Stream

{:ok, complete, incomplete} = PcapFileEx.HTTP2.analyze("capture.pcap")

all_exchanges = complete ++ Enum.map(incomplete, & &1)

by_stream = Enum.group_by(all_exchanges, & &1.stream_id)

Enum.each(by_stream, fn {stream_id, exchanges} ->
  IO.puts("Stream #{stream_id}: #{length(exchanges)} exchange(s)")
end)

Mid-Connection Capture

When capture starts after the HTTP/2 connection is established:

Limitations

1. Client identification: Falls back to stream ID semantics (odd = client-initiated)
2. HPACK dynamic table: May have missing entries (static table always works)
3. SETTINGS frames: Deferred until client is identified

Best Practices

{:ok, complete, incomplete} = PcapFileEx.HTTP2.analyze("mid_connection.pcap")

# Expect more incomplete exchanges in mid-connection captures
IO.puts("Complete: #{length(complete)}, Incomplete: #{length(incomplete)}")

# Some headers may be missing due to HPACK state
Enum.each(complete, fn ex ->
  # Check for missing headers
  if is_nil(ex.request.method) do
    IO.puts("Warning: Stream #{ex.stream_id} missing method (HPACK state issue)")
  end
end)

Filtering by Port

Filter to specific HTTP/2 ports:

# Standard h2c port
{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap", port: 80)

# Custom port
{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap", port: 8080)

# gRPC port
{:ok, complete, _} = PcapFileEx.HTTP2.analyze("capture.pcap", port: 50051)

Testing HTTP/2 Code

Generating Test Fixtures

Use the provided capture script:

cd test/fixtures
./capture_http2_traffic.sh
# Generates: http2_sample.pcap, http2_sample.pcapng

Requirements:

• Python 3 with h2 library (pip install h2)
• Wireshark's dumpcap

Synthetic Segments for Unit Tests

For unit tests, create synthetic segments instead of using real PCAPs:

# Connection preface
@preface "PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n"

# Build a frame
defp frame(type, flags, stream_id, payload) do
  type_byte = case type do
    :data -> 0x00
    :headers -> 0x01
    :settings -> 0x04
    # ...
  end

  length = byte_size(payload)
  <<length::24, type_byte::8, flags::8, 0::1, stream_id::31, payload::binary>>
end

# Create segment
defp segment(flow_key, direction, data, timestamp \\ DateTime.utc_now()) do
  %{
    flow_key: flow_key,
    direction: direction,
    data: data,
    timestamp: timestamp
  }
end

# Example test
test "simple GET request" do
  flow_key = {{{127, 0, 0, 1}, 50000}, {{127, 0, 0, 1}, 8080}}

  # Use HPACK indexed representations for headers
  # Index 2 = :method GET, Index 4 = :path /, Index 6 = :scheme http
  request_headers = <<0x82, 0x84, 0x86>>
  response_headers = <<0x88>>  # Index 8 = :status 200

  segments = [
    segment(flow_key, :a_to_b, @preface),
    segment(flow_key, :a_to_b, frame(:settings, 0, 0, <<>>)),
    segment(flow_key, :b_to_a, frame(:settings, 0, 0, <<>>)),
    segment(flow_key, :a_to_b, frame(:headers, 0x05, 1, request_headers)),
    segment(flow_key, :b_to_a, frame(:headers, 0x04, 1, response_headers)),
    segment(flow_key, :b_to_a, frame(:data, 0x01, 1, "Hello"))
  ]

  {:ok, complete, _} = PcapFileEx.HTTP2.analyze_segments(segments)

  assert length(complete) == 1
  [ex] = complete
  assert ex.request.method == "GET"
  assert ex.response.status == 200
end

HPACK Static Table Indices

Common HPACK static table indices for testing:

Use indexed representation: <<0x80 | index>> (e.g., <<0x82>> for GET)

Performance Considerations

Large Captures

For large PCAP files, HTTP/2 analysis processes all TCP flows:

# Filter by port to reduce processing
{:ok, complete, _} = PcapFileEx.HTTP2.analyze("huge.pcap", port: 8080)

Memory Usage

Exchanges are accumulated in memory. For very large captures with many exchanges, consider processing incrementally or filtering.

Common Mistakes

Mistake 1: Expecting TLS HTTP/2

# DON'T: Expect h2 (TLS) to work
{:ok, _, _} = PcapFileEx.HTTP2.analyze("https_traffic.pcap")
# Returns empty - can't decrypt TLS!

# DO: Use cleartext h2c captures
{:ok, complete, _} = PcapFileEx.HTTP2.analyze("h2c_traffic.pcap")

Mistake 2: Ignoring Incomplete Exchanges

# DON'T: Only check complete exchanges
{:ok, complete, _incomplete} = PcapFileEx.HTTP2.analyze("capture.pcap")

# DO: Check both for full picture
{:ok, complete, incomplete} = PcapFileEx.HTTP2.analyze("capture.pcap")
IO.puts("Complete: #{length(complete)}, Incomplete: #{length(incomplete)}")

Mistake 3: Assuming Headers Exist

# DON'T: Assume all headers present (may fail for mid-connection)
ex.request.headers.regular["content-type"]

# DO: Guard against nil
content_type = ex.request.headers && ex.request.headers.regular["content-type"]

Mistake 4: Wrong Frame Flags in Tests

# DON'T: Forget END_HEADERS flag (headers incomplete!)
frame(:headers, 0x01, 1, headers)  # Only END_STREAM

# DO: Include END_HEADERS (0x04)
frame(:headers, 0x05, 1, headers)  # END_STREAM (0x01) + END_HEADERS (0x04)

HTTP/2 Error Codes

Reference for RST_STREAM and GOAWAY error codes:

Summary: HTTP/2 Best Practices

1. Use analyze/2 for PCAP files, analyze_segments/2 for pre-parsed segments
2. Check both complete and incomplete exchanges for full picture
3. Filter by port for large captures with mixed traffic
4. Use decoded_body for auto-decoded JSON/text/multipart content
5. Set decode_content: false when you need raw binary bodies
6. Handle mid-connection captures gracefully (expect HPACK issues)
7. Use HPACK static table indices for test fixtures
8. Include END_HEADERS flag (0x04) in test HEADERS frames
9. Check for nil headers when processing exchanges
10. Use trailers for gRPC status codes