Overviewerlang logo

Module gpb

This module contains a data-driven encoder/decoder plus some auxiliary helper functions.

Description

This module contains a data-driven encoder/decoder plus some auxiliary helper functions.

For information on how to generate code for a set of .proto files, see the gpb_compile module.

The reason for keeping this (slower) data-driven encoder/decoder is mostly to be able to cross-check with the generated encoder/decoder. and also to some extent for easier inter-op testing.

The encoder/decoder in this module works on records or tuples, but the functions msg_to_map/3 and msg_from_map/4 can convert to and from maps.

A note about format versions of definitions: Functions in this module that take definitions, such as encode_msg/2 and decode_msg/3 expect the definitions to be on the latest format version. If you use functions in gpb_compile to parse proto files to definitions, for example gpb_compile:file/2, then make sure to include the option {proto_defs_version, gpb_defs:latest_defs_version()}. This is in contrast to the advice in the [doc/dev-guide/proto-defs-versions.md] to tool writers to use a hard-wired version number, but this module is part of gpb and thus developed in tandem with any changes to the definitions, whereas the definitions returned from gpb_compile can occasionally default to an earlier-than-latest version for compatibility reasons.

Data Types

field()

field() = 
    #field{name = atom() | undefined,
           fnum = integer() | undefined,
           rnum = pos_integer() | undefined,
           type =
               gpb_field_type() |
               gpb_internal_intermediary_ref() |
               gpb_internal_intermediary_map_ref() |
               undefined,
           occurrence =
               required | optional | repeated | defaulty |
               undefined,
           opts = [term()]} |
    #gpb_oneof{name = atom() | undefined,
               rnum = pos_integer() | undefined,
               fields =
                   [#field{name = atom() | undefined,
                           fnum = integer() | undefined,
                           rnum = pos_integer() | undefined,
                           type =
                               gpb_field_type() |
                               gpb_internal_intermediary_ref() |
                               gpb_internal_intermediary_map_ref() |
                               undefined,
                           occurrence =
                               required | optional | repeated |
                               defaulty | undefined,
                           opts = [term()]}] |
                   undefined,
               opts = [term()]}

gpb_field_type()

gpb_field_type() = 
    int32 | int64 | uint32 | uint64 | sint32 | sint64 | fixed32 |
    fixed64 | sfixed32 | sfixed64 | bool | float | double |
    string | bytes |
    {enum, atom()} |
    {msg, atom()} |
    {group, atom()} |
    {map, gpb_map_key(), gpb_map_value()} |
    unknown

gpb_internal_intermediary_map_ref()

gpb_internal_intermediary_map_ref() = 
    {map,
     gpb_map_key(),
     gpb_map_value() | gpb_internal_intermediary_ref()}

gpb_internal_intermediary_ref()

gpb_internal_intermediary_ref() = 
    {ref, term()} |
    {msg, list()} |
    {group, list()} |
    {enum, list()}

gpb_map_key()

gpb_map_key() = 
    int32 | int64 | uint32 | uint64 | sint32 | sint64 | fixed32 |
    fixed64 | sfixed32 | sfixed64 | bool | string

gpb_map_value()

gpb_map_value() = gpb_scalar() | {enum, atom()} | {msg, atom()}

gpb_scalar()

gpb_scalar() = 
    int32 | int64 | uint32 | uint64 | sint32 | sint64 | fixed32 |
    fixed64 | sfixed32 | sfixed64 | bool | float | double |
    string | bytes

map_opt()

map_opt(Other) = 
    {maps_unset_optional, omitted | present_undefined} |
    {maps_oneof, flat | tuples} |
    Other

map_opts()

map_opts(OtherOpts) = [map_opt(OtherOpts)]

map_opts()

map_opts() = map_opts(none())

proplist_def()

proplist_def() = 
    {{msg, Name :: atom()}, [proplist_field()]} | term()

proplist_defs()

proplist_defs() = [proplist_def()]

proplist_field()

proplist_field() = 
    [proplist_field_item()] | [proplist_oneof_item()]

proplist_field_item()

proplist_field_item() = 
    {name, atom()} |
    {fnum, integer()} |
    {rnum, pos_integer()} |
    {type, gpb_field_type()} |
    {occurrence, required | optional | repeated} |
    {opts, [term()]}

proplist_oneof_item()

proplist_oneof_item() = 
    {name, atom()} |
    {rnum, pos_integer()} |
    {field, [proplist_field_item()]}

proplist_rpc()

proplist_rpc() = [proplist_rpc_item()]

proplist_rpc_item()

proplist_rpc_item() = 
    {name, atom()} |
    {input, [field()]} |
    {output, [field()]} |
    {input_stream, boolean()} |
    {output_stream, boolean()} |
    {opts, [term()]}

Function Index

check_scalar/2Verify type and range of a boolean, integral, floating point, string, bytes or value.
decode_msg/3Decode a binary to a message on record format.
decode_packet/3Decode a varint-length-delimited packet.
decode_varint/1Equivalent to decode_varint(Bin, 64).
decode_varint/2Decode an unsigned varint.
decode_wiretype/1Decode an integer 0..5 to a wire type.
defs_records_to_proplists/1In definitions, convert msg field records to proplists.
encode_msg/2Encode a message on record format to a binary.
encode_varint/1Encode an unsigned varint to a binary.
encode_wiretype/1Encode a wire type for a protobuf field type.
field_record_to_proplist/1Convert one field definition on field record format to a proplist.
field_records_to_proplists/1Convert msg field definitions on field record format to proplists.
is_allowed_as_key_type/1Test whether a field type is allowed as key type in a msg<_,_> type field.
is_msg_proto3/2Test whether a message was defined with "proto3" syntax.
is_type_packable/1Test whether a field is allowed to have the option [packed] or not.
map_item_pseudo_fields/2Make a list of fields for a pseudo message, given a key type and a value type for a map<_,_> type field, for use in encoding, verification.
map_item_pseudo_fields_for_decoding/2Make a list of fields for a pseudo message, given a key type and a value type for a map<_,_> type field, for use in decoding.
merge_msgs/3Merge messages on record format.
msg_from_map/4Convert a message, on map format to a tuple, so that it can be used with eg encode_msg/2.
msg_to_map/3Convert a message, as returned by eg decode_msg/3 on tuple format to a map.
proplist_to_field_record/1Convert a proplist for an msg field to an msg field record.
proplists_to_defs_records/1In definitions, convert msg fields as proplists to field records.
proplists_to_field_records/1Convert proplists for msg fields to msg field records.
proplists_to_rpc_records/1Convert proplists to rpc records.
proto2_type_default/2Return the default for a type of a message field in a "proto2" file.
proto3_type_default/2Return the default for a type of a message field in a "proto3" file.
rpc_record_to_proplist/1Convert an rpc record to a proplist.
rpc_records_to_proplists/1Convert rpc records to proplists.
verify_msg/2Verify type and range of fields in a message on record format.
version_as_list/0Return the version on a list format, so that it can be compared.
version_as_string/0Return the version as a string.
version_source/0Retrieve the source of the version.

Function Details

check_scalar/2

check_scalar(Value :: any(), Type :: gpb_scalar()) ->
                ok | {error, Reason :: term()}

Verify type and range of a boolean, integral, floating point, string, bytes or value.

decode_msg/3

decode_msg(Bin :: binary(),
           MsgName :: atom(),
           MsgDefs :: gpb_defs:defs()) ->
              tuple()

Decode a binary to a message on record format.

If a field would occur more than once in the message, it is subject to merging, as per how protobuf is expected to work. See merge_msgs/3 for more details.

decode_packet/3

decode_packet(Type, Bin :: binary(), Opts) -> Result

Decode a varint-length-delimited packet. This function works like erlang:decode_packet/3, but the length is a varint encoded value.

The Type is either uint32 or uint64, indicating the maximum number of bits considered as length of the packet.

Available options (just as for erlang:decode_packet/3):
{packet_size, MaxNumBytes :: integer() >= 0}
This defines a maximum packet size. 0, which is the default, means no upper limit. If the encoded length indicates a packet of size larger than packet_size, then {error, invalid} will be returned instead.
This function will return:
{ok, Packet :: binary(), Rest :: binary()}
One packet was extracted in Packet, trailing data is in Rest
{more, Length} when
Length = TotalSize :: integer() >= 1 | undefined
This signifies that more data is needed to extracted one packet. If sufficient data was present to know the size of the packet, an integer is returned indicating the total number of bytes needed to extract one packet.
{error, Reason :: term()}
If the length is malformed, for instance overly many bytes are used to encode the length, then {error,invalid} will be returned. The idea is to protect against a denial of service in case an adversary would send a valid (possibly small) length encoded using a huge number of bytes.
Example: Here is an example of how to use this function to receive varint length delimited packets from a TCP/IP socket in active and binary mode. 
     receive_tcp_data(Socket, Pending) when is_binary(Pending) ->
         receive
            {tcp, Socket, IncomingBin} ->
                Data = <<Pending/binary, IncomingBin/binary>>,
                %% We may now have either a partial packet,
                %% or one packet and possibly some more,
                %% or several packets and possibly some more.
                %% Handle them in a loop:
                NewPending = handle_packets(Data),
                receive_tcp_data(Socket, NewPending)
         end.
 
     handle_packets(Data) ->
         %% In this example, we do not expect any decoding errors,
         %% for instance a malformed varint encoded length,
         %% so for simplicity fail fast on any error.
         case gpb:decode_packet(uint32, Data, []) of
             {ok, Packet, Rest} ->
                 handle_one_packet(Packet), % do some work here
                 handle_packets(Rest);
             {more, _Length} ->
                 Data
         end.

decode_varint/1

decode_varint(Bin :: binary()) ->
                 {non_neg_integer(), Rest :: binary()}

Equivalent to decode_varint(Bin, 64).

decode_varint/2

decode_varint(Bin :: binary(), MaxNumBits :: pos_integer()) ->
                 {non_neg_integer(), binary()}

Decode an unsigned varint. The decoded integer will have be at most MaxNumBits bits long. Any higher bits will be masked away. If the binary contains an overly long encoded integer, this function will fail.

decode_wiretype/1

decode_wiretype(X1 :: non_neg_integer()) ->
                   varint | bits32 | bits64 | group_start |
                   group_end | length_delimited

Decode an integer 0..5 to a wire type

defs_records_to_proplists/1

defs_records_to_proplists(Defs :: gpb_defs:defs()) ->
                             proplist_defs()

In definitions, convert msg field records to proplists.

encode_msg/2

encode_msg(Msg :: tuple(), MsgDefs :: gpb_defs:defs()) -> binary()

Encode a message on record format to a binary.

encode_varint/1

encode_varint(N :: integer()) -> binary()

Encode an unsigned varint to a binary.

encode_wiretype/1

encode_wiretype(X1 :: gpb_field_type()) -> non_neg_integer()

Encode a wire type for a protobuf field type.

field_record_to_proplist/1

field_record_to_proplist(Field ::
                             #field{name = atom() | undefined,
                                    fnum = integer() | undefined,
                                    rnum =
                                        pos_integer() | undefined,
                                    type =
                                        gpb_field_type() |
                                        gpb_internal_intermediary_ref() |
                                        gpb_internal_intermediary_map_ref() |
                                        undefined,
                                    occurrence =
                                        required | optional |
                                        repeated | defaulty |
                                        undefined,
                                    opts = [term()]}) ->
                            [proplist_field_item()]

Convert one field definition on field record format to a proplist.

field_records_to_proplists/1

field_records_to_proplists(Fields :: [field()]) ->
                              [proplist_field()]

Convert msg field definitions on field record format to proplists.

is_allowed_as_key_type/1

is_allowed_as_key_type(X1 :: gpb_field_type()) -> boolean()

Test whether a field type is allowed as key type in a msg<_,_> type field.

is_msg_proto3/2

is_msg_proto3(Name :: atom(), MsgDefs :: gpb_defs:defs()) ->
                 boolean()

Test whether a message was defined with "proto3" syntax.

If a file with "proto3" syntax import another file with "proto2" syntax, or vice versa, then some messages will be of "proto2" while others will be "proto3".

is_type_packable/1

is_type_packable(X1 :: gpb_field_type()) -> boolean()

Test whether a field is allowed to have the option [packed] or not.

map_item_pseudo_fields/2

map_item_pseudo_fields(KeyType :: gpb_map_key(),
                       ValueType :: gpb_map_value()) ->
                          [field()]

Make a list of fields for a pseudo message, given a key type and a value type for a map<_,_> type field, for use in encoding, verification.

At encoding a field value is to be included even if the proto syntax is "proto3" and the value to encode is the type's default value.

map_item_pseudo_fields_for_decoding/2

map_item_pseudo_fields_for_decoding(KeyType :: gpb_map_key(),
                                    ValueType :: gpb_map_value()) ->
                                       [field()]

Make a list of fields for a pseudo message, given a key type and a value type for a map<_,_> type field, for use in decoding.

At decoding if the field would not be present in the binary to decode, it is expected to have be the type's default value, even when the proto syntax is "proto2".

merge_msgs/3

merge_msgs(PrevMsg :: tuple(),
           NewMsg :: tuple(),
           MsgDefs :: gpb_defs:defs()) ->
              tuple()

Merge messages on record format.

Merging of messages M1 and M2 are defined in protobuf to work like this:
integers, floats, booleans, strings, bytes and enums
If the field is set in M2, it overwrites any field in M1.
Repeated fields
Elements of the field in M2 are appended to that of M1
Sub messages
Fields that are sub messages are merged recursively.
Map fields
A map is treated as a repeated field in the sense that map elements of M2 are added to M1. If a key of exists in both M1 and M2, the map element of M2 overwrites that of M1.
Oneof fields
A oneof field of M2 generally overwrites a oneof field of M1. If the oneof field is a sub message, they are merged recursively.

msg_from_map/4

msg_from_map(Map :: map(),
             MsgName :: atom(),
             Defs :: gpb_defs:defs(),
             Opts :: map_opts()) ->
                tuple()

Convert a message, on map format to a tuple, so that it can be used with eg encode_msg/2. The options control how the map representation is to be interpreted, see msg_to_map/3 for further discussion on this.

msg_to_map/3

msg_to_map(Msg :: tuple(),
           Defs :: gpb_defs:defs(),
           MapOpts :: map_opts()) ->
              map()

Convert a message, as returned by eg decode_msg/3 on tuple format to a map.

The MapOpts look similar to the options understood by gpb_compile, but is only a limited subset of them and they concern only different map representations and not the record representation. While it can support some of the map options understood by gpb_compile it does not support mapfields_as_maps for the records representation, as this is not supported by the rest of the code in this module.

proplist_to_field_record/1

proplist_to_field_record(PL :: [proplist_field_item()]) ->
                            #field{name = atom() | undefined,
                                   fnum = integer() | undefined,
                                   rnum =
                                       pos_integer() | undefined,
                                   type =
                                       gpb_field_type() |
                                       gpb_internal_intermediary_ref() |
                                       gpb_internal_intermediary_map_ref() |
                                       undefined,
                                   occurrence =
                                       required | optional |
                                       repeated | defaulty |
                                       undefined,
                                   opts = [term()]}

Convert a proplist for an msg field to an msg field record.

proplists_to_defs_records/1

proplists_to_defs_records(Defs :: proplist_defs()) ->
                             gpb_defs:defs()

In definitions, convert msg fields as proplists to field records.

proplists_to_field_records/1

proplists_to_field_records(PLs :: [proplist_field()]) -> [field()]

Convert proplists for msg fields to msg field records.

proplists_to_rpc_records/1

proplists_to_rpc_records(PLs :: [proplist_rpc()]) ->
                            [#rpc{name = atom() | undefined,
                                  input = any(),
                                  output = any(),
                                  input_stream =
                                      boolean() | undefined,
                                  output_stream =
                                      boolean() | undefined,
                                  opts = [term()] | undefined}]

Convert proplists to rpc records.

proto2_type_default/2

proto2_type_default(Type :: gpb_field_type(),
                    MsgDefs :: gpb_defs:defs()) ->
                       term()

Return the default for a type of a message field in a "proto2" file.

proto3_type_default/2

proto3_type_default(Type :: gpb_field_type(),
                    MsgDefs :: gpb_defs:defs()) ->
                       term()

Return the default for a type of a message field in a "proto3" file.

rpc_record_to_proplist/1

rpc_record_to_proplist(Rpc ::
                           #rpc{name = atom() | undefined,
                                input = any(),
                                output = any(),
                                input_stream =
                                    boolean() | undefined,
                                output_stream =
                                    boolean() | undefined,
                                opts = [term()] | undefined}) ->
                          proplist_rpc()

Convert an rpc record to a proplist.

rpc_records_to_proplists/1

rpc_records_to_proplists(Rpcs ::
                             [#rpc{name = atom() | undefined,
                                   input = any(),
                                   output = any(),
                                   input_stream =
                                       boolean() | undefined,
                                   output_stream =
                                       boolean() | undefined,
                                   opts = [term()] | undefined}]) ->
                            [proplist_rpc()]

Convert rpc records to proplists.

verify_msg/2

verify_msg(Msg :: tuple() | term(), MsgDefs :: gpb_defs:defs()) ->
              ok

Verify type and range of fields in a message on record format.

version_as_list/0

version_as_list() -> [integer() | string()]

Return the version on a list format, so that it can be compared. The version_as_list is better if you want to be able to compare versions, for instance to see if one version is larger/later than another.

For the case of non-tagged versions, this scheme often works, but is a bit of a kluge, since in erlang, numbers are the smallest of types, and a string such as "gb996fbe" cannot (and generally should not) be converted to a number. So for non-tagged versions, one should only check whether they are or are not equal, not whether one is larger/later or smaller/earlier than another.

This will return for example: 
     "2.1"             -> [2,1]
     "2.1-53-gb996fbe" -> [2,1,0,0,53,"gb996fbe"]
     "2.1.1"           -> [2,1,1]
     "2.2"             -> [2,2]
(Lists are better than tuples when doing version comparisons. For tuples this holds: {2,2} < {2,1,1}, since tuples are first compared by size then element by element for tuples of the same size. For lists, it holds instead that: [2,1,1] < [2,2].)

version_as_string/0

version_as_string() -> string()

Return the version as a string. Valid version format is: 

       <n>.<m>            % e.g. 2.1, 2.1.1, etc (any number of dots and ints)
       <n>.<m>-<o>-<text> % e.g. 2.1-53-gb996fbe means: a commit after 2.1
The format is what git describe --always --tags produces, given that all tags are always on the format <n>.<m>.

version_source/0

version_source() -> string()

Retrieve the source of the version. If it is extracted from git, it will be "git", else "file".

Overviewerlang logo

Generated by EDoc