Protobuf parsing in Python

Protocol Buffers provides a compact, efficient binary format for structured data, making it suitable for APIs and inter-machine communication. The post introduces Protobuf through a Python metrics example and explains how to serialize and deserialize messages. It also shows how to stream multiple messages by prefixing each with its length, since Protobuf messages are not inherently self-delimiting.

Protocol Buffers Basics

• A .proto file defines the structure of a message.
• The example Metric message contains:
  • A name
  • A type
  • A floating-point value
  • Repeated string tags
• The protoc compiler generates language-specific code, such as Python’s metric_pb2.py.
• Python can serialize a message with SerializeToString() and restore it with ParseFromString().

Streaming Multiple Messages

• A single Protobuf message can be parsed directly, but consecutive messages need delimiters.
• Protobuf does not automatically indicate where one message ends and the next begins.
• The recommended approach is to prepend each serialized message with its byte length.
• The length is encoded as a Varint, which uses fewer bytes for smaller integers.
• This mirrors Java’s writeDelimitedTo and parseDelimitedFrom behavior and is also how the kube-state-metrics API chains messages.

Varints and Python Implementation

• Python’s Protobuf library does not provide public convenience methods for delimited messages.
• The implementation uses internal helpers:
  • _VarintBytes to encode message lengths
  • _DecodeVarint32 to read them
• Serialization writes the length followed by the message bytes.
• Deserialization reads the length, extracts the corresponding byte range, and parses it as a Metric.
• The example loads the entire stream into memory, though a production implementation could process data incrementally.

For APIs that exchange sequences of structured records, length-prefixed Protobuf messages offer an efficient and interoperable alternative to plain-text formats. Teams should account for message framing explicitly and use generated code plus appropriate streaming logic when handling multiple messages.