Data Exchange Formats: JSON, XML, Protobuf, Avro, and MessagePack

A message broker and an HTTP server move bytes. The contract between producer and consumer gives those bytes meaning: which fields exist, what they mean, and how that meaning can evolve.

This makes format selection more than “text versus binary.” Four questions matter more:

1. How is the schema defined?
2. Which changes remain compatible with old clients and stored data?
3. How will an engineer inspect a production message?
4. Does another encoding produce a measurable benefit for this payload and stack?

A compact format map

Format	Schema	Main strength	Main cost
JSON	Optional: JSON Schema or OpenAPI	Familiar tools and human-readable payloads	The contract can remain informal
XML	Optional: XSD	Mature schema and namespace tooling	Verbosity and tooling complexity
YAML	Depends on the use case	Human-authored configuration and specs	Parser ambiguity and awkward wire behaviour
Protobuf	.proto, field numbers, generated code	Compact contracts for typed clients	Code generation and compatibility discipline
Avro	Writer schema resolved against reader schema	Schema evolution for streams and files	Binary data is unusable without the correct schema
MessagePack	No built-in schema	A binary model close to JSON	Contract evolution remains an application concern

JSON can have strict validation, while a binary encoding does not make business data correct by itself. A schema validates structure and types; application invariants still belong in the application.

JSON, XML, and YAML serve different roles

JSON remains a practical default for public and internal HTTP APIs. It is easy to inspect, replay with standard tools, and support across languages. Long-lived contracts still benefit from OpenAPI or JSON Schema, automated compatibility checks, and an explicit rule for unknown fields.

XML is not limited to SOAP. XSD can describe structure, types, and constraints, while namespaces combine vocabularies. XML does not enforce validation on its own; the receiver must apply the schema and handle schema versions deliberately.

YAML is better suited to configuration and human-authored specifications such as OpenAPI. It rarely provides an advantage as a transport encoding. Parser versions and settings can interpret scalar values differently, and indentation raises the cost of an accidental edit.

“Binary” does not automatically mean “faster”

Protobuf, Avro, and MessagePack often reduce payload size and parsing work. The end-to-end result depends on the data, libraries, language, compression, and downstream processing. For a small HTTP response, network setup and business logic may dominate serialisation time.

I therefore benchmark real messages rather than quote somebody else’s chart. Useful measurements include:

• size before and after the production compression setting;
• CPU and memory for writing and reading;
• p95 and p99 for the whole request or pipeline;
• engineering time for client generation, inspection, and schema updates;
• the behaviour of an old consumer receiving a new message.

The last item usually matters most.

Protobuf: field numbers are the wire contract

A small schema looks like this:

syntax = "proto3";

message Message {
  string sender = 1;
  string text = 2;
}

The binary format identifies fields by number rather than name. The core evolution rules follow from that design:

• adding a field with a new number is normally wire-safe;
• changing an existing field number is not safe;
• a removed number must not acquire a new meaning and should be marked reserved;
• wire compatibility does not guarantee generated-code or business-semantic compatibility;
• type changes have specific rules and are not universally safe.

Protobuf is a natural fit for gRPC and typed internal contracts. Its value is not only compactness: the schema becomes version-controlled source code.

Avro: writer schema meets reader schema

Avro is common in streaming and batch pipelines. Binary data is read using both the schema that wrote it and the schema expected by the reader.

{
  "type": "record",
  "name": "Message",
  "fields": [
    { "name": "sender", "type": "string" },
    { "name": "text", "type": "string" }
  ]
}

“Avro supports schema evolution” does not mean that every change is safe. A new reader can handle an added field when the reader schema defines an appropriate default for data written before that field existed. Renames, enum changes, and incompatible types need their own migration strategy.

Kafka deployments often keep schemas in a registry and enforce a compatibility policy when a schema is published. Kafka itself does not choose the payload format: JSON, Avro, and Protobuf are all byte arrays to the broker.

MessagePack: compact values without a schema system

MessagePack encodes numbers, strings, arrays, maps, and binary values. It can be straightforward to adopt where the application already works with JSON-like structures.

It does not include a schema-evolution mechanism. A map with string field names still transmits those names in every message. Replacing the map with positional arrays may reduce size, but makes the contract less transparent and more fragile.

MessagePack is a focused option when the JSON data model is adequate and measurement shows that payload size or parsing cost has become a real constraint.

Defining compatibility

Teams should agree on the terms:

• backward compatibility: a new consumer can read old data;
• forward compatibility: an old consumer can read new data;
• full compatibility: both directions work across an agreed version range.

For events, I prefer additive evolution: add optional fields, preserve the meaning of existing fields, and update consumers before removing anything. Removal follows a deprecation period; an incompatible semantic change gets a new event type or major contract version.

A version number alone does not provide safety. The system needs automated schema checks, contract tests, and an explicit producer/consumer support matrix.

An event envelope

Alongside the domain payload, a stable envelope is useful:

{
  "event_id": "01J...",
  "event_type": "customer.updated",
  "schema_version": 3,
  "occurred_at": "2026-07-17T12:00:00Z",
  "producer": "customer-profile",
  "payload": {}
}

The ID supports tracing and deduplication. Type and version select the contract. Event time helps analyse delay and likely order. It does not establish a global order: source clocks can disagree and events can arrive late. Strict per-object ordering needs an agreed sequence or version and rules for gaps. With Protobuf or Avro, the same metadata can live directly in the schema rather than inside a JSON wrapper.

My selection process

1. Define the data lifecycle and how long old records must remain readable.
2. Choose the required backward and forward compatibility policy.
3. Check language support, tooling, registry options, and production diagnostics.
4. Measure realistic payloads.
5. Put schema review and compatibility tests into CI.

For many HTTP APIs, JSON plus a well-maintained schema is enough. Protobuf fits gRPC. Avro is useful for retained streams with readers on different schema versions. MessagePack remains a targeted choice when the JSON model works but measurements expose the cost of its text representation.

The authoritative evolution rules live in the Protocol Buffers guide, the Apache Avro specification, and the JSON Schema specification.

Exercise: add a field without breaking the reader

Consider an illustrative request event. Version one contains an identifier and a status:

{ "request_id": "r-42", "status": "accepted" }

Version two adds an optional channel:

{ "request_id": "r-42", "status": "accepted", "channel": "web" }

An old JSON validator that rejects additional properties may reject the new record. A new reader that requires channel cannot read the old one. “Additive” alone does not establish compatibility.

Pair or change	Behaviour to verify
Old reader, new record	Accepts the record while ignoring permitted unknown fields; rejection means incompatibility
New reader, old record	Handles a missing channel under an agreed rule without inventing provenance
New status value	Does not treat an unknown state as success
Archived record a year later	Remains readable with a supported schema and implementation

Put these combinations in contract tests. Verify meaning too: changing accepted from “received” to “approved” can break the business process while every parser remains happy.

---

Originally published on my Telegram channel @it_underside.

Yours, DPUPP