JSON for IoT: MQTT Sensor Payloads, AWS IoT Core, and Edge Parsing

IoT JSON Payload Design Principles

A well-designed IoT JSON payload minimizes byte count while remaining parseable by cloud rules engines, time-series databases, and human debuggers. The four core principles are: use Unix epoch integers for timestamps, keep the structure flat, include a device identifier, and encode units in key names or a metadata field. Flat structures reduce ArduinoJson StaticJsonDocument memory requirements on constrained microcontrollers — each nesting level adds overhead. Shorten key names for high-frequency payloads: "t" instead of "temperature", "h" instead of "humidity" — document the mapping in your firmware schema registry.

// Minimal IoT JSON payload — 52 bytes
{"deviceId":"sensor-42","ts":1716115200,"t":23.5,"h":62}

// Annotated full payload — 120 bytes — for non-constrained devices
{
  "deviceId":  "sensor-42",      // unique device identifier
  "ts":        1716115200,       // Unix epoch seconds (NOT ISO 8601)
  "tempC":     23.5,             // unit in key name avoids ambiguity
  "humidity":  62,               // relative humidity percent 0-100
  "pressHPa":  1013.2,           // hPa (hectopascal)
  "battery":   87,               // battery level percent
  "rssi":      -67               // WiFi/LoRa signal strength dBm
}

// Avoid: nested objects increase StaticJsonDocument size
// BAD for constrained devices:
{
  "device": { "id": "sensor-42", "battery": 87 },
  "sensors": { "temperature": { "value": 23.5, "unit": "C" } }
}

Always include a deviceId field — cloud rules engines use it to route messages and storage schemas use it as a partition key. For multi-sensor devices, use a flat structure with prefixed keys ("tempC", "tempF") rather than nested sensor objects. If you need to send multiple readings in a single MQTT publish, use an array of flat objects: [{"ts":1716115200,"t":23.5},{"ts":1716115260,"t":23.6}] — this batches readings while keeping individual objects parseable. For validation patterns, see JSON data validation.

MQTT JSON Message Format

MQTT decouples topic structure from payload format — the JSON payload is just a byte string to the broker. A well-designed topic hierarchy enables wildcard subscriptions and per-device routing without parsing the payload. The convention sensors/{deviceId}/data allows sensors/+/data to subscribe to all devices, and sensors/# to subscribe to all topics under the sensors prefix. QoS 1 (at least once delivery) is the standard choice for sensor data — QoS 0 (fire and forget) risks data loss on unreliable networks, and QoS 2 (exactly once) doubles handshake overhead without meaningful benefit for time-series data where a duplicate reading is harmless.

# Python paho-mqtt publisher — sends JSON sensor payload
import json
import time
import paho.mqtt.client as mqtt

BROKER = "broker.example.com"
DEVICE_ID = "sensor-42"
TOPIC = f"sensors/{DEVICE_ID}/data"

client = mqtt.Client(client_id=DEVICE_ID, clean_session=False)
client.username_pw_set("iot-user", "secret")
client.tls_set()  # TLS for production
client.connect(BROKER, 8883)

payload = {
    "deviceId": DEVICE_ID,
    "ts":       int(time.time()),   # Unix epoch seconds
    "tempC":    23.5,
    "humidity": 62,
    "battery":  87,
}

# QoS 1: at-least-once delivery
# retain=True: broker stores last message for new subscribers
client.publish(TOPIC, json.dumps(payload), qos=1, retain=True)

# Subscriber: wildcard for all devices
def on_message(client, userdata, msg):
    data = json.loads(msg.payload)
    print(f"Device {data['deviceId']}: temp={data['tempC']}°C")

sub_client = mqtt.Client()
sub_client.on_message = on_message
sub_client.connect(BROKER, 1883)
sub_client.subscribe("sensors/+/data", qos=1)  # + = single-level wildcard
sub_client.loop_forever()

Retained messages let new subscribers receive the most recent sensor reading immediately on connect — useful for dashboards that need current state without waiting for the next publish cycle. For large deployments, use MQTT 5.0 topic aliases to reduce the repeated topic string overhead across thousands of messages. Message expiry intervals (MQTT 5.0) prevent stale sensor readings from being delivered to late subscribers. For data format context, see OpenTelemetry JSON for a comparison with observability payloads.

AWS IoT Core JSON Rules

AWS IoT Core rules engine evaluates SQL statements against the JSON payload of every incoming MQTT message and routes matching messages to AWS services without a server. The SQL dialect supports JSON path expressions, functions (timestamp(), topic(), clientid()), and WHERE clauses for filtering. Rules are defined in the IoT Core console or via CloudFormation and execute within milliseconds of message arrival.

-- Route all sensor readings to DynamoDB
SELECT
  deviceId,
  ts           AS timestamp,
  tempC        AS temperature,
  humidity,
  topic(2)     AS deviceIdFromTopic,   -- extracts segment from topic path
  timestamp()  AS awsTimestamp         -- AWS server-side timestamp ms
FROM 'sensors/+/data'
WHERE tempC IS NOT NULL

-- Alert rule: high temperature to SNS
SELECT deviceId, tempC, ts
FROM 'sensors/+/data'
WHERE tempC > 40

-- Store raw JSON to S3 (full payload)
SELECT * FROM 'sensors/#'

-- Device Shadow document structure
-- Published by device to: $aws/things/sensor-42/shadow/update
{
  "state": {
    "reported": {
      "tempC":    23.5,
      "humidity": 62,
      "fanSpeed": 1
    }
  }
}

-- Desired state update (from cloud application)
-- Published to: $aws/things/sensor-42/shadow/update
{
  "state": {
    "desired": {
      "fanSpeed": 2
    }
  }
}

-- Delta document (IoT Core publishes when reported != desired)
-- Delivered to: $aws/things/sensor-42/shadow/update/delta
{
  "state": {
    "fanSpeed": 2
  },
  "metadata": { "fanSpeed": { "timestamp": 1716115200 } },
  "version": 12,
  "timestamp": 1716115210
}

Device Shadow documents decouple device connectivity from application state — the cloud application reads and writes the shadow regardless of whether the device is online. When the device reconnects, it receives the delta document and applies any pending desired changes. Named shadows (MQTT 5.0 feature on AWS) allow a single device to maintain multiple independent state domains, for example one shadow for sensor config and another for actuator control. For JSON validation patterns used in cloud pipelines, see JSON data validation.

Azure IoT Hub JSON Schema

Azure IoT Hub accepts JSON payloads from devices over MQTT, AMQP, or HTTPS. The device-to-cloud (D2C) message body is free-form JSON — IoT Hub does not enforce a schema on the payload, but message routing queries can filter on body properties using $body expressions or on application properties set as message headers. IoT Central (the managed SaaS layer on top of IoT Hub) does enforce a JSON schema defined by device templates and PnP (Plug and Play) interfaces.

// Azure IoT Hub D2C message — Python azure-iot-device SDK
from azure.iot.device import IoTHubDeviceClient, Message
import json, time

CONNECTION_STRING = "HostName=hub.azure-devices.net;DeviceId=sensor-42;SharedAccessKey=..."
client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)
client.connect()

payload = {
    "deviceId":  "sensor-42",
    "ts":        int(time.time()),
    "tempC":     23.5,
    "humidity":  62,
}
msg = Message(json.dumps(payload), content_encoding="utf-8", content_type="application/json")

# Application properties for routing (accessible without parsing body)
msg.custom_properties["sensorType"] = "environmental"
msg.custom_properties["location"]   = "building-A"

client.send_message(msg)

// IoT Hub routing query — filter by body field (requires content_type=application/json)
// Route to storage when temp exceeds threshold:
// $body.tempC > 35 AND $body.humidity < 20

// IoT Central JSON — PnP interface format (DTDL component)
{
  "thermostat": {
    "__t": "c",               // component marker required by PnP
    "temperature": 23.5,
    "targetTemperature": 22.0
  },
  "humidity": 62,
  "timestamp": "2025-05-19T14:00:00Z"   // ISO 8601 required by IoT Central
}

IoT Central diverges from raw IoT Hub in one important way: it requires ISO 8601 timestamps in the JSON body rather than Unix epochs, because the device template schema defines timestamp fields as dateTime. When targeting both IoT Hub (direct) and IoT Central, include both formats: "ts": 1716115200 for routing queries and "timestamp": "2025-05-19T14:00:00Z" for IoT Central telemetry ingestion. Message routing queries on IoT Hub can extract and filter on body fields without Lambda — use this to fan out D2C messages to multiple endpoints (Event Hub, Service Bus, Blob Storage) based on JSON content.

Edge JSON Parsing on ESP32/Arduino

ArduinoJson is the standard JSON library for ESP32, ESP8266, Arduino Uno, and Raspberry Pi Pico. It provides both serialization and deserialization with zero dynamic allocation when using StaticJsonDocument. The document capacity (in bytes) must be set at compile time — use the ArduinoJson Assistant at arduinojson.org/v6/assistant/ to calculate the required capacity for your schema. A rule of thumb: allocate 1.5× the maximum expected JSON string length.

#include <ArduinoJson.h>
#include <PubSubClient.h>  // MQTT library

// Serialize: build and publish a JSON sensor reading
void publishSensorData(PubSubClient& mqttClient, const char* deviceId,
                       float tempC, int humidity) {
  // StaticJsonDocument: fixed memory on stack (no heap allocation)
  // 256 bytes handles: deviceId(30) + ts(10) + tempC(5) + humidity(3) + keys/overhead
  StaticJsonDocument<256> doc;

  doc["deviceId"] = deviceId;
  doc["ts"]       = (unsigned long)time(nullptr);  // Unix epoch
  doc["tempC"]    = serialized(String(tempC, 1));  // 1 decimal place
  doc["humidity"] = humidity;
  doc["battery"]  = analogRead(A0) * 100 / 4095;  // 0-100%

  char buf[256];
  size_t len = serializeJson(doc, buf);

  char topic[64];
  snprintf(topic, sizeof(topic), "sensors/%s/data", deviceId);
  mqttClient.publish(topic, buf, len);
}

// Deserialize: parse incoming JSON command (e.g., desired shadow delta)
void handleCommand(const char* payload, unsigned int length) {
  StaticJsonDocument<128> doc;
  DeserializationError err = deserializeJson(doc, payload, length);

  if (err) {
    Serial.print("JSON parse error: ");
    Serial.println(err.c_str());  // IncompleteInput, NoMemory, InvalidInput, etc.
    return;
  }

  // Access fields — returns 0/null if missing (safe, no exception)
  int fanSpeed = doc["fanSpeed"] | 0;         // default 0 if key missing
  bool ledOn   = doc["led"]      | false;
  float setpt  = doc["setpoint"] | 22.0f;

  if (fanSpeed > 0) setFanSpeed(fanSpeed);
  if (ledOn)        digitalWrite(LED_PIN, HIGH);
}

// Memory budget on ESP32 (240KB SRAM):
// StaticJsonDocument<512>  →  512 bytes stack
// StaticJsonDocument<2048> →  2KB  stack — safe for most use cases
// DynamicJsonDocument      →  heap allocation — use when size is variable

The DeserializationError variants to handle are: IncompleteInput (truncated MQTT payload), NoMemory (document capacity too small — increase StaticJsonDocument size), and InvalidInput (malformed JSON). Use doc["key"] | defaultValue to safely access optional fields without a null check. For producing JSON from sensor readings, prefer serializeJson(doc, mqttClient) which streams directly to the MQTT client buffer instead of an intermediate char array — saves stack memory on memory-constrained MCUs like Arduino Uno (2KB SRAM). For broader JSON compression options, see JSON compression.

JSON Compression for IoT

JSON's verbosity — repeated key names, string delimiters, and textual number representation — makes it inefficient for bandwidth-constrained IoT links: LoRaWAN (51–222 bytes per packet), NB-IoT (1,400 bytes recommended max), and Sigfox (12 bytes uplink). Three compression strategies apply at different points in the stack: binary format substitution (CBOR, MessagePack), key shortening, and delta/batch encoding.

// JSON vs CBOR size comparison — same sensor payload

// JSON: 57 bytes
{"deviceId":"sensor-42","ts":1716115200,"tempC":23.5,"h":62}

// CBOR: ~38 bytes (33% smaller) — binary, not human-readable
// Encoding: map(4) + text(8)"deviceId" + text(9)"sensor-42"
//           + text(2)"ts" + uint(1716115200)
//           + text(5)"tempC" + float16(23.5)
//           + text(1)"h" + uint(62)

// MessagePack: ~40 bytes — similar to CBOR, better library support

// Key shortening — 37 bytes (35% smaller, still JSON)
{"d":"sensor-42","t":1716115200,"c":23.5,"h":62}

// Delta encoding — only send changed values
// Message 1 (baseline): {"d":"sensor-42","t":1716115200,"c":23.5,"h":62}
// Message 2 (delta):    {"d":"sensor-42","t":1716115260,"c":23.6}
//   — humidity unchanged, omitted

// LoRaWAN-optimized: binary packed (no JSON)
// 8 bytes: 4-byte timestamp + 2-byte temp (×10, signed) + 2-byte humidity
// Requires custom encoder/decoder on both ends

# Python CBOR encoding (pip install cbor2)
import cbor2, json

payload = {"deviceId": "sensor-42", "ts": 1716115200, "tempC": 23.5, "h": 62}
json_bytes = json.dumps(payload).encode()  # 57 bytes
cbor_bytes = cbor2.dumps(payload)          # ~38 bytes

print(f"JSON: {len(json_bytes)}B  CBOR: {len(cbor_bytes)}B  "
      f"savings: {(1 - len(cbor_bytes)/len(json_bytes))*100:.0f}%")

CBOR is the IETF standard (RFC 8949) binary format with the same data model as JSON — all JSON types map directly to CBOR types. It is supported in AWS IoT Core (as binary payload with a Lambda decoder) and in CoAP (the IoT alternative to HTTP). MessagePack is an alternative with broader embedded library support and similar size savings. For most WiFi-connected ESP32 projects, key shortening alone (renaming "temperature" to "t") achieves 20–30% size reduction with zero library overhead. For a thorough comparison, see JSON vs CBOR and JSON compression.

Time-Series JSON Storage

IoT sensor data is inherently time-series: high write throughput, time-based queries, and retention policies. Two dominant storage options for JSON-originated IoT data are InfluxDB (purpose-built TSDB with line protocol) and TimescaleDB (PostgreSQL extension with JSONB support). Both accept JSON-originated data but differ in ingestion format and query language.

# InfluxDB line protocol — convert JSON to line protocol before writing
# Format: measurement,tag_key=tag_val field_key=field_val timestamp_ns

# JSON payload received from MQTT:
# {"deviceId":"sensor-42","ts":1716115200,"tempC":23.5,"humidity":62}

# Converted to InfluxDB line protocol (nanosecond timestamp):
sensors,deviceId=sensor-42 tempC=23.5,humidity=62i 1716115200000000000

# Python: batch write JSON readings to InfluxDB
from influxdb_client import InfluxDBClient, Point, WritePrecision
from influxdb_client.client.write_api import SYNCHRONOUS
import json

client = InfluxDBClient(url="http://localhost:8086", token="...", org="iot")
write_api = client.write_api(write_options=SYNCHRONOUS)

def write_sensor_json(payload_str: str):
    data = json.loads(payload_str)
    point = (
        Point("sensors")
        .tag("deviceId", data["deviceId"])
        .field("tempC",    data["tempC"])
        .field("humidity", data["humidity"])
        .time(data["ts"], WritePrecision.SECONDS)
    )
    write_api.write(bucket="iot-data", record=point)

# TimescaleDB — store JSONB, query with -> and ->> operators
-- Schema
CREATE TABLE sensor_readings (
  time       TIMESTAMPTZ NOT NULL,
  device_id  TEXT        NOT NULL,
  payload    JSONB       NOT NULL
);
SELECT create_hypertable('sensor_readings', 'time');
CREATE INDEX ON sensor_readings (device_id, time DESC);
CREATE INDEX ON sensor_readings ((payload->>'tempC'));  -- expression index

-- Insert from JSON string
INSERT INTO sensor_readings
VALUES (
  to_timestamp(1716115200),
  'sensor-42',
  '{"tempC":23.5,"humidity":62}'::jsonb
);

-- Query: last hour average temperature per device
SELECT
  device_id,
  AVG((payload->>'tempC')::float)  AS avg_temp,
  MAX((payload->>'tempC')::float)  AS max_temp
FROM sensor_readings
WHERE time > NOW() - INTERVAL '1 hour'
GROUP BY device_id
ORDER BY avg_temp DESC;

InfluxDB excels at high-frequency writes (millions of points per second) and built-in downsampling tasks — define a Flux task to compute hourly averages and write them to a separate bucket for long-term retention. TimescaleDB is the better choice when you need relational joins (e.g. device metadata table), SQL familiarity, and JSONB flexibility for variable-schema payloads. For both, batch writes in groups of 100–1,000 points: individual per-message HTTP writes at 1,000 messages/hour create unnecessary overhead. A single batched POST every 60 seconds with 60 readings is far more efficient. For broader JSON storage patterns, see OpenTelemetry JSON for time-series telemetry storage comparison.

Definitions

MQTT

Message Queuing Telemetry Transport — a lightweight publish/subscribe messaging protocol designed for constrained devices and low-bandwidth networks. MQTT uses a broker (e.g. Mosquitto, AWS IoT Core, HiveMQ) to route messages from publishers to subscribers by topic. JSON is the most common payload format for MQTT messages in IoT applications.

QoS (Quality of Service)

MQTT delivery guarantee levels: QoS 0 (at most once — fire and forget, no acknowledgement), QoS 1 (at least once — broker acknowledges, may deliver duplicates), and QoS 2 (exactly once — four-part handshake, highest overhead). QoS 1 is the standard choice for IoT sensor data: reliable delivery with acceptable overhead, and duplicate readings are harmless for time-series storage.

Device Shadow

An AWS IoT Core feature that maintains a persistent JSON document representing the last-known state (reported) and desired state (desired) of a device. The shadow persists when the device is offline. When the device reconnects, it receives a delta document containing the difference between reported and desired states, enabling reliable bidirectional device control.

Rules Engine

The AWS IoT Core component that evaluates SQL-like statements against incoming MQTT JSON payloads and routes matching messages to AWS services. Rules support JSON path expressions, built-in functions (timestamp(), topic()), WHERE clause filtering, and multiple simultaneous actions (e.g. write to DynamoDB AND trigger Lambda AND store to S3) from a single rule.

ArduinoJson

The most widely used JSON library for Arduino, ESP32, and ESP8266 microcontrollers. It provides StaticJsonDocument (fixed stack allocation) and DynamicJsonDocument (heap allocation) with deserialize and serialize functions. A 200-byte JSON payload deserializes in approximately 1ms on ESP32 at 240MHz with a 512-byte StaticJsonDocument.

CBOR

Concise Binary Object Representation (RFC 8949) — a binary data format with the same data model as JSON (numbers, strings, arrays, maps, booleans, null). CBOR encodes the same IoT payload 30–40% smaller than JSON by eliminating string key repetition and using compact binary number encoding. CBOR is supported in CoAP, LwM2M, and as a binary MQTT payload with cloud-side decoder.

Time-Series Database

A database optimized for storing and querying data points indexed by timestamp. Purpose-built TSDBs like InfluxDB use columnar storage, automatic compression, and downsampling to handle millions of IoT sensor writes per second efficiently. TimescaleDB extends PostgreSQL with hypertables that partition time-series data automatically, enabling standard SQL queries on JSONB sensor payloads.

Frequently asked questions

What JSON format should I use for IoT sensor data?

Use a flat JSON object with a deviceId string, a Unix epoch integer timestamp (not ISO 8601), and numeric sensor readings with consistent key names. A minimal example: {"deviceId":"sensor-42","ts":1716115200,"tempC":23.5,"humidity":62}. Unix epoch integers save 12 bytes over ISO 8601 strings and are trivially comparable in SQL and time-series queries. Keep the structure flat — avoid nested objects for simple telemetry — to minimize ArduinoJson StaticJsonDocument capacity on constrained devices. Encode units in key names ("tempC", "pressHPa") to prevent ambiguity. For validation of your schema, see JSON data validation.

How do I send JSON over MQTT?

Serialize your JSON object to a string and publish it as the MQTT message payload. In Python with paho-mqtt: client.publish("sensors/sensor-42/data", json.dumps(payload), qos=1). On ESP32 with ArduinoJson: StaticJsonDocument<256> doc; doc["tempC"] = 23.5; char buf[256]; serializeJson(doc, buf); client.publish(topic, buf). Use QoS 1 for reliable delivery of sensor readings. Set retain=True to let new subscribers receive the most recent reading on connect. Structure topics as sensors/{deviceId}/data to enable sensors/+/data wildcard subscriptions that match all devices.

How does AWS IoT Core process JSON messages?

AWS IoT Core rules engine evaluates SQL statements against incoming JSON payloads. A rule like {"SELECT * FROM 'sensors/+/data' WHERE tempC > 30"} matches any device topic and filters by the tempC field. Actions route matching messages to DynamoDB (INSERT with JSON field mapping), Lambda (full JSON payload as event), S3 (store raw JSON), Kinesis, or SNS — multiple actions can trigger from a single rule. The Device Shadow document stores reported and desired state, and IoT Core publishes a delta document to the device when they differ, enabling bidirectional device control without a server.

How do I parse JSON on an Arduino or ESP32?

Use the ArduinoJson library. Declare a StaticJsonDocument sized to your payload: StaticJsonDocument<256> doc;. Deserialize: DeserializationError err = deserializeJson(doc, payload);. Check for errors: if err == DeserializationError::NoMemory, increase the document size. Access fields with safe defaults: float temp = doc["tempC"] | 0.0f; — returns the default value if the key is missing. A 200-byte JSON payload deserializes in approximately 1ms on ESP32. For publishing, use serializeJson(doc, buf); then client.publish(topic, buf);.

Should I use JSON or CBOR for IoT devices?

Use JSON for WiFi-connected or cellular devices where cloud tooling compatibility and human debuggability matter. Use CBOR when bandwidth is severely constrained (LoRaWAN, Sigfox, NB-IoT) — CBOR reduces payload size by 30–40% with no change to the data model. CBOR requires library support on both device (tinycbor, cbor-cpp) and cloud (a decoder Lambda before IoT Core rules engine). For ESP32 projects sending data every few seconds over WiFi or LTE, JSON with key shortening achieves 20–30% size savings with zero library overhead — a practical middle ground. See JSON vs CBOR for a detailed comparison.

How do I store IoT JSON data in a time-series database?

For InfluxDB, convert JSON to line protocol: sensors,deviceId=sensor-42 tempC=23.5,humidity=62i 1716115200000000000 (timestamp in nanoseconds). Use the InfluxDB client library to batch 100–1,000 points per write request. For TimescaleDB, create a hypertable with a JSONB column and use expression indexes on frequently queried fields: CREATE INDEX ON sensor_data ((payload->>'tempC'));. Query with payload->>'tempC' to extract field values as text, cast to numeric for aggregation. Always batch writes — individual per-message writes at 1,000 messages/hour create unnecessary HTTP overhead.

What is AWS IoT Device Shadow JSON format?

The Device Shadow document has a state object with reported (device-updated) and desired (application-updated) sub-objects: {"state":{"reported":{"tempC":23.5,"fanSpeed":1},"desired":{"fanSpeed":2}}}. When reported and desired differ, AWS IoT Core publishes a delta document to $aws/things/{name}/shadow/update/delta containing only the differing fields. The device subscribes to this topic, applies changes, then publishes the updated reported state. Shadows include a version integer that increments on each update — use it to detect stale updates and prevent race conditions.

How do I minimize JSON payload size for battery-powered IoT devices?

Apply these techniques in order of impact: (1) Use Unix epoch integers instead of ISO 8601 strings — saves 12 bytes per message. (2) Shorten key names: "t" instead of "temperature", "h" instead of "humidity" — document the mapping in firmware. (3) Round to useful precision: 23.5 instead of 23.500001. (4) Send only changed values (delta encoding). (5) Batch multiple readings into a JSON array before publishing. (6) Switch to CBOR for 30–40% further size reduction. A minimal optimized sensor payload can be under 40 bytes: {"ts":1716115200,"t":23.5,"h":62} — compared to 120 bytes for the verbose equivalent. See JSON compression for more techniques.