JSON Structured Logging: pino, Python, Fields, Redaction & Correlation

What is structured JSON logging?

Structured logging replaces free-text log strings with typed key-value objects serialized as JSON. Each log entry is a self-describing document where every piece of information occupies a named field with a consistent type across all log lines. A plain text logger emits something like:

2024-01-15 10:30:00 ERROR DB timeout after 5032ms for request abc123

To extract the duration from this line, a log aggregator needs a regex such as /after (\d+)ms/ — fragile, slow, and broken by any phrasing change. The equivalent structured log line is:

{"level":"error","timestamp":"2024-01-15T10:30:00.000Z","message":"DB timeout","requestId":"abc123","durationMs":5032,"service":"api"}

Every aggregator can ingest this line directly: durationMs is a number field that supports range queries (durationMs > 1000), level is filterable by exact match, and requestId links this entry to every other log line in the same request. Three concrete gains justify switching from text to structured logs: (1) query speed — typed field lookups are 10–100× faster than regex scans; (2) correlation — a shared requestId turns a scattered log stream into a per-request trace; (3) alerting precision — alerts can fire on durationMs > 5000 AND level = "error" without brittle pattern matching.

Standard fields for every JSON log entry

Consistent field names across all services are the single most important convention for a structured logging setup — inconsistent naming breaks cross-service queries and dashboard panels. The 5 universal fields below should appear in every log entry regardless of language or framework:

Pick one casing convention — camelCase or snake_case — and apply it consistently across every service. Mixing requestId in Node.js with request_id in Python means your aggregator treats them as 2 different fields. Most teams standardize on camelCase because it matches JavaScript native style and JSON convention. To understand how JSON.stringify handles your log objects, including how undefined and circular references behave, see the stringify guide.

JSON logging in Node.js with pino

Pino is the fastest Node.js JSON logger — benchmarked at 20,000–50,000 synchronous log calls per second, roughly 5× faster than winston and 8× faster than bunyan. Its speed comes from a minimal hot path: it serializes the log object with fast-json-stringify and writes the result to stdout in a single process.stdout.write call. Pretty-printing is delegated to a separate pino-pretty process in development.

npm install pino

// logger.js
import pino from 'pino'

const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  timestamp: pino.stdTimeFunctions.isoTime, // ISO 8601 instead of Unix ms
  base: { service: 'payments-api' },        // merged into every log line
})

export default logger

// usage
import logger from './logger.js'

logger.info({ requestId: 'abc123', durationMs: 42 }, 'Request completed')
// {"level":30,"time":"2024-01-15T10:30:00.000Z","service":"payments-api","requestId":"abc123","durationMs":42,"msg":"Request completed"}

For Express, add pino-http to automatically log every request with method, URL, status code, response time, and a generated reqId:

npm install pino-http

import express from 'express'
import pinoHttp from 'pino-http'
import logger from './logger.js'

const app = express()
app.use(pinoHttp({ logger }))

app.get('/health', (req, res) => {
  req.log.info({ durationMs: 1 }, 'Health check')  // child logger with reqId bound
  res.json({ status: 'ok' })
})
// Automatic output on every request:
// {"level":30,"time":"...","reqId":"...","req":{"method":"GET","url":"/health"},"res":{"statusCode":200},"responseTime":3,"msg":"request completed"}

Pino's numeric log levels (TRACE=10, DEBUG=20, INFO=30, WARN=40, ERROR=50, FATAL=60) are more efficient to compare than strings and map directly to Datadog's and Elasticsearch's numeric severity fields. Setting level: 'warn' in production drops all INFO and DEBUG lines at the logger level — no serialization cost for suppressed levels. Validate your pino output by piping a log line into Jsonic's formatter to confirm all expected fields are present. For more on JSON output from APIs, see REST API JSON response design.

JSON logging in Python with python-json-logger

Python's standard logging module uses text formatters by default.python-json-logger replaces the formatter with JsonFormatter, which serializes each LogRecord to a JSON object on one line. It integrates with the standard logging hierarchy so all existing logger.info()calls emit JSON without any code changes beyond the configuration block.

pip install python-json-logger

# logger.py
import logging
from pythonjsonlogger import jsonlogger

def get_logger(name: str) -> logging.Logger:
    logger = logging.getLogger(name)
    if not logger.handlers:
        handler = logging.StreamHandler()
        formatter = jsonlogger.JsonFormatter(
            "%(asctime)s %(name)s %(levelname)s %(message)s",
            datefmt="%Y-%m-%dT%H:%M:%S.%fZ",
            rename_fields={"asctime": "timestamp", "levelname": "level"},
        )
        handler.setFormatter(formatter)
        logger.addHandler(handler)
        logger.setLevel(logging.INFO)
    return logger

# usage
from logger import get_logger

log = get_logger(__name__)

log.info("Request completed", extra={"requestId": "abc123", "durationMs": 42, "service": "payments-api"})
# {"timestamp":"2024-01-15T10:30:00.000000Z","name":"__main__","level":"INFO","message":"Request completed","requestId":"abc123","durationMs":42,"service":"payments-api"}

The extra dict is the standard Python mechanism for adding structured fields — every key in extra is merged into the top-level JSON object byJsonFormatter. For Django, configure the formatter in LOGGINGsettings:

# settings.py (Django)
LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "json": {
            "()": "pythonjsonlogger.jsonlogger.JsonFormatter",
            "fmt": "%(asctime)s %(name)s %(levelname)s %(message)s",
            "rename_fields": {"asctime": "timestamp", "levelname": "level"},
        }
    },
    "handlers": {
        "console": {"class": "logging.StreamHandler", "formatter": "json"}
    },
    "root": {"handlers": ["console"], "level": "INFO"},
}

For FastAPI or Flask, create the logger in application startup and bind arequest_id using Python's contextvars.ContextVar — the equivalent of Node's AsyncLocalStorage — so every log statement within a request automatically includes the ID without passing it explicitly. See the JSONL format guide for how newline-delimited JSON log streams are structured.

Redacting sensitive fields from JSON logs

Logging sensitive data — passwords, tokens, credit card numbers, personally identifiable information — creates a compliance and security risk. Pino's built-inredact option removes fields before serialization, so the sensitive value never reaches the output stream, log aggregator, or disk.

import pino from 'pino'

const logger = pino({
  level: 'info',
  redact: {
    paths: [
      'req.headers.authorization',
      'req.headers.cookie',
      'body.password',
      'body.creditCard',
      'user.ssn',
      '*.token',         // wildcard: any top-level object's "token" field
    ],
    censor: '[Redacted]',  // default — can be any string or function
  },
})

logger.info({
  req: { headers: { authorization: 'Bearer secret-token', 'content-type': 'application/json' } },
  body: { email: 'alice@example.com', password: 's3cr3t' },
}, 'Signup request')
// Output — authorization and password replaced:
// {"level":30,"time":"...","req":{"headers":{"authorization":"[Redacted]","content-type":"application/json"}},"body":{"email":"alice@example.com","password":"[Redacted]"},"msg":"Signup request"}

For Python, implement a logging.Filter that scrubs sensitive keys before the record reaches the formatter:

import logging

SENSITIVE_KEYS = {"password", "token", "authorization", "secret", "credit_card"}

class RedactingFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        for key in list(vars(record).keys()):
            if key.lower() in SENSITIVE_KEYS:
                setattr(record, key, "[Redacted]")
        return True

handler = logging.StreamHandler()
handler.addFilter(RedactingFilter())

Fields to always redact in production: Authorization and Cookierequest headers, any field named password, secret, token, apiKey, or creditCard, social security numbers, and full names + email combinations in jurisdictions covered by GDPR or CCPA. Verify redaction is working in development by inspecting raw stdout output — paste a sample log line into Jsonic and search for the sensitive value to confirm it has been replaced. For more on handling JSON in APIs, see REST API JSON response design.

Correlating JSON logs across a single HTTP request

Without correlation, 500 log lines from a busy server are an unordered stream — you cannot tell which DB query belongs to which HTTP request. A requestIdfield on every log line solves this: filter on one ID and you get an ordered trace of exactly what happened during that request. Implementing this in Node.js uses AsyncLocalStorage to propagate the ID without threading it through every function argument.

// context.js — AsyncLocalStorage store
import { AsyncLocalStorage } from 'async_hooks'
export const als = new AsyncLocalStorage()

// middleware.js — set the store at the start of each request
import { randomUUID } from 'crypto'
import { als } from './context.js'

export function requestContext(req, res, next) {
  const requestId = req.headers['x-request-id'] ?? randomUUID()
  res.setHeader('x-request-id', requestId)         // echo back to client
  als.run({ requestId }, next)
}

// logger.js — read from the store in the mixin
import pino from 'pino'
import { als } from './context.js'

export const logger = pino({
  level: 'info',
  mixin() {
    return als.getStore() ?? {}                     // adds requestId to every log line
  },
})

// app.js
import express from 'express'
import { requestContext } from './middleware.js'
import { logger } from './logger.js'

const app = express()
app.use(requestContext)

app.get('/orders/:id', async (req, res) => {
  logger.info('Fetching order')                     // auto-includes requestId
  const order = await db.orders.findById(req.params.id)
  logger.info({ durationMs: 15 }, 'Order fetched') // same requestId
  res.json(order)
})

For distributed tracing across microservices, propagate the requestId as an X-Request-ID or X-Correlation-ID HTTP header on every outbound call and read it on every inbound request. A request that passes through 3 microservices will produce log lines in 3 different services, all sharing the same ID — a single aggregator query on that ID reconstructs the full distributed trace without a dedicated tracing system. In Python, use contextvars.ContextVarfor the same pattern:

# Python correlation with contextvars
import uuid
from contextvars import ContextVar

request_id_var: ContextVar[str] = ContextVar("request_id", default="")

# FastAPI middleware
from fastapi import Request

async def request_id_middleware(request: Request, call_next):
    rid = request.headers.get("x-request-id") or str(uuid.uuid4())
    token = request_id_var.set(rid)
    response = await call_next(request)
    request_id_var.reset(token)
    response.headers["x-request-id"] = rid
    return response

# In your logger extra
log.info("Order fetched", extra={"requestId": request_id_var.get(), "durationMs": 15})

To understand the JSONL format used by many log shipping agents (Fluentd, Vector, Filebeat), see JSONL (newline-delimited JSON). For examples of well-structured JSON objects, see JSON examples.

Log levels, aggregators, and shipping structured logs

Log levels are the first-line filter for log volume. In production, setting the minimum level to warn (40) drops all info (30) and debug (20) lines at the logger — no serialization, no I/O cost. Use the following level guidelines:

Most log aggregators accept JSON log streams directly. Three common shipping patterns: (1) stdout to aggregator agent — run your app in a container, let Docker/Kubernetes capture stdout, and configure Fluentd, Vector, or Filebeat to tail the container log and forward to Elasticsearch or Datadog. No code changes needed. (2) pino transport — pino-datadog-transport, pino-elasticsearch, and similar packages ship logs directly from the logger without an agent. (3) CloudWatch Logs (AWS Lambda / ECS) — anything written to stdout is automatically captured; configure a CloudWatch Logs Insights query on your JSON fields: filter level = "error" | stats count() by errorCode. Always ship logs asynchronously — the pino async mode and transport workers prevent log I/O from blocking your request handler. For context on how JSON structures data, see JSON.stringify in depth.

Frequently asked questions

What is structured JSON logging and why is it better than plain text logs?

Structured JSON logging writes every log entry as a JSON object — {"level":"error","timestamp":"2024-01-15T10:30:00Z","message":"DB timeout","requestId":"abc123","durationMs":5032} — instead of a free-text string like "2024-01-15 10:30:00 ERROR DB timeout (5032ms)". The key advantage is machine readability without regex parsing. Log aggregators (Datadog, CloudWatch, Loki, Elasticsearch) can ingest JSON logs 10–100× faster because they parse fields directly. Querying is precise: filter WHERE level="error" AND durationMs>1000 runs against typed fields, not pattern-matched strings. Adding a requestId field to every log in a request enables correlation across 50–500 log lines per request, turning a scattered log stream into a coherent trace. Structured logs are also forward-compatible: adding a new field to your JSON object does not break existing queries against other fields. To explore JSON object structure fundamentals, see JSON examples.

How do I set up JSON logging in Node.js with pino?

Install pino with npm install pino, then create a logger: import pino from 'pino'; const logger = pino({ level: 'info' }); logger.info({ requestId: 'abc123', durationMs: 42 }, 'Request completed');. Pino writes one JSON object per line to stdout. Each line includes level (numeric: 30=info, 40=warn, 50=error), time (Unix milliseconds), msg, pid, and hostname automatically. To get ISO 8601 timestamps instead of Unix ms, add timestamp: pino.stdTimeFunctions.isoTime to the options object. For Express or Fastify, use pino-http middleware to auto-log every request with method, url, statusCode, responseTime, and reqId. Pino benchmarks at 20,000–50,000 logs/sec — roughly 5× faster than winston and 8× faster than bunyan — because it uses a fast JSON serializer and defers pretty-printing to a separate pino-pretty process. For more on JSON in Node.js APIs, see REST API JSON response design.

How do I implement JSON logging in Python?

Install python-json-logger with pip install python-json-logger. Then configure the standard logging module to use JsonFormatter: import logging; from pythonjsonlogger import jsonlogger; logger = logging.getLogger(); handler = logging.StreamHandler(); handler.setFormatter(jsonlogger.JsonFormatter("%(asctime)s %(name)s %(levelname)s %(message)s")); logger.addHandler(handler); logger.setLevel(logging.INFO). Each call to logger.info("message", extra={"requestId": "abc123", "durationMs": 42}) outputs a single JSON object with asctime, name, levelname, message, requestId, and durationMs. The extra dict is the standard Python logging mechanism for adding custom fields — all keys in extra are merged into the top-level JSON object by JsonFormatter. For Django or Flask, configure the formatter in LOGGING settings or app.loggerrespectively. For the JSONL line-format context, see JSONL format.

What fields should every JSON log entry include?

The 5 universal fields every JSON log entry should include are: level (string or numeric severity — "info", "error", or pino numeric 30/50), timestamp (ISO 8601 string: "2024-01-15T10:30:00.000Z" — never Unix seconds alone, aggregators need the timezone), message (human-readable summary of the event), requestId (UUID or trace ID that ties all logs from one HTTP request together), and service (the name of the application or microservice emitting the log). Beyond these, add durationMs for any timed operation (DB query, HTTP call, function), userId for authenticated requests, statusCode for HTTP responses, and errorCode / stackTrace for errors. Keep field names consistent in camelCase or snake_case across all services — mixing conventions breaks cross-service queries in aggregators. See JSON examples for reference object structures.

How do I redact sensitive data from JSON logs?

Pino has built-in redaction via the redact option: pino({ redact: ['req.headers.authorization', 'user.password', 'body.creditCard'] }). Any field matching a path in the redact array is replaced with [Redacted] before the JSON is serialized — the original value never reaches the output stream. For nested or dynamic paths, use fast-redact (the library pino uses internally) directly. In Python with python-json-logger, implement a logging.Filter subclass that scrubs sensitive keys from the record.__dict__ before formatting. Fields to always redact: Authorization and Cookie request headers, password / secret / token fields in request bodies, credit card numbers, social security numbers, and any PII (email, phone, full name in some jurisdictions). Verify redaction is working by inspecting raw log output in development — use Jsonic to pretty-print and search the JSON for sensitive patterns.

How do I correlate JSON logs across a single HTTP request?

Request correlation works by generating a unique requestId (UUID v4) at the entry point of each HTTP request and propagating it to every log statement, downstream service call, and database query within that request's lifetime. In Node.js with pino-http, the middleware automatically generates and attaches req.id to the request object and includes it in every log line. For manual propagation, use AsyncLocalStorage: store the requestId in a context at the start of the request, then read it in your logger's mixin or bindings. In Express: const { AsyncLocalStorage } = require('async_hooks'); const als = new AsyncLocalStorage(); app.use((req, res, next) => { als.run({ requestId: req.headers['x-request-id'] || uuid() }, next); }); const logger = pino({ mixin: () => als.getStore() || {} }); For distributed tracing across microservices, propagate the requestId as an X-Request-ID or X-Correlation-ID HTTP header and log it in every service that handles the request. For more on building JSON-based REST APIs, see REST API JSON response design.