JSON Audit Logging: Schema Design, Immutability, and SIEM Integration

JSON Audit Log Schema Design

A well-designed audit log schema balances completeness with parsability. Five fields are required in every audit event; additional fields are context-dependent but should be standardized across all services before the first event is written — schema migrations on append-only audit tables are painful.

A complete minimal audit event looks like this:

{
  "schemaVersion": "1.0",
  "timestamp": "2025-05-19T14:32:00.000Z",
  "actor": {
    "id": "usr_42",
    "type": "user",
    "name": "alice"          // optional — pseudonymize for GDPR
  },
  "action": "document.delete",
  "resource": {
    "type": "document",
    "id": "doc_99",
    "name": "Q2 Financial Report"
  },
  "outcome": "success",
  "correlationId": "req_abc123",
  "ip": "198.51.100.0",      // last octet zeroed for GDPR
  "sessionId": "sess_xyz789",
  "metadata": {
    "reason": "user-requested",
    "previousVersion": "v3"
  },
  "hash": "a3f9c2d1..."      // HMAC chain value
}

Use a dot-namespaced action vocabulary — "user.login", "document.delete", "role.assign" — so SIEM queries can filter by prefix (action: "user.*" returns all user lifecycle events). Add a schemaVersion field so downstream consumers can adapt their parsing logic when the schema evolves. For JSON data validation patterns to enforce this schema at write time, see that guide.

OWASP Logging Recommendations

The OWASP Logging Cheat Sheet defines the categories of events that must be captured and the sensitive data that must never appear in logs. Following these recommendations is a prerequisite for passing security audits and penetration tests.

Events that must be logged:

• Authentication events — successful login, failed login (with reason: invalid password vs. account locked), logout, MFA challenge success/failure, and password reset
• Access control failures— any request that returns 401 or 403, including the requested resource and the actor's roles at the time
• Input validation failures — requests rejected by input validators, particularly for fields that could indicate injection attempts (SQL, XSS, command injection)
• Admin actions — all actions performed by privileged accounts: user creation, role assignment, configuration changes, and data exports
• High-value transactions — any business event with financial, legal, or regulatory significance: invoice approval, contract signing, bulk data deletion

Data that must never appear in audit logs:

• Passwords, even hashed — use a boolean "passwordChanged": true
• Session tokens, JWT access tokens, API keys, or any secret credential
• Payment card numbers (PAN), CVV codes, or full bank account numbers
• Unmasked PII beyond what is strictly necessary (see GDPR section)

OWASP also specifies severity levels for audit events. Map them to your JSON severity field: "INFO" for routine successful actions, "WARN" for access control failures, "ERROR" for authentication failures or unexpected outcomes, and "CRITICAL" for admin privilege escalation or bulk data operations. For broader JSON security hardening beyond logging, see that guide.

Pino Structured Audit Logging

Pino's child logger pattern is ideal for audit logging: create a root logger with audit-specific bindings, then bind per-request context (actor, sessionId, correlationId) to a child logger so every audit event in that request automatically carries the right context without repetition.

npm install pino

// audit-logger.ts
import pino from 'pino'
import crypto from 'crypto'

const SECRET_KEY = process.env.AUDIT_HMAC_SECRET ?? ''
let prevHash = 'GENESIS'

const rootLogger = pino({
  level: 'info',
  timestamp: pino.stdTimeFunctions.isoTime,
  base: { service: 'payments-api', schemaVersion: '1.0' },
  // Redact PII and secrets — values never reach the output stream
  redact: {
    paths: [
      'actor.email',          // store hashed email separately
      'metadata.password',
      '*.token',
      '*.secret',
    ],
    censor: '[REDACTED]',
  },
})

export function createAuditLogger(requestContext: {
  correlationId: string
  sessionId: string
  actor: { id: string; type: string }
  ip: string
}) {
  // Anonymize IP: zero the last octet (IPv4)
  const anonymizedIp = requestContext.ip.replace(/.d+$/, '.0')

  return rootLogger.child({
    ...requestContext,
    ip: anonymizedIp,
  })
}

export function auditEvent(
  logger: pino.Logger,
  action: string,
  resource: { type: string; id: string },
  outcome: 'success' | 'failure' | 'denied',
  metadata?: Record<string, unknown>,
) {
  const entry = {
    action,
    resource,
    outcome,
    metadata,
  }

  // Compute HMAC chain hash
  const hash = crypto
    .createHmac('sha256', SECRET_KEY)
    .update(prevHash + JSON.stringify(entry))
    .digest('hex')

  prevHash = hash

  logger.info({ ...entry, hash }, 'audit')
}

// usage in a request handler
import { createAuditLogger, auditEvent } from './audit-logger.js'

app.delete('/documents/:id', async (req, res) => {
  const auditLog = createAuditLogger({
    correlationId: req.headers['x-request-id'] ?? crypto.randomUUID(),
    sessionId: req.session.id,
    actor: { id: req.user.id, type: 'user' },
    ip: req.ip,
  })

  await db.documents.delete(req.params.id)

  auditEvent(
    auditLog,
    'document.delete',
    { type: 'document', id: req.params.id },
    'success',
    { reason: req.body.reason },
  )

  res.status(204).end()
})

The redact option ensures that even if a developer accidentally passes an email or token field into metadata, it is replaced with [REDACTED] before the JSON is written to stdout. Use Pino's INFO level for all audit events — they are not debug noise and should always be present in production. For the full Pino setup guide including transports and aggregator shipping, see JSON structured logging.

SIEM Integration: Splunk and Elastic

Audit logs derive their value from being queryable — a SIEM (Security Information and Event Management) system like Splunk or Elasticsearch provides the search, alerting, and dashboarding layer. Both accept JSON natively.

Splunk HTTP Event Collector (HEC)

// Send a single audit event to Splunk HEC
const SPLUNK_HEC_URL = 'https://splunk.example.com:8088/services/collector'
const SPLUNK_TOKEN = process.env.SPLUNK_HEC_TOKEN

async function sendToSplunk(auditEntry: object) {
  const payload = {
    event: auditEntry,
    time: Date.now() / 1000,          // Unix epoch SECONDS (not ms)
    source: 'audit-service',
    sourcetype: '_json',
    index: 'audit',
  }

  await fetch(SPLUNK_HEC_URL, {
    method: 'POST',
    headers: {
      'Authorization': `Splunk ${SPLUNK_TOKEN}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(payload),
  })
}

// Batch ingestion — concatenate JSON objects (no comma, no array wrapper)
async function sendBatchToSplunk(entries: object[]) {
  const body = entries
    .map(e => JSON.stringify({ event: e, time: Date.now() / 1000, sourcetype: '_json', index: 'audit' }))
    .join('')                         // newline-delimited, no commas

  await fetch(SPLUNK_HEC_URL, {
    method: 'POST',
    headers: { 'Authorization': `Splunk ${SPLUNK_TOKEN}`, 'Content-Type': 'application/json' },
    body,
  })
}

Elasticsearch ingest pipeline

# Create an ingest pipeline that normalizes audit events on ingest
PUT /_ingest/pipeline/audit-normalize
{
  "description": "Normalize JSON audit log events",
  "processors": [
    {
      "date": {
        "field": "timestamp",
        "formats": ["ISO8601"],
        "target_field": "@timestamp"
      }
    },
    {
      "set": {
        "field": "event.kind",
        "value": "event"
      }
    },
    {
      "set": {
        "field": "event.category",
        "value": "authentication",
        "if": "ctx.action?.startsWith('user.login')"
      }
    }
  ]
}

# Index an audit event using the pipeline
POST /audit-logs-2025.05/_doc?pipeline=audit-normalize
{
  "schemaVersion": "1.0",
  "timestamp": "2025-05-19T14:32:00.000Z",
  "actor": { "id": "usr_42", "type": "user" },
  "action": "document.delete",
  "resource": { "type": "document", "id": "doc_99" },
  "outcome": "success"
}

For Elasticsearch, use an Index Lifecycle Management (ILM) policy to roll over the audit index monthly and freeze indices older than 6 months. Map the action, outcome, and actor.type fields as keyword (not text) to enable exact-match aggregations and fast cardinality queries. For guidance on the underlying JSON telemetry layer, see OpenTelemetry JSON.

Tamper Detection with HMAC Chaining

HMAC chaining creates a cryptographic dependency between consecutive audit records: any modification, deletion, or insertion of a record breaks the chain from that point forward. A verification script detects the break in O(n) time by recomputing hashes from the genesis record.

// hmac-chain.ts — Node.js HMAC chain implementation
import crypto from 'crypto'

const SECRET_KEY = process.env.AUDIT_HMAC_SECRET!  // store in HSM or secrets manager

/**
 * Compute the hash for a new audit entry.
 * Call this before writing the entry to storage.
 */
export function computeHash(prevHash: string, entry: object): string {
  return crypto
    .createHmac('sha256', SECRET_KEY)
    .update(prevHash + JSON.stringify(entry))
    .digest('hex')
}

// Genesis hash — used for the very first audit record
export const GENESIS_HASH = crypto
  .createHmac('sha256', SECRET_KEY)
  .update('GENESIS')
  .digest('hex')

/**
 * Write an audit entry with its chain hash.
 * Returns the new prevHash to pass into the next call.
 */
export async function writeAuditEntry(
  db: AuditDb,
  prevHash: string,
  entryWithoutHash: object,
): Promise<string> {
  const hash = computeHash(prevHash, entryWithoutHash)
  const entry = { ...entryWithoutHash, hash }
  await db.audit.insert(entry)
  return hash
}

// verify-chain.ts — verification script
import { computeHash, GENESIS_HASH } from './hmac-chain.js'

export async function verifyAuditChain(db: AuditDb): Promise<{
  valid: boolean
  brokenAtId?: string
}> {
  const records = await db.audit.findAll({ orderBy: 'timestamp', direction: 'asc' })

  let expectedPrevHash = GENESIS_HASH

  for (const record of records) {
    const { hash, ...entryWithoutHash } = record
    const expectedHash = computeHash(expectedPrevHash, entryWithoutHash)

    if (hash !== expectedHash) {
      return { valid: false, brokenAtId: record.id }
    }

    expectedPrevHash = hash
  }

  return { valid: true }
}

Run the verification script on a schedule (daily or on-demand for investigations). The chain proves records have not been altered after writing — it does not prevent a compromised application from writing fraudulent entries. For storage-level immutability (preventing deletes entirely), combine HMAC chaining with the append-only storage patterns in the next section. For JSON signing at the API payload level, see that guide.

GDPR and PII in JSON Audit Logs

GDPR's right to erasure (Article 17) conflicts directly with audit log immutability: you cannot delete individual records from an append-only store. The solution is pseudonymization — replace personal identifiers with one-way hashes before the event is written, and store the mapping (hash ↔ identity) in a separate, access-controlled lookup table. Deleting a subject's entry from the lookup table effectively erases their identity from the audit log without modifying the log itself.

// gdpr-pseudonymize.ts
import crypto from 'crypto'

const PSEUDONYM_SALT = process.env.PSEUDONYM_SALT!  // rotate annually per retention policy

/**
 * Returns a stable pseudonym for a given PII value.
 * The same input always produces the same output (for cross-event correlation),
 * but the mapping is only reversible via the lookup table.
 */
export function pseudonymize(pii: string): string {
  return 'psh_' + crypto
    .createHmac('sha256', PSEUDONYM_SALT)
    .update(pii.toLowerCase().trim())
    .digest('hex')
    .slice(0, 16)   // truncate for readability — still 64-bit collision resistance
}

/**
 * Anonymize an IPv4 address by zeroing the last octet.
 * For IPv6, use zeroing of the last 80 bits.
 */
export function anonymizeIp(ip: string): string {
  if (ip.includes(':')) {
    // IPv6 — keep first 48 bits (first 3 groups), zero the rest
    const groups = ip.split(':')
    return groups.slice(0, 3).join(':') + '::0'
  }
  return ip.replace(/.d+$/, '.0')
}

// Usage in audit event construction:
const auditEntry = {
  actor: {
    id: pseudonymize(user.email),     // "psh_a3f9c2d1b4e5f6a7" — reversible via lookup table
    type: 'user',
  },
  ip: anonymizeIp(req.ip),           // "198.51.100.0"
  action: 'document.delete',
  // ...
}

Define your retention policy explicitly: 12 months is common for application audit logs; 24–36 months for financial or healthcare audit trails. Implement automated deletion of the pseudonym lookup table entries (not the audit log itself) at the end of the retention window. Document the legal basis for audit log processing in your Record of Processing Activities (RoPA). For broader JSON data validation including schema-level enforcement of what fields may appear, see that guide.

Append-Only Storage Patterns

Append-only storage enforces immutability at the infrastructure level, complementing HMAC chaining at the application level. Three practical patterns cover most deployment environments.

PostgreSQL append-only trigger

-- Create the audit table
CREATE TABLE audit_log (
  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  timestamp   TIMESTAMPTZ NOT NULL,
  actor_id    TEXT NOT NULL,
  action      TEXT NOT NULL,
  resource_id TEXT NOT NULL,
  outcome     TEXT NOT NULL,
  payload     JSONB NOT NULL,      -- full audit event JSON
  hash        TEXT NOT NULL        -- HMAC chain value
);

-- Append-only trigger: reject UPDATE and DELETE
CREATE OR REPLACE FUNCTION audit_log_immutable()
RETURNS TRIGGER LANGUAGE plpgsql AS $$
BEGIN
  RAISE EXCEPTION 'audit_log is append-only: UPDATE and DELETE are not permitted';
END;
$$;

CREATE TRIGGER enforce_audit_immutability
  BEFORE UPDATE OR DELETE ON audit_log
  FOR EACH ROW EXECUTE FUNCTION audit_log_immutable();

-- Grant INSERT-only to the application role (no UPDATE, no DELETE)
REVOKE ALL ON audit_log FROM audit_writer;
GRANT INSERT ON audit_log TO audit_writer;
GRANT SELECT ON audit_log TO audit_reader;

Event-sourcing audit table

// Event-sourcing pattern: audit_events is the source of truth.
// Application state is a projection derived from events.
// Correcting an error means appending a "superseded" event, not editing.

const supersededEvent = {
  schemaVersion: '1.0',
  timestamp: new Date().toISOString(),
  actor: { id: 'usr_admin', type: 'system' },
  action: 'audit.supersede',
  resource: { type: 'audit_event', id: 'evt_original_id' },
  outcome: 'success',
  metadata: {
    reason: 'Original event had incorrect resource ID',
    supersedes: 'evt_original_id',
    correctedFields: { 'resource.id': 'doc_99' },
  },
}

// Write the superseding event — never modify the original
await db.audit.insert(supersededEvent)

AWS S3 Object Lock

// Write audit events to S3 with COMPLIANCE mode Object Lock.
// In COMPLIANCE mode, even the root account cannot delete the object
// before the retention period expires.
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'

const s3 = new S3Client({ region: 'us-east-1' })

async function writeAuditToS3(entry: object, retentionDays = 365) {
  const key = `audit/${new Date().toISOString().slice(0, 10)}/${crypto.randomUUID()}.json`
  const retainUntil = new Date(Date.now() + retentionDays * 86_400_000)

  await s3.send(new PutObjectCommand({
    Bucket: process.env.AUDIT_BUCKET!,
    Key: key,
    Body: JSON.stringify(entry),
    ContentType: 'application/json',
    ObjectLockMode: 'COMPLIANCE',
    ObjectLockRetainUntilDate: retainUntil,
  }))
}

Use S3 Object Lock for long-term archival (12+ months) where low query latency is not required. Use PostgreSQL with the append-only trigger for operational audit logs that require fast querying. Combine both: write to PostgreSQL for 90-day hot storage, then export to S3 Object Lock for the remainder of the retention period.

Definitions

Audit log

An immutable, time-ordered record of security- and compliance-relevant events — who did what to which resource and when — used for forensic investigation, compliance reporting, and accountability.

Actor

The entity that performed an action: a human user (identified by user ID), a service account, or an automated job. The actorfield in a JSON audit event must be stable (not the user's display name, which can change) and pseudonymized if it contains PII.

Action

A dot-namespaced verb describing what the actor did: "user.login", "document.delete", "role.assign". The namespace prefix (e.g. "user.") enables prefix-based filtering in SIEM queries.

Outcome

Whether the action succeeded ("success"), failed due to a system error ("failure"), or was rejected by an access control check ("denied"). All three must be logged — failed and denied actions are often the most security-relevant.

HMAC chain

A tamper-detection mechanism where each audit log entry stores hash = HMAC-SHA256(prevHash + JSON.stringify(entry)). Any modification to any record breaks the hash chain from that point forward, detectable by a verification script.

Pseudonymization

A GDPR-compliant technique that replaces direct personal identifiers (email, name, phone) with one-way hashes before writing to the audit log. The mapping from hash to identity is stored in a separate, access-controlled lookup table; deleting an entry from the lookup table effectively erases the subject's identity from the audit trail without modifying immutable log records.

Append-only storage

A storage configuration that allows INSERT operations but rejects UPDATE and DELETE. Implemented via PostgreSQL triggers, S3 Object Lock, or application-level event-sourcing patterns. Ensures that the audit record cannot be silently altered after the fact.

SIEM

Security Information and Event Management — a platform (Splunk, Elasticsearch, Microsoft Sentinel) that aggregates, indexes, and analyzes security events from multiple sources, enabling real-time alerting, threat detection, and compliance dashboards across audit log streams.

Frequently asked questions

What should a JSON audit log contain?

A JSON audit log entry must contain: actor (who performed the action — user ID, service account, or API key identifier), action (what was done — a verb such as "user.login", "record.delete", or "permission.grant"), resource (the object affected — type and identifier, e.g. {"type":"invoice","id":"inv_123"}), timestamp (ISO 8601 in UTC — "2025-05-19T14:32:00.000Z"), and outcome (whether the action succeeded or failed — "success", "failure", or "denied"). Optional but valuable fields include ip (client IP address, hashed for GDPR compliance), sessionId (links all actions in one session), metadata (contextual details such as changed field values), and hash (HMAC chain value for tamper detection). OWASP further recommends logging the source system, event type, and severity.

How do I design a JSON audit log schema?

Start with five required top-level fields: actor, action, resource, timestamp (ISO 8601 UTC), and outcome. Use a dot-namespaced action vocabulary — "user.login", "document.delete", "role.assign" — so log queries can filter by prefix. Nest contextual data under metadata rather than polluting the top level. Agree on camelCase or snake_case and never mix them across services. Add a schemaVersion field ("1.0") so consumers can evolve parsing logic when the schema changes. For compliance, include a correlationId that links the audit event to the originating HTTP request or job ID. Use JSON data validation at write time to enforce the schema.

How do I send JSON audit logs to Splunk?

Use the Splunk HTTP Event Collector (HEC). Enable HEC in Splunk Settings, create a token, then POST JSON to https://<host>:8088/services/collector with Authorization: Splunk <token> and Content-Type: application/json. The payload wraps your audit event: {"event": {"actor":"usr_42","action":"document.delete","outcome":"success"}, "time": 1747663920, "source": "audit-service", "sourcetype": "_json", "index": "audit"}. The "time" field is Unix epoch seconds (not milliseconds). For batch ingestion, concatenate JSON objects without commas or array brackets. Use a pino transport to ship directly from Node.js without a sidecar agent.

How do I make audit logs tamper-proof?

HMAC chaining makes audit logs tamper-evident: each entry includes a hash field computed as HMAC-SHA256(previousHash + JSON.stringify(entry)). Any modification to any entry — or deletion of entries — breaks the chain, detectable by a verification script that recomputes hashes from the genesis record forward. In Node.js: const hash = crypto.createHmac("sha256", SECRET_KEY).update(prevHash + JSON.stringify(entry)).digest("hex"). Store the chain secret in a hardware security module or secrets manager. For storage-level immutability, combine HMAC chaining with PostgreSQL append-only triggers or AWS S3 Object Lock in COMPLIANCE mode.

Can I store PII in JSON audit logs?

Under GDPR, audit logs containing PII (email addresses, IP addresses, full names) are subject to data subject rights including the right to erasure — which conflicts with audit log immutability. The recommended approach is pseudonymization: instead of storing "alice@example.com", store a one-way hash (e.g. "psh_a3f9c2d1...") and maintain a separate, access-controlled lookup table mapping hash to identity. IP addresses should be anonymized by zeroing the last octet (IPv4). Define a retention policy — audit logs typically have a 12–24 month retention window after which the pseudonym lookup table entries (not the log itself) are deleted.

What is HMAC chaining in audit logs?

HMAC chaining is a tamper-detection technique where each log entry stores a hash that depends on the previous entry's hash and the current entry's content. To verify the chain, start from the genesis record, then for each subsequent record recompute hash = HMAC(secret, prevHash + JSON.stringify(entry)) and compare against the stored hash. If any record has been modified, deleted, or inserted out of order, the recomputed hash will not match from that point forward. Verification is O(n) and requires only the secret key and the ordered record sequence — no blockchain infrastructure needed. The chain proves records have not been altered after the fact; it does not prevent a compromised application from writing fraudulent records initially.

How do I handle GDPR compliance in audit logging?

Four controls are required: (1) Pseudonymization — replace direct identifiers with one-way hashes before writing to the log; store the mapping separately with strict access controls. (2) IP anonymization — zero the last octet of IPv4 addresses before storage. (3) Defined retention — document the legal basis and retention period (typically 12–24 months) in your Record of Processing Activities; delete the pseudonym lookup table after the retention window. (4) Right to erasure— implement a "data subject key" pattern: encrypt PII-derived fields with a per-user key stored separately; to erase, delete the key, rendering encrypted fields irretrievable without altering log chain integrity.

What is the difference between audit logs and application logs?

Application logs record system events for debugging: DB query timings, cache misses, worker crashes. Their audience is the engineering team; retention is typically 7–30 days; events may be sampled under load. Audit logs record business and security events for compliance: user logins, document deletions, role assignments. Their audience is security teams, auditors, and compliance officers; retention is 12–24 months or longer per regulatory requirement; events must be complete and tamper-evident — dropping or sampling audit events is a compliance violation. Audit logs belong in append-only storage with restricted delete permissions; application logs belong in a queryable time-series store optimized for high write throughput. For the application logging layer, see JSON structured logging.