Observability

Stages is the contract a microservice codes against; Ingest is how a result is stored. This page is for whoever runs the platform: it documents what the workflows engine (hub-workflows) writes to its logs and traces as it drives a run, so you can answer the questions that come up in practice — did this recording open a run? did it reach a stage? why not? did a stage time out? — and so the logs you forward to us are precise enough to act on.

The engine is deliberately chatty about its routing decisions: every run leaves a trace from the moment it opens to the moment it finalizes, even when it dispatches no stage at all. That is the case that used to be invisible (“a run was created but never reached a stage”) and is now explicit.

Where the logs come from

All of the lines below are emitted by the standalone hub-workflows engine — the service that consumes the classify hand-off, dispatches stages and tracks runs to completion. It is separate from your stage microservices, which log on their own (services.<name>.logLevel).

• Format — structured JSON, one object per line, written to stdout (collect it with your platform’s normal pod-log pipeline; kubectl logs deploy/hub-workflows).
• Verbosity — the LOG_LEVEL environment variable. Recognised values are error and debug; anything else (including unset) means info. info shows the full run lifecycle and routing outcomes plus all warnings and errors; debug adds the per-stage condition explanations; error suppresses everything but failures.
• Levels you’ll see — logrus writes the level as "level":"info", "warning" or "error". Faults are emitted at warning or error, healthy lifecycle and routing at info, so you can alert on severity alone without parsing message text.

Anatomy of a log line

Every lifecycle and fault line carries a consistent set of correlation fields so you can pivot from one line to the whole run. A started line looks like:

{
  "level": "info",
  "msg": "workflow run started",
  "time": "2026-06-24T08:30:01Z",
  "traceId": "8f3a1c2b4d5e6f70",
  "organisationId": "64f0a1b2c3d4e5f600112233",
  "fileName": "front-gate/2026/06/12/08-30-00.mp4",
  "mediaKey": "front-gate/2026/06/12/08-30-00.mp4"
}

Every line in this document — the lifecycle, the routing decisions and the ingest/persistence failures — carries these correlation fields as structured keys (traceId, mediaKey, organisationId, and runId once the run is loaded). You can filter the entire life of one recording on mediaKey or traceId alone and group faults by operation/reason; you never have to parse values out of the message text.

The run lifecycle (info)

A healthy run emits these, in order — they are your “happy path” trace. Seeing started then completed with no stage dispatched in between is the signature of a run created but never reached a stage.

dispatchedStages / resolvedStages on the completion line let you confirm at a glance how much the run did: dispatchedStages: 0 is the explicit never reached a stage outcome.

Routing decisions (info & debug)

When a result lands (or a run opens), the engine re-evaluates the conditional stages and says what it decided. Every line carries mediaKey, runId and the rest of the correlation fields:

Troubleshooting a stage that never fires: set LOG_LEVEL=debug and read its workflow conditional stage routing decision line — matched: false with the reason naming the failing need and the value it saw, which is almost always a path/value mismatch in the condition or an upstream that wasn’t in availableOperations yet.

Faults — warnings & errors

These are the lines to alert on and the ones most useful to send us. All carry the correlation fields above; the error field holds the underlying cause.

Result-persistence (ingest) failures

When a stage hands back a block envelope, the engine persists it through the shared ingest core. These lines carry the full correlation fields plus operation; the error cases use a reason field to distinguish the failure site, so you alert on msg+level and group by reason:

Reading a run end-to-end

Pick the recording’s mediaKey (or its traceId) and filter the engine logs on it; the lifecycle reads as a short story. A few signatures:

Healthy run with a conditional stage

workflow run started
workflow conditional stages matched              matchedOperations=[speed] runId=…
workflow stage dispatched                        operation=speed queue=hub-workflows-speed runId=…
workflow upstream operation resolved             operation=speed
workflow run completed                           dispatchedStages=1 resolvedStages=1 timedOut=false

Created but never reached a stage (the previously-invisible case)

workflow run started
workflow no conditional stages matched           availableOperations=[classify] runId=…
workflow run completed                           dispatchedStages=0 resolvedStages=0 timedOut=false

→ Working as designed if no stage’s rule matched this recording; if you expected one to, switch to LOG_LEVEL=debug and read the workflow conditional stage routing decision line (matched: false) and its reason.

Stuck / crashed stage

workflow run started
workflow stage dispatched            operation=speed …
workflow run timed out with stages outstanding   dispatchedStages=1 resolvedStages=0 unresolvedStages=1

→ The speed microservice received the run but never routed it back. Check that pod.

Distributed tracing

The engine participates in OpenTelemetry tracing under the service name hub-workflows. For every message it continues the trace carried on the run’s traceId and opens a span tagged with the operation and fileName, so a single recording’s spans line up across the analysis service, the engine and your stages in your tracing backend.

The same traceId is on the WorkflowRun your stage receives. Propagate it on your stage’s own logs and spans (see the traceId field) and the whole run — pipeline → engine → your microservice — stays correlatable under one id. If the tracing backend is unreachable the engine logs failed to connect to tracing backend (at startup) or tracing failed for event (per message) and keeps processing — tracing is best-effort and never blocks a run.

Metrics

The engine exposes Prometheus metrics on :8080/metrics — the hook your monitoring scrapes. Alongside the engine’s queue processing time (<queue>_processing_time), it exports lifecycle and fault counters so you can alert on rates and ratios across the fleet without parsing logs:

Useful signals to alert on: a rising workflow_runs_timed_out_total (stuck stages), any workflow_stage_ingest_failures_total{reason="permanent_rejection"} (real data loss), and a high workflow_runs_no_stage_total / workflow_runs_completed_total ratio (recordings opening runs but matching no stage). Each counter has a one-to-one structured log line above, so a metric spike points straight at the lines to read.

Retries & dead-lettering

A run whose result can’t be persisted is re-queued rather than lost, but the loop is bounded so a permanently-failing stage can’t grow the queue forever:

• WORKFLOWS_MAX_RETRIES (default 5) caps how many times one run is re-queued; past the cap it is sent to the dead-letter queue.
• Retries are delayed by a fixed backoff (5s) and a malformed message body (not a WorkflowRun) is dead-lettered immediately.

If runs are disappearing, check the dead-letter queue and the workflow stage result ingest failed lines (reason: transient persistence failure, retrying) — a recording that retries up to the cap and then stops is being dead-lettered.

Sending us accurate logs

When you open an issue, the single most useful thing is the engine’s JSON log lines for the affected recording, unredacted of the correlation fields:

1. Find the recording’s mediaKey (or traceId).
2. Grep the hub-workflows logs for it and include the whole run — started through completed/timed out.
3. Include any warning/error lines (they carry the error field with the root cause).
4. If a conditional stage isn’t firing, re-run with LOG_LEVEL=debug and include the workflow conditional stage routing decision lines (matched: true/false with their reason) for that stage.
5. Note the traceId so we can line it up with the analysis service and your stage.

That set lets us reconstruct exactly what the engine decided and why, without guessing.