Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Observability Hooks

ObservabilityHooks holds one callback per lifecycle point. Start from defaultObservabilityHooks and set fields:

myHooks = Arb.defaultObservabilityHooks
  { Arb.onJobSuccess = \job startTime endTime ->
      liftIO $ recordHistogram "jobs.duration" (diffUTCTime endTime startTime)
  , Arb.onJobFailedAndMovedToDLQ = \job err ->
      liftIO $ sendAlert (Arb.primaryKey job) err
  , Arb.onJobHeartbeat = \job now startTime ->
      liftIO $ recordGauge "jobs.running_duration" (realToFrac $ diffUTCTime now startTime)
  }

config <- Worker.transactionalWorkerConfig 5 handler
let instrumented = config { Worker.observabilityHooks = myHooks }

Hooks run in the pool monad.

Hook Invocation

Every claimed job fires onJobClaimed, then one row of:

OutcomeHooks
The handler returnsonJobSuccess
The handler fails with attempts leftonJobFailure, then onJobRetry with the backoff
The handler fails permanently or spends its last attemptonJobFailure, then onJobFailedAndMovedToDLQ
The handler cancels a tree or a branchonJobCancelled
The job went away mid-flightonJobUnavailable
The handler nacks the jobnone

onJobHeartbeat fires on each lease extension of a running job.

Hook Composition

ObservabilityHooks is a Monoid. <> runs the left callback, then the right. The right callback runs even when the left one throws. withHooks combines a record with a configuration's hooks:

let instrumented = Worker.withHooks (myHooks <>) config

Hook Restrictions

A hook's return value is ignored. If a hook throws, the worker logs the exception at Warning and carries on.

onJobSuccess reports a completed ack operation, not a durable outer transaction commit. In a batched handler's outer transaction, it fires when the callback's savepoint is released. If the outer transaction rolls back, the job is redelivered and the hook may fire again. See Batched Handlers.

Reaper activity reports through onMaintenance on WorkerConfig.

Hooks in Another Monad

hoistObservabilityHooks runs hooks written in one monad from a worker in another, through a monad morphism such as lift:

let pool = config {Worker.observabilityHooks = Arb.hoistObservabilityHooks lift appHooks}

arbiter-orville provides orvilleHooks for this.

Callback arguments: ObservabilityHooks Haddocks.