Observability¶

ArqonHPO provides comprehensive observability through structured logs, Prometheus metrics, and real-time dashboards.

Structured Logs¶

The CLI uses tracing and emits structured logs to stderr.

Enable JSON Output¶

arqonhpo --log-format json --log-level info <command>

Log Levels¶

Log Fields¶

Prometheus Metrics¶

Enable metrics endpoint:

arqonhpo --metrics-addr 127.0.0.1:9898 <command>

# Or with dashboard
arqonhpo dashboard --state state.json --metrics-addr 127.0.0.1:9898

Scrape at http://127.0.0.1:9898/metrics.

Counters¶

Gauges¶

Histograms¶

Example Queries (PromQL)¶

# Average evaluations per second
rate(arqonhpo_tell_calls_total[5m])

# 99th percentile eval latency
histogram_quantile(0.99, rate(arqonhpo_eval_duration_seconds_bucket[5m]))

# Violation rate by type
rate(arqonhpo_violations_total[5m])

# Config update frequency
rate(arqonhpo_config_generation[1m])

# Safe mode duration
changes(arqonhpo_safe_mode_active[1h])

Grafana Dashboard¶

Import this JSON or build from these panels:

Recommended Panels¶

1. Throughput — rate(arqonhpo_tell_calls_total[5m])
2. Best Value — arqonhpo_best_value
3. Eval Latency p99 — histogram_quantile(0.99, ...)
4. Violations — rate(arqonhpo_violations_total[5m]) by type
5. Budget Remaining — arqonhpo_budget_remaining
6. Safe Mode Status — arqonhpo_safe_mode_active

Alert Rules¶

groups:
  - name: arqonhpo
    rules:
      - alert: HighViolationRate
        expr: rate(arqonhpo_violations_total[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High violation rate in ArqonHPO"

      - alert: SafeModeActive
        expr: arqonhpo_safe_mode_active == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "ArqonHPO entered safe mode"

Tracing Spans¶

When log-level=trace, spans are emitted:

OpenTelemetry (Planned)¶

OTel export planned for v0.4. Track: Issue #XX

TUI Monitoring¶

Real-time terminal dashboard:

arqonhpo tui --state state.json --events events.jsonl

See TUI Reference for details.

Web Dashboard¶

Browser-based monitoring:

arqonhpo dashboard --state state.json --addr 127.0.0.1:3030

See Dashboard Reference for API endpoints.

Python Observability Patterns¶

The ArqonHPO solver logic runs in-memory. To enable real-time monitoring via the TUI or Dashboard (which watch the filesystem), you must explicitly "mirror" the solver's state to disk.

The "State Mirror" Pattern¶

Use the dump_state utility provided by the SDK to broadcast internal state updates.

import arqonhpo

# In your optimization loop:
while (batch := solver.ask()):
    # ... evaluate batch ...

    # Mirror state to disk for TUI/Dashboard
    arqonhpo.dump_state(
        config=config, 
        history=history, 
        run_id="my-run-001",
        delay=0.1  # Optional: slight delay for smooth TUI visualization
    )

    solver.tell(results)

[!TIP] Human-in-the-Loop: Adding a small delay (e.g., 0.1s - 0.5s) in the loop makes the visualization smooth enough for humans to follow the optimization journey in real-time.

Invariant: Minimization Only!¶

[!WARNING] ArqonHPO is a Minimizer. It strictly seeks the lowest possible objective value. If your metric is logical "Score" or "Accuracy" (where higher is better), you MUST invert it before reporting.

The Maximization Trap: If you report accuracy (e.g., 0.95), the solver will try to minimize it towards 0.0.

Correct Usage: Use arqonhpo.invert_score or manually negate.

accuracy = 0.95
loss = arqonhpo.invert_score(accuracy) # Returns 0.05
# OR
loss = -accuracy

Audit Events¶

All safety-relevant events are logged to the audit trail:

Access via:

• Dashboard API: GET /api/events
• CLI export: arqonhpo export --state state.json

Next Steps¶

• Dashboard — Web UI reference
• Hotpath API — Internal telemetry APIs
• Safety — Understanding violations