Metrics¶
OrionBelt serves Prometheus metrics when METRICS_ENABLED=true. They answer
at /metrics on a port of their own, not the API's, so they stay off a public
API unless that port is published.
| Setting | Default | What it sets |
|---|---|---|
METRICS_ENABLED |
false |
Serve /metrics |
METRICS_HOST |
0.0.0.0 |
The address it listens on |
METRICS_PORT |
9090 |
Its port |
Off, nothing of it runs: no request or query is counted, and the Prometheus client library is not loaded.
Requests and queries¶
| Metric | Type | Labels | What it counts |
|---|---|---|---|
obsl_http_requests_total |
counter | method, route, status |
REST API requests |
obsl_http_request_duration_seconds |
histogram | method, route |
How long they took |
obsl_queries_total |
counter | surface, dialect, source, outcome |
Queries run on a database |
obsl_query_duration_seconds |
histogram | surface, dialect, source |
How long they took, results fetched |
methodis the request's method. A method HTTP does not define isother, since a client can send any word as one.routeis the route's template, such as/v1/sessions/{session_id}/query/execute, so one series covers every session. A path no route has isunmatched.surfaceis where the query came in:rest,pgwireorflight.otheris everything else, such as the command line.sourceis the data source's name, or the dialect for a source not named.outcomesays how the query ended:ok;error: the database refused it;unavailable: it could not run, for example no connection was free or the source is not set up.- What counts as a query. A result served from the result cache runs none. A federated query counts each of its legs, on the leg's own source.
Caches and sessions¶
Read at each scrape.
| Metric | Type | What it says |
|---|---|---|
obsl_build_info |
gauge | 1, with the running version |
obsl_sessions |
gauge | Sessions open |
obsl_result_cache_backend |
gauge | 1, with the result cache's backend |
obsl_result_cache_entries |
gauge | Results cached |
obsl_result_cache_bytes |
gauge | Their size |
obsl_result_cache_lookups_total |
counter | Lookups, by result: hit, miss |
obsl_compilation_cache_entries |
gauge | Compiled queries kept |
obsl_compilation_cache_bytes |
gauge | Their estimated size |
obsl_compilation_cache_lookups_total |
counter | Lookups, by result: hit, miss, bypass (not cacheable) |
obsl_compilation_cache_evictions_total |
counter | Compiled queries evicted to stay within the limits |
The result cache's numbers are the ones GET /v1/cache/stats shows. A scrape
reads them on a thread of its own, not the API's, and does not wait for the
queries using the cache. A scrape the cache cannot answer leaves them out.
DuckDB worker pools¶
For each DuckDB source on Hardcharger workers and
each of its workers, labelled source and worker (the host:port the source
lists). Read at each scrape.
| Metric | Type | What it says |
|---|---|---|
obsl_duckdb_worker_up |
gauge | 1 while the worker takes queries, 0 while the pool keeps it out |
obsl_duckdb_worker_connections |
gauge | Connections by state: in_use (held by a query), idle, opening (a connect given up on that has not come back) |
obsl_duckdb_worker_statements_total |
counter | Statements the worker ran, a session's setup (DUCKDB_CONNECTION_SQL) as one, those that failed or that the pool ended included. One it refused before running counts as a retry |
obsl_duckdb_worker_retries_total |
counter | Queries run again because the worker did not start them, by reason: unreachable, connect_timeout, session_not_found, rate_limited, server_draining |
obsl_duckdb_worker_statements_ended_total |
counter | Statements the pool ended, by reason: deadline or unresponsive |
Each worker also serves Hardcharger's own metrics, on its port 9091
(HARDCHARGER_ENABLE_METRICS), with its statement queue.
The process¶
Where the system reports them: CPU, memory and open files
(process_*), the Python version (python_info), and garbage collection
(python_gc_*).
Scraping¶
scrape_configs:
- job_name: obsl
static_configs:
- targets: ["obsl:9090"]
- job_name: hardcharger
static_configs:
- targets: ["hardcharger-1:9091", "hardcharger-2:9091"]
In the repository's Docker Compose setup (deploy/compose/), both answer on
the project network only. Publish a port to scrape them from the host.