Skip to main content

Monitoring Fabric-X with Prometheus

Every Fabric-X role exposes a Prometheus /metrics endpoint over HTTP. ChainLaunch allocates a dedicated host port per role, publishes it on the Docker bridge, and surfaces a ready-to-scrape URL on each node's detail page.

This page covers what's exposed, how to find the URLs, and how to wire them into Prometheus and Grafana.

What's exposed​

Each role serves Prometheus metrics at /metrics on its monitoring port. Every node also exposes Go runtime profiling at /debug/pprof/ on the same port.

Orderer group — 4 endpoints per node​

RoleDefault monitoring portNotes
RouterGRPC base + 100Client-broadcast counters, batcher dispatch latency
BatcherGRPC base + 101Batch formation, mempool occupancy
ConsenterGRPC base + 102Arma consensus rounds, decisions, blocks per second
AssemblerGRPC base + 103Block assembly, replication channel size

Committer — 5 endpoints per node​

RoleDefault monitoring portNotes
SidecarGRPC base + 100Block fetch, last-committed-block, ledger height
CoordinatorGRPC base + 101Validation pipeline depth, dependency graph
ValidatorGRPC base + 102Per-worker throughput, prepared/validated/committed counts
VerifierGRPC base + 103Signature verification rate, parallel-executor backlog
Query-serviceGRPC base + 104Query latency, view-aggregation window

The +100 offset keeps monitoring ports out of the way of the GRPC port range and adjacent allocations. ChainLaunch scans for free ports starting at the offset and skips any GRPC port already assigned to the same group.

Finding the URLs​

Web UI​

Open any Fabric-X node's detail page. The Prometheus Metrics card shows one URL per role with copy and open-in-tab buttons:

  • On a parent group node (orderer-group or committer), every role URL is listed.
  • On a per-role child node (e.g. the batcher), the card shows just that role's URL.

URLs are formatted as http://<externalIp>:<monitoringPort>/metrics.

REST API​

GET /api/v1/nodes/{id} returns the URLs in the response body:

{
  "id": 12,
  "nodeType": "FABRICX_ORDERER_GROUP",
  "fabricXOrdererGroup": {
    "externalIp": "10.0.1.4",
    "routerPort": 17000,
    "routerMonitoringPort": 17100,
    "routerMetricsUrl": "http://10.0.1.4:17100/metrics",
    "batcherMetricsUrl": "http://10.0.1.4:17101/metrics",
    "consenterMetricsUrl": "http://10.0.1.4:17102/metrics",
    "assemblerMetricsUrl": "http://10.0.1.4:17103/metrics"
  }
}

Per-role child nodes also surface a single fabricXChild.metricsUrl.

Quick check from the host​

curl -s http://10.0.1.4:17100/metrics | head -20

You should see Prometheus exposition format (# HELP, # TYPE, then samples). If you get a connection refused, the most common cause is the role still booting — wait 30 seconds and retry.

Wiring into Prometheus​

Add a static-config job per network to your prometheus.yml:

scrape_configs:
  - job_name: chainlaunch-fabricx-orderer
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets:
          - 10.0.1.4:17100   # party 1 router
          - 10.0.1.4:17101   # party 1 batcher
          - 10.0.1.4:17102   # party 1 consenter
          - 10.0.1.4:17103   # party 1 assembler
        labels:
          party: "1"
          role_group: orderer

  - job_name: chainlaunch-fabricx-committer
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets:
          - 10.0.1.4:17110   # party 1 sidecar
          - 10.0.1.4:17111   # party 1 coordinator
          - 10.0.1.4:17112   # party 1 validator
          - 10.0.1.4:17113   # party 1 verifier
          - 10.0.1.4:17114   # party 1 query-service
        labels:
          party: "1"
          role_group: committer

Add equivalent target lists for each party. Suggested labels:

  • party — 1-indexed party id, matches the partyId on the node.
  • role — router, batcher, consenter, etc., for selectable dashboards.
  • role_group — orderer or committer, for top-level rollups.

The metric names follow the upstream fabric-x-committer and fabric-x-orderer naming. Inspect the raw /metrics output once to discover what's available — the upstream projects don't yet publish a stable metric reference.

TLS for the monitoring endpoint​

The monitoring port serves plain HTTP by default. For production deployments where the host is on an untrusted network, the upstream fabric-x-committer config schema accepts a monitoring.tls block sharing the GRPC TLS material, but ChainLaunch does not wire it through the templates today. Track this gap if you need it.

Useful starting queries​

A few PromQL queries that work against the upstream metrics. They're starting points — the exact metric names depend on which upstream version you're running.

Orderer-side throughput​

# Decisions per second across all consenters
sum(rate(consensus_decisions_count_total[1m])) by (party)

Committer-side throughput​

# Blocks committed per second
rate(committer_committed_blocks_total[1m])

# Pipeline depth — alert if this grows unbounded
validator_pending_txs_count

Liveness​

# A scrape failure means the role's monitoring port is unreachable
up{job=~"chainlaunch-fabricx.*"}

Default ports — when you see them​

ChainLaunch picks free ports during Init(), starting from GRPC base + 100. If a host doesn't already have anything in that range, you'll typically see the allocation pattern shown in the tables above.

You can pre-pin specific monitoring ports on create by passing routerMonitoringPort, sidecarMonitoringPort, etc. in the orderer-group or committer config. Pass 0 (or omit the field) to auto-allocate.

Upstream defaults from fabric-x-committer v1.0.0-alpha for reference (these are not what ChainLaunch uses by default — ChainLaunch derives ports from the GRPC base instead):

RoleUpstream default
Sidecar2114
Verifier2115
Validator2116
Query-service2117
Coordinator2119

Troubleshooting​

connection refused on /metrics​

The role is up but the monitoring port isn't bound. Most common causes:

  • The role started against a config file written before monitoring support was added. Recreate the network so Init() re-runs against the current templates.
  • Another process on the host is already bound to the monitoring port. Check lsof -i :<port>. The Init port allocator scans for free ports, so this only happens when something binds the port after Init wrote the config.

connection refused on /metrics when the GRPC port works​

The container is running but failed to bind the monitoring port. Inspect the container logs (docker logs <container>) for monitoring server stopped with an error. v1.0.0-alpha logs the bind URL on startup — search for Prometheus serving on URL.

URL field is empty in the API response​

metricsUrl is rendered from externalIp + monitoringPort. If either is zero, the field is omitted. Legacy nodes created before monitoring support landed have monitoringPort: 0; they need to be recreated.

See also​