Log Streaming (lbcli)

This article covers the same flow as Log Streaming (REST API), but uses lbcli instead of raw curl commands.

Prerequisites

• A running Lightbits cluster.
• lbcli installed and on your $PATH.
• The system:cluster-admin JWT for the cluster.

Step 1: Setting your JWT and Server

xxxxxxxxxx

export SERVER=server00   # replace with your server hostname or IP

export JWT=eyJhbGciOiJSUzI1NiIsImtpZCI6InN5c3RlbTpyb290IiwidHlwIjoiSldUIn0...

# ^ obtain from /etc/cluster-manager/system_jwt on any cluster node

All lbcli commands below pass the JWT via -J $JWT and the server via --mgmt-endpoint $SERVER. You can also set LBCLI_MGMT_ENDPOINT in your environment to avoid repeating --mgmt-endpoint on every call.

Step 2: (Optional) Creating a TLS Bundle

A TLS bundle holds the certificate material used for mTLS connections to an Rsyslog target. Skip this step if you only need a Loki target.

List the bundles to verify:

Step 3: Creating Sources

A source tells Alloy which log stream to collect on each server. You can collect from journald (by syslog identifier) or from a file path.

Control-plane Services (journald)

Data-plane Services (journald)

Event Log Files (Optional)

Event logs are written to files only when emit_service_events_to_file is enabled in the Ansible config.

List all sources to verify:

Get a specific source:

Step 4: Creating Targets

A target is a log destination. You can have multiple targets; every collected log line is forwarded to all enabled targets.

Loki Target (No TLS)

Rsyslog Target (mTLS, Requires TLS Bundle from Step 2)

List the targets to verify:

Get a specific target:

Step 5: Verifying Collector Status

Once the configuration is applied, Alloy polls lightbox-exporter for the rendered pipeline config and begins shipping logs. Check that each server's Alloy agent is healthy:

When everything is healthy, each collector entry shows the following:

If PIPELINE shows DEGRADED or ERROR, check the target connectivity.

Modifying the Configuration

Sources and targets can be updated or deleted without restarting any service. Alloy picks up the new pipeline config on its next remotecfg poll (default: 1 minute).

Only the flags you explicitly pass are changed; unspecified fields keep their current values.

Inspecting the Rendered Alloy Pipeline Config

You can inspect the pipeline config that lightbox-exporter renders for Alloy by querying the remotecfg endpoint on the server directly (no auth required — node-local only):