Troubleshooting Runbook

This article provides walkthroughs and suggestions for debugging common issues in a production Lightbits cluster. It follows a symptom-based approach, starting with the observed issue (e.g., loss of service), and progressing through diagnostic steps to identify the root cause.

Warning

Recovery steps mentioned in this document should always be tried in a non-production environment first if possible. Information in the documentation may differ from your specific version. Proceed with caution when running commands on a live cluster.

Primary Lightbits Services

Service Name

Description

Log Access

node-manager 

Lightbits node services manager (volumes, snapshots)

/var/log/node-manager.log

journalctl -u node.manager

gftl

Lightbits backend service

journalctl -u gftl

duroslight-0/1 

Lightbits frontend service

journalctl -u duroslight-0

journalctl -u duroslight-1

cluster-manager 

Lightbits cluster operations

/var/log/cluster-manager.log
journalctl -u cluster-manager

upgrade-manager

Lightbits upgrade operations

/var/log/upgrade-manager.log

journalctl -u upgrade-manager

api-service

Lightbits RestAPI and command-line operations

/var/log/api-service.log

journalctl -u api-service

lightbox-exporter 

Lightbits statistics engine

/var/log/lightbox-exporter.log

journalctl -u lightbox-exporter

profile-generator

Optimize the server resource allocation to Lightbits services.

/var/log/profile-generator.log

journalctl -u profile-generator

discovery-service

Handle client connections

/var/log/discovery-client.log

journalctl -u discovery-service.service

etcd 

Lightbits internal database

journalctl -u etcd

System Health Checklist

General Debugging and Information Gathering

Loss of Service at the Client Side

Video Walkthrough

If an application fails to read/write, follow these steps to isolate the failure domain.



Node Status Issues

Video Walkthrough

Volume Issues

Video Walkthrough

References