Metrics Families

AI Tools

A metrics family is a group of metrics that are exposed by a single collector on every scrape.

This document lists the metrics exported by lightbox-exporter.

Note

This document details all of the metrics and their respective labels as exported by the lightbox exporter. However, Prometheus may be configured to add additional labels to each metric, such as job or instance, Please see the following documentation for additional information: https://prometheus.io/docs/concepts/jobs_instances/.

Chrony

Collects metrics from chrony NTP daemon.

Name	Type	Comments	Labels
`lightbox_chrony_reference_info`	Gauge	Information about the current reference source.	ref_id,ref_host
`lightbox_chrony_stratum`	Gauge	The stratum of the current preferred source.
`lightbox_chrony_reference_timestamp_seconds`	Gauge	Chrony tracking reference timestamp.
`lightbox_chrony_system_offset_seconds`	Gauge	The current estimated drift of system time from true time.
`lightbox_chrony_last_offset_seconds`	Gauge	The estimated local offset on the last clock update.
`lightbox_chrony_rms_offset_seconds`	Gauge	Chrony tracking long-term average of the offset.
`lightbox_chrony_frequency_ppms`	Gauge	The rate by which the system's clock would be incorrect if chronyd was not correcting it, in PPMs.
`lightbox_chrony_residual_frequency_ppms`	Gauge	For the currently selected reference source, the difference between the frequency it suggests and the one currently in use, in PPMs.
`lightbox_chrony_skew_ppms`	Gauge	The estimated error bound on the frequency, in PPMs.
`lightbox_chrony_root_delay_seconds`	Gauge	The total of the network path delays to the stratum-1 computer from which the computer is ultimately synchronized.
`lightbox_chrony_root_dispersion_seconds`	Gauge	The absolute bound on the computer’s clock accuracy.
`lightbox_chrony_update_interval_seconds`	Gauge	The time elapsed since the last measurement from the reference source was processed, in seconds.
`lightbox_chrony_leap_status`	Gauge	The leap status, which can be Normal, Insert second, Delete second, or Not synchronized.
`lightbox_chrony_source_state_info`	Gauge	The chrony source state information.	ref_host,source_mode,source_state
`lightbox_chrony_source_stratum`	Gauge	The stratum of the source, as reported in its most recently received sample.	ref_host
`lightbox_chrony_source_poll_rate_seconds`	Gauge	The rate at which the source is being polled.	ref_host
`lightbox_chrony_source_reach_register`	Gauge	The source reachability register.	ref_host
`lightbox_chrony_source_last_received_seconds`	Gauge	The number of seconds ago the last sample was received from the source.	ref_host
`lightbox_chrony_source_original_offset_seconds`	Gauge	The adjusted offset between the local clock and the source.	ref_host
`lightbox_chrony_source_measured_offset_seconds`	Gauge	The actual measured offset between the local clock and the source.	ref_host
`lightbox_chrony_source_offset_margin_seconds`	Gauge	The error margin in the offset measurement between the local clock and the source.	ref_host
`lightbox_chrony_source_sample_points`	Gauge	The number of sample points currently being retained for the server.	ref_host
`lightbox_chrony_source_residual_runs`	Gauge	The number of runs of residuals having the same sign following the last regression.	ref_host
`lightbox_chrony_source_sample_interval_span_seconds`	Gauge	The interval between the oldest and newest samples.	ref_host
`lightbox_chrony_source_frequency_ppm`	Gauge	The estimated residual frequency for the server.	ref_host
`lightbox_chrony_source_frequency_skew_ppm`	Gauge	The estimated error bounds on the residual frequency estimation.	ref_host
`lightbox_chrony_source_std_dev_seconds`	Gauge	The estimated sample standard deviation.	ref_host

Clustering

Collects information on Lightbits resources. Metrics will expose information such as volumes state, volumes usage, nodes state, nodes usage, etc.

Name	Type	Comments	Labels
`lightbox_clustering_node_info`	Gauge	Node information. The following values are exported via the metric's labels: Lightbits version, node UUID, server UUID	server_id, node_id, node_lightos_version
`lightbox_clustering_num_connected_hosts`	Gauge	Number of hosts connected to the cluster.
`lightbox_clustering_host_unconnected_nodes`	Gauge	Number of nodes that host should be connected to but is not.	host_nqn, host_address, hostname
`lightbox_clustering_volume_logical_used_storage`	Gauge	Logical storage space used by volume (total logical address space used by this volume), given in bytes.	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_physical_used_storage`	Gauge	Physical storage space used by volume, given in bytes.	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_rebuild_progress`	Gauge	Rebuild progress of volume's replicated data, given in precentage.	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_state_fully_protected`	Gauge	Volume is in state fully protected.	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_state_degraded`	Gauge	Volume is in state degraded.	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_state_read_only`	Gauge	Volume is in state read only.	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_state_not_available`	Gauge	Volume is in state not available.	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_migrating`	Gauge	A volume replica is being migrated.	server_id, node_ id, volume_id, volume_name, nsid, project_id
`volume_compression_ratio`	Gauge	Volume's compression ratio logical used (data written)/physical used.	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_physical_owned_capacity`	Gauge	The capacity that would be freed when the volume is deleted (will have identical value as lightbox_clustering_ volume_physical_capacity).	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_physical_capacity`	Gauge	The capacity used by this volume layer (physical capacity since last snapshot/rollback operation on volume).	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_physical_owned_memory`	Gauge	The amount of memory that would be freed by deleting this volume (will have identical value as lightbox_clustering_volume_physical_memory).	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_physical_memory`	Gauge	Memory used by this volume layer (memory used since last snapshot/rollback operation on volume).	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_volume_user_written`	Gauge	The amount of user data written in this layer (data written since last snapshot/rollback operation on volume).	server_id, node_id, volume_id, volume_name, nsid, project_id, project_name
`lightbox_clustering_snapshot_physical_owned_capacity`	Gauge	Capacity that would be freed when this snapshot is deleted (some of the capacity used by snapshot exposed by lightbox_clustering_snapshot___physical_capacity may still be used by volume or new snapshots, so is not freed following the snapshot delete).	node_id, snapshot_id, snapshot_name, src_volume_uuid
`lightbox_clustering_snapshot_physical_capacity`	Gauge	The physical storage used by this snapshot.	node_id, snapshot_id, snapshot_name, src_volume_uuid
`lightbox_clustering_snapshot_physical_owned_memory`	Gauge	The amount of memory would be freed by deleting this snapshot (some of the memory used by snapshot exposed by lightbox_clustering_ snapshot_physical_memory may still be used by volume or new snapshots, so is not freed following the snapshot delete).	node_id, snapshot_id, snapshot_name, src_volume_uuid
`lightbox_clustering_snapshot_physical_memory`	Gauge	The memory used by this snapshot.	node_id, snapshot_id, snapshot_name, src_volume_uuid
`lightbox_clustering_snapshot_user_written`	Gauge	Amount of user data stored at this snapshot (before compression).	node_id, snapshot_id, snapshot_name, src_volume_uuid
`lightbox_clustering_node_num_managed_devices`	Gauge	Number of SSD devices managed by node.	server_id, node_id
`lightbox_clustering_node_local_rebuild_progress`	Gauge	Node local EC rebuild progress [percentage].	server_id, node_id
`lightbox_clustering_node_rebuild_from_peer`	Gauge	Node is recovering replicated data.	server_id, node_id
`lightbox_clustering_node_logical_used_storage`	Gauge	Logical storage space used (data written), given in bytes.	server_id, node_id, node_name
`lightbox_clustering_node_physical_used_storage_including_parity`	Gauge	Physical storage space occupied by all data, including EC overhead given in bytes.	server_id, node_id, node_name
`lightbox_clustering_node_managed_physical_storage`	Gauge	All managed and healthy SSD devices capacities, given in bytes.	server_id, node_id, node_name
`lightbox_clustering_node_physical_used_storage`	Gauge	Physical storage space occupied by all volumes, given in bytes.	server_id, node_id, node_name
`lightbox_clustering_node_free_physical_storage`	Gauge	Available free physical storage capacity, given in bytes.	server_id, node_id, node_name
`lightbox_clustering_node_effective_physical_storage`	Gauge	Effective physical storage excluding overhead of EC and OVP, given in bytes.	server_id, node_id, node_name
`lightbox_clustering_node_estimated_free_logical_storage`	Gauge	Estimated available free logical (data written) storage capacity, given in bytes.	server_id, node_id, node_name
`lightbox_clustering_node_compression_ratio`	Gauge	Compression ratio logical used (data written)/physical used.	server_id, node_id, node_name
`lightbox_clustering_node_estimated_logical_storage`	Gauge	Node's estimated logical available storage (node effective physical storage x compression ratio).	server_id, node_id, node_name
`lightbox_clustering_node_power_up_after_abrupt_shutdown`	Gauge	Node powerup that follows an abrupt shutdown of server or node-manager service.	server_id, node_id
`lightbox_`clustering_`node_`power`up_progress`	Gauge	Progress of node-manager service power up [percentage].	serverid, node id
`lightbox_clustering_node_state_inactive`	Gauge	Node state Inactive.	server_id, node_id
`lightbox_clustering_node_state_active`	Gauge	Node state Active.	server_id, node_id
`lightbox_clustering_node_read_only_mode`	Gauge	Node is in Read Only state.	server_id, node_id, node_name
`lightbox_clustering_node_info`	Gauge	Node information. The following values are exported via the metric's labels: Lightbits version, node UUID, server UUID.	server_id, node_id, node_lightos_version
`lightbox_clustering_node_logical_storage`	Gauge	Sum of provisioned volumes over node, sum of size of all the allocated volumes over node - given in bytes.	server_id, node_id, node_name
`lightbox_clustering_node_unrecoverable_data_integrity_errors`	Counter	The number of unrecoverable data integrity errors experienced by the node.	server_id, node_id, node_name
`lightbox_clustering_node_recoverable_data_integrity_errors`	Counter	The number of recoverable data integrity errors experienced by the node.	server_id, node_id, node_name
`lightbox_clustering_ssd_unrecoverable_data_integrity_errors`	Counter	The amount of the SSD's unrecoverable data integrity errors.	server_id, node_id, node_name, serial_number
`lightbox_clustering_ssd_recoverable_data_integrity_errors`	Counter	The amount of the SSD's recoverable data integrity errors.	server_id, node_id, node_name, serial_number

cpufreq

Exposes kernel/system CPU frequency.

Name	Type	Comments	Labels
`lightbox_cpu_frequency_hertz`	Gauge	Current CPU thread frequency, in hertz.	cpu

datapath

Exposes information from the datapath components (Lightbits GFTL).

Name	Type	Comments	Labels
`lightbox_be_config_md_ram_capacity_used`	Counter	GFTL metadata RAM usage (in equivalent storage capacity, MD usage is expressed as the equivelant amount of data that can be stored for this amount of MD usage).	node_id, server_id
`lightbox_be_config_md_ram_capacity_limit`	Counter	GFTL metadata RAM limit (in equivalent storage capacity, the MD limit is expressed as the equivalent maximum amount of data that can be stored limited by MD usage).	node_id, server_id
`lightbox_be_poll_write_cmds`	Counter	Number of commands the backend wrote.	core_id, node_id, server_id
`lightbox_be_poll_objects`	Counter	Number of objects the backend wrote.	core_id, node_id, server_id
`lightbox_be_num_bio_error`	Counter	Number of inflight read requests.	core_id, node_id, server_id
`lightbox_be_num_bio_error_reconstruct`	Counter	Number of inflight read requests.	core_id, node_id, server_id
`lightbox_be_bio_error_with_lf_bypass`	Counter	Number of inflight read requests.	core_id, node_id, server_id
`lightbox_be_inflight_read_requests`	Counter	Number of inflight read requests.	core_id, node_id, server_id
`lightbox_be_bios_per_request`	Counter	Number of blockio per request.	core_id, node_id, server_id
`lightbox_be_compress_ratio_sum`	Counter	Compression ratio total.	core_id, node_id, server_id
`lightbox_be_compress_ratio_count`	Counter	Compression ratio count.	core_id, node_id, server_id
`lightbox_be_num_of_gc_units`	Counter	Number of write units written by GC since boot (counter).	core_id, node_id, server_id
`lightbox_be_num_of_user_units`	Counter	Number of write units of user data written since boot (counter).	coreid, node id, server_id
`lightbox_be_write_units_start`	Counter	Number of write unit start.	core_id, node_id, server_id
`lightbox_be_write_units_end`	Counter	Number of write unit end.	core_id, node_id, server_id
`lightbox_be_idle_cycles`	Gauge	GFTL idle cycles.	core_id, node_id, role, server_id
`lightbox_be_busy_cycles`	Gauge	GFTL busy cycles.	core_id, node_id, role, server_id
`lightbox_be_num_resets`	Counter	Number of SSD device resets.	device_name
`lightbox_be_read_errors`	Counter	Number of SSD device read errors.	core_id, device_name
`lightbox_be_write_errors`	Counter	Number of SSD device write errors.	core_id, device_name
`lightbox_be_add_disk_success`	Counter	Number of successfully added SSD devices.	node_id, server_id
`lightbox_be_add_disk_failure`	Counter	Number of failures to add SSD devices.	node_id, server_id
`lightbox_be_num_disk_failures`	Counter	Number of SSD devices failures.	node_id, server_id
`lightbox_be_num_of_gc_skips`	Counter	Number of gc skips.	core_id, node_id, server_id
`lightbox_be_write_unit_duration_sum`	Counter	Write unit count - number of write units written since boot (counter); this includes both user and gc write units.	core_id, node_id, server_id
`lightbox_be_read_storage_duration_sum`	Counter	Read storage duration sum.	core_id, node_id, server_id
`lightbox_be_read_storage_duration_count`	Counter	Read storage duration count.	core_id, node_id, server_id
`lightbox_be_no_available_lf_cmds`	Counter	Number of unavailable lf commands.	core_id, node_id, server_id
`lightbox_be_write_buffer_allocation_failure`	Counter	Number of write buffer allocation failure.	core_id, node_id, server_id
`lightbox_be_write_buffer_allocation_success`	Counter	Number of write buffer allocation success.	core_id, node_id, server_id
`lightbox_be_num_blkdev_threshold_reached`	Counter	Number of inflight read requests.	core_id, node_id, server_id
`lightbox_be_inflight_4k_requests_irate`	Counter	Number of inflight read requests.	core_id, node_id, server_id
`lightbox_be_total_units`	Counter	Total units already recovered.	node_id, server_id
`lightbox_be_num_processed_storage_scheme_units`	Counter	Number of processed units.	node_id, server_id
`lightbox_be_num_completed_units`	Counter	Number of completed recovery units.	node_id, server_id
`lightbox_be_num_units_to_recover`	Counter	Number of units to be recovered.	node_id, server_id
`lightbox_be_effective_write_unit_size`	Counter	Write unit size.	node_id, server_id
`lightbox_be_cherry_picked_units`	Counter	Number of write units cherry picked by GC.	core_id, node_id, server_id
`lightbox_be_oldest_generation`	Counter	Generation of the oldest write unit.	core_id, node_id, server_id
`lightbox_be_oldest_valid_generation`	Counter	Generation of the next write unit to be cherry picked.	core_id, node_id, server_id
`lightbox_be_writer_effective_position`	Counter	Generation of the newest write unit.	core_id, node_id, server_id
`lightbox_be_processed_delete_objects`	Counter	Delete objects processed by writer_lf.	core_id, node_id, server_id
`lightbox_be_delete_objs_not_canceled`	Counter	Number of delete objects not canceled.	core_id, node_id, server_id
`lightbox_be_delete_total_blocks_not_canceled`	Counter	Number of delete 4k blocks not canceled.	core_id, node_id, server_id
`lightbox_be_remove_delete_objects`	Counter	Number of discarded trim tombstones.	core_id, node_id, server_id

duroslight

Exposes information from the Duroslight service (Lightbits FE).

Name	Type	Comments	Labels
`lightbox_alien_receive_batch_queue_length`	Gauge	Current received batch queue length.	nodeid, server id, shard
`lightbox_alien_total_received_messages`	Counter	Total number of received messages.	nodeid, server id, shard
`lightbox_alien_total_sent_messages`	Counter	Total number of sent messages.	nodeid, server id, shard
`lightbox_fe_compare_handler_nr_compare_cmds`	Counter	Total number of compare commands.	core_id, node_id, server_id, shard
`lightbox_fe_compare_handler_nr_compare_mismatch`	Counter	Number of objects compare mismatch.	core_id, node_id, server_id, shard
`lightbox_fe_compare_handler_nr_compare_mismatch_small_obj`	Counter	Number of small objects compare mismatch.	core_id, node_id, server_id, shard
`lightbox_fe_compare_handler_nr_compare_mismatch_wb`	Counter	Number of objects compare mismatch served from the write-buffer.	core_id, node_id, server_id, shard
`lightbox_fe_compare_handler_nr_compare_obj_hit_wb`	Counter	Number of compare objects served from the write-buffer.	core_id, node_id, server_id, shard
`lightbox_fe_ctrl_id_used`	Counter	Number of ctrls used per pool.	node_id, pool_id, server_id, shard
`lightbox_fe_failed_abort_commands`	Counter	Number of failed abort commands.	core_id, node_id, server_id, shard
`lightbox_fe_incoming_protocol_log_page_reqs`	Gauge	Number of log page requests.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_log_page_reqs_end`	Gauge	Number of log page requests, ended.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_log_page_reqs_start`	Gauge	Number of log page requests, started.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_log_page_trim_reqs`	Gauge	Number of TRIM log page requests.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_log_page_trim_reqs_end`	Gauge	Number of TRIM log page requests, ended.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_log_page_trim_reqs_start`	Gauge	Number of TRIM log page requests, started.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_pending_packets`	Gauge	Number of pending packets.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_pending_packets_end`	Gauge	Number of pending packets, ended.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_pending_packets_start`	Gauge	Number of pending packets, started.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_read_obj`s	Gauge	Number of read objects.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_read_objs_end`	Gauge	Number of read objects, ended.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_read_objs_start`	Gauge	Number of read objects, started.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_read_reqs`	Gauge	Number of read requests.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_read_reqs_end`	Gauge	Number of read requests, ended.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_read_reqs_start`	Gauge	Number of read requests, started.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_reads_missing_nsid`	Counter	Number of rebuild reads missing nsid.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_replication_errors`	Gauge	Number of replication errors.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_replication_requests`	Gauge	Number of replication requests.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_replication_requests_end`	Gauge	Number of replication requests, ended.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_incoming_protocol_replication_requests_start`	Gauge	Number of replication requests, started.	core_id, counter, node_id, remote_addr, server_id, shard
`lightbox_fe_limiter_outstanding_read_requests`	Gauge	Number of limiter outstanding read requests.	core_id, node_id, server_id, shard
`lightbox_fe_limiter_outstanding_write_requests`	Gauge	Number of limiter outstanding write requests..	core_id, node_id, server_id, shard
`lightbox_fe_limiter_read_timeouts`	Counter	Number of limiter read timeouts.	core_id, node_id, server_id, shard
`lightbox_fe_limiter_write_timeouts`	Counter	Number of limiter write timeouts.	core_id, node_id, server_id, shard
`lightbox_fe_n_replicator_trim_requests_rx`	Counter	Number of replicator trim commands per tx.	core_id, node_id, server_id, shard
`lightbox_fe_n_replicator_write_requests_rx`	Counter	Number of replicator write requests per rx.	core_id, node_id, server_id, shard
`lightbox_fe_n_replicator_write_requests_tx`	Counter	Number of replicator write requests per tx.	core_id, node_id, server_id, shard
`lightbox_fe_n_volume_replicator_write_requests_rx`	Gauge	Number of replicator write requests rx per volume.	core_id, node_id, server_id, shard
`lightbox_fe_n_volume_replicator_write_requests_tx`	Gauge	Number of replicator write requests tx per volume.	core_id, node_id, server_id, shard
`lightbox_fe_nr_fuse_requests`	Counter	Number of fuse requests.	core_id, node_id, server_id, shard
`lightbox_fe_nr_read_bytes_0_1K`	Gauge	Read bytes for 0-1K requests.	core_id, node_id, server_id, shard
`lightbox_fe_nr_read_bytes_1K_8K`	Gauge	Read bytes for 1K-8K requests.	core_id, node_id, server_id, shard
`lightbox_fe_nr_read_bytes_32K_128K`	Gauge	Read bytes for 32K-128K requests.	core_id, node_id, server_id, shard
`lightbox_fe_nr_read_bytes_8K_32K`	Gauge	Read bytes for 8K-32K requests.	core_id, node_id, server_id, shard
`lightbox_fe_nr_read_requests_0_1K`	Gauge	Read requests of 0-1K.	core_id, node_id, server_id, shard
`lightbox_fe_nr_read_requests_1K_8K`	Gauge	Read requests of 1K-8K.	core_id, node_id, server_id, shard
`lightbox_fe_nr_read_requests_32K_128K`	Gauge	Read requests of 32K-128K.	core_id, node_id, server_id, shard
`lightbox_fe_nr_read_requests_8K_32K`	Gauge	Read requests of 8K-32K.	core_id, node_id, server_id, shard
`lightbox_fe_nr_write_bytes_0_1K`	Gauge	Write bytes for 0-1K requests.	core_id, node_id, server_id, shard
`lightbox_fe_nr_write_bytes_1K_8K`	Gauge	Write bytes for 1K-8K requests.	core_id, node_id, server_id, shard
`lightbox_fe_nr_write_bytes_32K_128K`	Gauge	Write bytes for 32K-128K requests.	core_id, node_id, server_id, shard
`lightbox_fe_nr_write_bytes_8K_32K`	Gauge	Write bytes for 8K-32K requests.	core_id, node_id, server_id, shard
`lightbox_fe_nr_write_requests_0_1K`	Gauge	Write requests of 0-1K.	core_id, node_id, server_id, shard
`lightbox_fe_nr_write_requests_1K_8K`	Gauge	Write requests of 1K-8K.	core_id, node_id, server_id, shard
`lightbox_fe_nr_write_requests_32K_128K`	Gauge	Write requests of 32K-128K.	core_id, node_id, server_id, shard
`lightbox_fe_nr_write_requests_8K_32K`	Gauge	Write requests of 8K-32K.	core_id, node_id, server_id, shard
`lightbox_fe_num_tcp_connections`	Gauge	Total number of TCP connections.	core_id, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_canceled_acked_replication_requests`	Counter	Number of cancelled acked replication requests.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_canceled_acked_sync_requests`	Counter	Number of cancelled acked sync requests.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_canceled_replication_requests`	Counter	Number of cancelled replication requests.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_canceled_requests_pending_persistence_in_storage`	Counter	Number of cancelled requests persistent in storage	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_canceled_sync_requests`	Counter	Number of cancelled sync requests.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_log_page_reqs`	Gauge	Number of log page requests.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_log_page_reqs_end`	Gauge	Number of log page requests, ended.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_log_page_reqs_start`	Gauge	Number of log page requests, started.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_read_reqs`	Gauge	Number of read requests.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_read_reqs_end`	Gauge	Number of read requests, ended.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_read_reqs_start`	Gauge	Number of read requests, started.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_replication_reqs`	Gauge	Number of replication requests.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_replication_reqs_end`	Gauge	Number of replication requests, ended.	core_id, count, node_id, server_id, shard
`lightbox_fe_outgoing_protocol_replication_reqs_start`	Gauge	Number of replication requests, started.	core_id, count, node_id, server_id, shard
`lightbox_fe_read_handler_read_cmds`	Gauge	Number of inflight read commands.	core_id, node_id, server_id, shard
`lightbox_fe_read_handler_resubmit_read_cmds`	Counter	Number of resubmitted read commands.	core_id, node_id, server_id, shard
`lightbox_fe_read_modify_write_handler_nr_4K_aligned_objs`	Counter	Number of 4k aligned rmw objects.	core_id, node_id, server_id, shard
`lightbox_fe_read_modify_write_handler_nr_merged_objs`	Counter	Number of rmw merged objects.	core_id, node_id, server_id, shard
`lightbox_fe_read_modify_write_handler_nr_read_failed_objs`	Counter	Number of failed reads on rmw objects.	core_id, node_id, server_id, shard
`lightbox_fe_read_modify_write_handler_nr_read_modify_objs`	Counter	Number of rmw objects.	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_0_1K_avg`	Gauge	Read requests for 0-1K size average duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_0_1K_count`	Gauge	Read requests for 0-1K size sample count.	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_0_1K_sum`	Gauge	Read requests for 0-1K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_1K_8K_avg`	Gauge	Read requests for 1K-8K size average duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_1K_8K_count`	Gauge	Read requests for 1K-8K size sample count.	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_1K_8K_sum`	Gauge	Read requests for 1K-8K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_32K_128K_avg`	Gauge	Read requests for 32K-128K size average duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_32K_128K_count`	Gauge	Read requests for 32K-128K size sample count.	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_32K_128K_sum`	Gauge	Read requests for 32K-128K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_8K_32K_avg`	Gauge	Read requests for 8K-32K size average duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_8K_32K_count`	Gauge	Read requests for 8K-32K size sample count.	core_id, node_id, server_id, shard
`lightbox_fe_read_request_duration_8K_32K_sum`	Gauge	Read requests for 8K-32K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_replicator_write_bandwidth_rx`	Counter	Number of replicator write bytes per rx.	core_id, node_id, server_id, shard
`lightbox_fe_replicator_write_bandwidth_tx`	Counter	Number of replicator write bytes per tx.	core_id, node_id, server_id, shard
`lightbox_fe_retries_after_version_error`	Counter	Number of replicator retries after version error.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_current_load`	Gauge	Current load in this shard, in percentage.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_memory_allocated_memory`	Gauge	Amount of allocated memory for the core.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_memory_cross_cpu_frees`	Counter	Number of cross-CPU frees.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_memory_free_memory`	Gauge	Amount of free memory for core.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_memory_large_allocations`	Counter	Number of large memory allocations.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_memory_live_objects`	Gauge	Number of live objects.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_memory_reclaims`	Counter	Number of memory reclaims.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_memory_total_memory`	Counter	Total amount of memory for the core.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_total_polls`	Counter	Total number of shard polls.	core_id, node_id, server_id, shard
`lightbox_fe_seastar_total_tasks`	Counter	Total number of shard tasks.	core_id, node_id, server_id, shard
`lightbox_fe_semaphore_service_allocated_units_r`	Gauge	Allocated number of read sem units.	core_id, name, node_id, server_id, shard
`lightbox_fe_semaphore_service_allocated_units_w`	Gauge	Allocated number of write sem units.	core_id, name, node_id, server_id, shard
`lightbox_fe_semaphore_service_req_units_r`	Gauge	Number of inflight read sem waits.	core_id, name, node_id, server_id, shard
`lightbox_fe_semaphore_service_req_units_w`	Gauge	Number of inflight write sem waits.	core_id, name, node_id, server_id, shard
`lightbox_fe_semaphore_service_total_units_r`	Gauge	Total number of read sem units.	core_id, name, node_id, server_id, shard
`lightbox_fe_semaphore_service_total_units_w`	Gauge	Total number of write sem units.	core_id, name, node_id, server_id, shard
`lightbox_fe_semaphore_service_wait_count_r`	Gauge	Read sem wait count.	core_id, name, node_id, server_id, shard
`lightbox_fe_semaphore_service_wait_count_w`	Gauge	Write sem wait count.	core_id, name, node_id, server_id, shard
`lightbox_fe_semaphore_service_wait_time_r`	Gauge	Read sem wait time.	core_id, name, node_id, server_id, shard
`lightbox_fe_semaphore_service_wait_time_w`	Gauge	Write sem wait time.	core_id, name, node_id, server_id, shard
`lightbox_fe_stage_buffer_objects_inserted`	Counter	Number of objects inserted.	core_id, node_id, server_id, shard
`lightbox_fe_stage_buffer_objects_override`	Counter	Number of objects overridden.	core_id, node_id, server_id, shard
`lightbox_fe_stage_buffer_objects_removed`	Counter	Number of objects removed.	core_id, node_id, server_id, shard
`lightbox_fe_success_abort_commands`	Counter	Number of successful abort commands.	core_id, node_id, server_id, shard
`lightbox_fe_trim_request_duration_count`	Counter	Total number of aggregate TRIM requests.	core_id, node_id, server_id, shard
`lightbox_trim_request_duration_sum`	Counter	Number of TRIM requests.	core_id, node_id, server_id, shard
`lightbox_fe_volume_read_qos_wait_count`	Gauge	Read request count for qos volume.	nodeid, project id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_qos_wait_time`	Gauge	Read request qos wait time (microseconds).	nodeid, project id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_request_duration_0_1K_count`	Gauge	Volume read requests for 0-1K size count.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_request_duration_0_1K_sum`	Gauge	Volume read requests for 0-1K size total duration (seconds).	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_request_duration_1K_8K_count`	Gauge	Volume read requests for 1K-8K size count.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_request_duration_1K_8K_sum`	Gauge	Volume read requests for 1K-8K size total duration (seconds).	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_request_duration_32K_128K_count`	Gauge	Volume read requests for 32K-128K size count.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_request_duration_32K_128K_sum`	Gauge	Volume read requests for 32K-128K size total duration (seconds).	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_request_duration_8K_32K_count`	Gauge	Volume read requests for 8K-32K size count.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_request_duration_8K_32K_sum`	Gauge	Volume read requests for 8K-32K size total duration (seconds).	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_requests_0_1K`	Gauge	Read requests for 0-1K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_requests_1K_8K`	Gauge	Read requests for 1K-8K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_requests_32K_128K`	Gauge	Read requests for 32K-128K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_requests_8K_32K`	Gauge	Read requests for 8K-32K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_total_0_1K`	Gauge	Read bytes for 0-1K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_total_1K_8K`	Gauge	Read bytes for 1K-8K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_total_32K_128K`	Gauge	Read bytes for 32K-128K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_read_total_8K_32K`	Gauge	Read bytes for 8K-32K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_replicator_write_bandwidth_rx`	Gauge	replicator_write_bandwidth_rx	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_replicator_write_bandwidth_tx`	Gauge	replicator_write_bandwidth_tx	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_qos_wait_count`	Gauge	Write request count for qos volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_qos_wait_time`	Gauge	Write request qos wait time (microseconds).	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_request_duration_0_1K_count`	Gauge	Volume write requests for 0-1K size count	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_request_duration_0_1K_sum`	Gauge	Volume write requests for 0-1K size total duration (seconds).	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_request_duration_1K_8K_count`	Gauge	Volume write requests for 1K-8K size count.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_request_duration_1K_8K_sum`	Gauge	Volume write requests for 1K-8K size total duration (seconds).	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_request_duration_32K_128K_count`	Gauge	Volume write requests for 32K-128K size count.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_request_duration_32K_128K_sum`	Gauge	Volume write requests for 32K-128K size total duration (seconds).	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_request_duration_8K_32K_count`	Gauge	Volume write requests for 8K-32K size count.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_request_duration_8K_32K_sum`	Gauge	Volume write requests for 8K-32K size total duration (seconds).	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_requests_0_1K`	Gauge	Write requests for 0-1K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_requests_1K_8K`	Gauge	Write requests for 1K-8K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_requests_32K_128K`	Gauge	Write requests for 32K-128K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_requests_8K_32K`	Gauge	Write requests for 8K-32K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_total_0_1K`	Gauge	Write bytes for 0-1K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_total_1K_8K`	Gauge	Write bytes for 1K-8K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_total_32K_128K`	Gauge	Write bytes for 32K-128K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_volume_write_total_8K_32K`	Gauge	Write bytes for 8K-32K size per volume.	node_id, project_id, project_name, server_id, volume_id, volume_name
`lightbox_fe_write_buffer_double_dup_objects_inserted`	Counter	Number of duplicate objects double inserted.	core_id, node_id, server_id, shard
`lightbox_fe_write_buffer_dup_objects_inserted`	Counter	Number of duplicate objects inserted.	core_id, node_id, server_id, shard
`lightbox_fe_write_buffer_inflight_objects`	Gauge	Number of objects inflight.	core_id, node_id, server_id, shard
`lightbox_fe_write_buffer_max_inflight_objects`	Gauge	Maximum number of inflight objects.	core_id, node_id, server_id, shard
`lightbox_fe_write_buffer_nr_write_ready`	Gauge	Number of objects that are write-ready.	core_id, node_id, server_id, shard
`lightbox_fe_write_buffer_objects_found`	Counter	Number of objects found.	core_id, node_id, server_id, shard
`lightbox_fe_write_buffer_objects_inserted`	Counter	Number of objects inserted.	core_id, node_id, server_id, shard
`lightbox_fe_write_buffer_objects_polled`	Counter	Number of objects polled.	core_id, node_id, server_id, shard
`lightbox_fe_write_buffer_objects_removed`	Counter	Number of objects removed.	core_id, node_id, server_id, shard
`lightbox_fe_write_handler_flush_due_to_resubmit_cmd`	Counter	Number of times a flush occurred to a resubmit of a write command.	core_id, node_id, server_id, shard
`lightbox_fe_write_handler_resubmit_write_cmds`	Counter	Number of resubmitted write commands.	core_id, node_id, server_id, shard
`lightbox_fe_write_handler_write_cmds`	Gauge	Number of inflight write commands.	core_id, node_id, server_id, shard
`lightbox_fe_write_handler_write_reqs`	Gauge	Number of inflight write requests.	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_0_1K_avg`	Gauge	Write requests for 0-1K size average duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_0_1K_count`	Gauge	Write requests for 0-1K size sample count.	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_0_1K_sum`	Gauge	Write requests for 0-1K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_1K_8K_avg`	Gauge	Write requests for 1K-8K size average duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_1K_8K_count`	Gauge	Write requests for 1K-8K size sample count.	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_1K_8K_sum`	Gauge	Write requests for 1K-8K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_32K_128K_avg`	Gauge	Write requests for 32K-128K size average duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_32K_128K_count`	Gauge	Write requests for 32K-128K size sample count.	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_32K_128K_sum`	Gauge	Write requests for 32K-128K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_8K_32K_avg`	Gauge	Write requests for 8K-32K size average duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_8K_32K_count`	Gauge	Write requests for 8K-32K size sample count.	core_id, node_id, server_id, shard
`lightbox_fe_write_request_duration_8K_32K_sum`	Gauge	Write requests for 8K-32K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_data_0_1K_count`	Gauge	Write latency to data done for 0-1K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_data_0_1K_sum`	Gauge	Write latency to data done for 0-1K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_data_1K_8K_count`	Gauge	Write latency to data done for 1K-8K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_data_1K_8K_sum`	Gauge	Write latency to data done for 1K-8K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_data_32K_128K_count`	Gauge	Write latency to data done for 32K-128K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_data_32K_128K_sum`	Gauge	Write latency to data done for 32K-128K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_data_8K_32K_count`	Gauge	Write latency to data done for 8K-32K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_data_8K_32K_sum`	Gauge	Write latency to data done for 8K-32K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_r2t_0_1K_count`	Gauge	Write latency to r2t data 0-1K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_r2t_0_1K_sum`	Gauge	Write latency to r2t data 0-1K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_r2t_1k_8K_count`	Gauge	Write latency to r2t data 1k-8K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_r2t_1k_8K_sum`	Gauge	Write latency to r2t data 1k-8K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_r2t_32k_128K_count`	Gauge	Write latency to r2t data 32k-128K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_r2t_32k_128K_sum`	Gauge	Write latency to r2t data 32k-128K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_r2t_8k_32K_count`	Gauge	Write latency to r2t data 8k-32K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_r2t_8k_32K_sum`	Gauge	Write latency to r2t data 8k-32K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_0_1K_count`	Gauge	Write latency to repl for 0-1K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_0_1K_sum`	Gauge	fe wrlat_repl_0_1K_sum).Write latency to repl for 0-1K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_1K_8K_count`	Gauge	Write latency to repl for 1K-8K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_1K_8K_sum`	Gauge	Write latency to repl for 1K-8K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_32K_128K_count`	Gauge	Write latency to repl for 32K-128K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_32K_128K_sum`	Gauge	Write latency to repl for 32K-128K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_8K_32K_count`	Gauge	Write latency to repl for 8K-32K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_8K_32K_sum`	Gauge	Write latency to repl for 8K-32K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_ackd_0_1K_count`	Gauge	Write latency to repl_ackd for 0-1K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_ackd_0_1K_sum`	Gauge	Write latency to repl_ackd for 0-1K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_ackd_1K_8K_count`	Gauge	Write latency to repl_ackd for 1K-8K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_ackd_1K_8K_sum`	Gauge	Write latency to repl_ackd for 1K-8K size total duration (microseconds)	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_ackd_32K_128K_count`	Gauge	Write latency to repl_ackd for 32K-128K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_ackd_32K_128K_sum`	Gauge	Write latency to repl_ackd for 32K-128K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_ackd_8K_32K_count`	Gauge	Write latency to repl_ackd for 8K-32K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_ackd_8K_32K_sum`	Gauge	Write latency to repl_ackd for 8K-32K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_sent_0_1K_count`	Gauge	Write latency to repl_sent for 0-1K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_sent_0_1K_sum`	Gauge	Write latency to repl_sent for 0-1K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_sent_1K_8K_count`	Gauge	Write latency to repl_sent for 1K-8K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_sent_1K_8K_sum`	Gauge	Write latency to repl_sent for 1K-8K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_sent_32K_128K_count`	Gauge	Write latency to repl_sent for 32K-128K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_sent_32K_128K_sum`	Gauge	Write latency to repl_sent for 32K-128K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_sent_8K_32K_count`	Gauge	Write latency to repl_sent for 8K-32K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_repl_sent_8K_32K_sum`	Gauge	Write latency to repl_sent for 8K-32K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sec_persist_0_1K_count`	Gauge	Write latency to sec_persist for 0-1K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sec_persist_0_1K_sum`	Gauge	Write latency to sec_persist for 0-1K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sec_persist_1K_8K_count`	Gauge	Write latency to sec_persist for 1K-8K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sec_persist_1K_8K_sum`	Gauge	Write latency to sec_persist for 1K-8K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sec_persist_32K_128K_count`	Gauge	Write latency to sec_persist for 32K-128K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sec_persist_32K_128K_sum`	Gauge	Write latency to sec_persist for 32K-128K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sec_persist_8K_32K_count`	Gauge	Write latency to sec_persist for 8K-32K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sec_persist_8K_32K_sum`	Gauge	Write latency to sec_persist for 8K-32K size total duration (microseconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sharded_0_1K_count`	Gauge	Write latency to sharded for 0-1K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sharded_0_1K_sum`	Gauge	Write latency to sharded for 0-1K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sharded_1K_8K_count`	Gauge	Write latency to sharded for 1K-8K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sharded_1K_8K_sum`	Gauge	Write latency to sharded for 1K-8K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sharded_32K_128K_count`	Gauge	Write latency to sharded for 32K-128K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sharded_32K_128K_sum`	Gauge	Write latency to sharded for 32K-128K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sharded_8K_32K_count`	Gauge	Write latency to sharded for 8K-32K size count.	core_id, node_id, server_id, shard
`lightbox_fe_wrlat_sharded_8K_32K_sum`	Gauge	Write latency to sharded for 8K-32K size total duration (seconds).	core_id, node_id, server_id, shard
`lightbox_httpd_connections_current`	Gauge	Current number of open connections.	node_id,server_id,service,shard
`lightbox_httpd_connections_total`	Counter	Total number of connections opened.	node_id,server_id,service,shard
`lightbox_httpd_read_errors`	Counter	Total number of errors while reading HTTP requests.	node_id,server_id,service,shard
`lightbox_httpd_reply_errors`	Counter	Total number of errors while replying to HTTP.	node_id,server_id,service,shard
`lightbox_httpd_requests_served`	Counter	Total number of HTTP requests served.	node_id,server_id,service,shard
`lightbox_memory_allocated_memory`	Gauge	Allocated memory size in bytes.	node_id, server_id, shard
`lightbox_memory_cross_cpu_free_operations`	Counter	Total number of cross-CPUs free.	node_id, server_id, shard
`lightbox_memory_free_memory`	Gauge	Free memory size in bytes.	node_id, server_id, shard
`lightbox_memory_free_operations`	Counter	Total number of free operations.	node_id, server_id, shard
`lightbox_memory_failed`	Counter	Total count of failed memory allocations.	node_id, server_id, shard
`lightbox_memory_malloc_live_objects`	Gauge	Number of live objects.	node_id, server_id, shard
`lightbox_memory_malloc_operations`	Counter	Total number of malloc operations.	node_id, server_id, shard
`lightbox_memory_reclaims_operations`	Counter	Total reclaims operations.	node_id, server_id, shard
`lightbox_memory_total_memory`	Gauge	Total memory size in bytes.	node_id, server_id, shard
`lightbox_reactor_abandoned_failed_futures`	Counter	Total number of abandoned failed futures and futures destroyed while still containing an exception.	node_id, server_id, shard
`lightbox_reactor_aio_bytes_read`	Counter	Total aio-reads bytes.	node_id, server_id, shard
`lightbox_reactor_aio_bytes_write`	Counter	Total aio-writes bytes.	node_id, server_id, shard
`lightbox_reactor_aio_errors`	Counter	Total aio errors.	node_id, server_id, shard
`lightbox_reactor_aio_outsizes`	Counter	Total number of aio operations that exceed the IO limit.	node_id, server_id, shard
`lightbox_reactor_aio_reads`	Counter	Total aio-reads operations.	node_id, server_id, shard
`lightbox_reactor_aio_writes`	Counter	Total aio-writes operations.	node_id, server_id, shard
`lightbox_reactor_awake_time_ms_total`	Counter	Total reactor awake time (wall clock).	node_id, server_id, shard
`lightbox_reactor_cpp_exceptions`	Counter	Total number of C++ exceptions.	node_id, server_id, shard
`lightbox_reactor_cpu_busy_ms`	Counter	Total CPU busy time in milliseconds.	node_id, server_id, shard
`lightbox_reactor_cpu_steal_time_ms`	Counter	Total steal time - the time in which something else was running while the reactor was runnable (not sleeping). Because this is in userspace, some time that could be legitimately thought of as steal time is not accounted as such. For example, if we are sleeping and can wake up but the kernel has not woken us up yet.	node_id, server_id, shard
`lightbox_reactor_cpu_steal_time_ms`	Counter	Total steal time - the time in which something else was running while the reactor was runnable (not sleeping). Because this is in userspace, some time that could be legitimately thought of as steal time is not accounted as such. For example, if we are sleeping and can wake up but the kernel has not woken us up yet.	node_id, server_id, shard
`lightbox_reactor_cpu_used_time_ms`	Counter	Total reactor thread CPU time (from CLOCK_THREAD_CPUTIME).	node_id, server_id, shard
`lightbox_reactor_fstream_read_bytes_blocked`	Counter	Counts the number of bytes read from disk that could not be satisfied from read-ahead buffers, and had to block. Indicates short streams, or incorrect read ahead configuration.	node_id, server_id, shard
`lightbox_reactor_fstream_reads`	Counter	Counts reads from disk file streams. A high rate indicates high disk activity. Contrast with other fstream_read* counters to locate bottlenecks.	node_id, server_id, shard
`lightbox_reactor_fstream_reads_ahead_bytes_discarded`	Counter	Counts the number of buffered bytes that were read ahead of time and were discarded because they were not needed, wasting disk bandwidth. Indicates over-eager read ahead configuration.	node_id, server_id, shard
`lightbox_reactor_fstream_reads_aheads_discarded`	Counter	Counts the number of times a buffer that was read ahead of time and was discarded because it was not needed, wasting disk bandwidth. Indicates over-eager read ahead configuration.	node_id, server_id, shard
`lightbox_reactor_fstream_reads_blocked`	Counter	Counts the number of times a disk read could not be satisfied from read-ahead buffers, and had to block. Indicates short streams, or an incorrect read ahead configuration.	node_id, server_id, shard
`lightbox_reactor_fsyncs`	Counter	Total number of fsync operations.	node_id, server_id, shard
`lightbox_reactor_io_threaded_fallbacks`	Counter	Total number of IO-threaded-fallbacks operations.	node_id, server_id, shard
`lightbox_reactor_logging_failures`	Counter	Total number of logging failures.	node_id, server_id, shard
`lightbox_reactor_polls`	Counter	Number of times pollers were executed.	node_id, server_id, shard
`lightbox_reactor_sleep_time_ms_total`	Counter	Total reactor sleep time (wall clock).	node_id, server_id, shard
`lightbox_reactor_tasks_pending`	Gauge	Number of pending tasks in the queue.	node_id, server_id, shard
`lightbox_reactor_tasks_processed`	Counter	Total tasks processed.	node_id, server_id, shard
`lightbox_reactor_timers_pending`	Gauge	Number of tasks in the timer-pending queue.	node_id, server_id, shard
`lightbox_reactor_utilization`	Gauge	CPU utilization.	node_id, server_id, shard
`lightbox_scheduler_queue_length`	Gauge	Size of backlog on this queue, in tasks; indicates whether the queue is busy and/or contended.	group,node_id,server_id,shard
`lightbox_scheduler_runtime_ms`	Counter	Accumulated runtime of this task queue; an increment rate of 1000ms per second indicates full utilization.	group,node_id,server_id,shard
`lightbox_scheduler_shares`	Gauge	Shares allocated to this queue.	group,node_id,server_id,shard
`lightbox_scheduler_starvetime_ms`	Counter	Accumulated starvation time of this task queue; an increment rate of 1000ms per second indicates the scheduler is unwell.	group,node_id,server_id,shard
`lightbox_scheduler_tasks_processed`	Counter	Count of tasks executing on this queue; indicates together with runtime_ms the length of tasks.	group,node_id,server_id,shard
`lightbox_scheduler_time_spent_on_task_quota_violations_ms`	Counter	Total amount in milliseconds tin violation of the task quota.	group,node_id,server_id,shard
`lightbox_scheduler_waittime_ms`	Counter	Accumulated waittime of this task queue; an increment rate of 1000ms per second indicates that the queue is waiting for something (e.g., IO).	group,node_id,server_id,shard

meminfo

OS memory usage and export information, from linux /proc/meminfo.

Name	Type	Comments
`lightbox_meminfo_Active_anon_bytes`	Gauge
`lightbox_meminfo_Active_bytes`	Gauge	Memory that has been used more recently, and usually not reclaimed unless absolutely necessary.
`lightbox_meminfo_Active_file_bytes`	Gauge
`lightbox_meminfo_AnonHugePages_bytes`	Gauge
`lightbox_meminfo_AnonPages_bytes`	Gauge
`lightbox_meminfo_Bounce_bytes`	Gauge
`lightbox_meminfo_Buffers_bytes`	Gauge	Memory in buffer cache in bytes.
`lightbox_meminfo_Cached_bytes`	Gauge	Memory in the pagecache (Diskcache) minus SwapCache. Does not include SwapCached.
`lightbox_meminfo_CmaFree_bytes`	Gauge
`lightbox_meminfo_CmaTotal_bytes`	Gauge
`lightbox_meminfo_CommitLimit_bytes`	Gauge
`lightbox_meminfo_Committed_AS_bytes`	Gauge	An estimate of how much RAM you would need to make a 99.99% guarantee that there is never OOM (out of memory) for this workload. Normally, the kernel will overcommit memory. This means for example that if you do a 1GB malloc, only when you start USING that malloc memory will you get real memory on demand, and only as much as you use. Other cases might include when you mmap a file that is shared only when you write to it, and you get a private copy of that data. While it normally is shared between processes, the Committed_AS is an estimate of how much RAM/swap you would need in the worst case.
`lightbox_meminfo_DirectMap1G_bytes`	Gauge
`lightbox_meminfo_DirectMap2M_bytes`	Gauge
`lightbox_meminfo_DirectMap4k_bytes`	Gauge
`lightbox_meminfo_Dirty_bytes`	Gauge	Memory that is waiting to get written back to the disk.
`lightbox_meminfo_HardwareCorrupted_bytes`	Gauge
`lightbox_meminfo_HugePages_Free`	Gauge
`lightbox_meminfo_HugePages_Rsvd`	Gauge
`lightbox_meminfo_HugePages_Surp`	Gauge
`lightbox_meminfo_HugePages_Total`	Gauge
`lightbox_meminfo_Hugepagesize_bytes`	Gauge
`lightbox_meminfo_Inactive_anon_bytes`	Gauge
`lightbox_meminfo_Inactive_bytes`	Gauge	The total amount of buffer or page cache memory, in bytes, that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes.
`lightbox_meminfo_Inactive_file_bytes`	Gauge
`lightbox_meminfo_KernelStack_bytes`	Gauge
`lightbox_meminfo_Mapped_bytes`	Gauge	Files that have been mapped, such as libraries.
`lightbox_meminfo_MemAvailable_bytes`	Gauge
`lightbox_meminfo_MemFree_bytes`	Gauge	Is sum of LowFree+HighFree (overall stat).
`lightbox_meminfo_MemTotal_bytes`	Gauge	Total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code).
`lightbox_meminfo_Mlocked_bytes`	Gauge
`lightbox_meminfo_NFS_Unstable_bytes`	Gauge
`lightbox_meminfo_PageTables_bytes`	Gauge	Amount of memory dedicated to the lowest level of page tables.
`lightbox_meminfo_SReclaimable_bytes`	Gauge
`lightbox_meminfo_SUnreclaim_bytes`	Gauge
`lightbox_meminfo_ShmemHugePages_bytes`	Gauge
`lightbox_meminfo_ShmemPmdMapped_bytes`	Gauge
`lightbox_meminfo_Shmem_bytes`	Gauge
`lightbox_meminfo_Slab_bytes`	Gauge	In-kernel data structures cache.
`lightbox_meminfo_SwapCached_bytes`	Gauge	Memory that was once swapped out, is swapped back in, but still also is in the swapfile (if memory is needed it does not need to be swapped out again because it is already in the swapfile. This saves I/O).
`lightbox_meminfo_SwapFree_bytes`	Gauge	Total amount of swap memory free. This is memory that has been evicted from RAM, and is temporarily on the disk.
`lightbox_meminfo_SwapTotal_bytes`	Gauge	Total amount of physical swap memory.
`lightbox_meminfo_Unevictable_bytes`	Gauge
`lightbox_meminfo_VmallocChunk_bytes`	Gauge	Largest contigiuous block of vmalloc area that is free.
`lightbox_meminfo_VmallocTotal_bytes`	Gauge	Total size of vmalloc memory area.
`lightbox_meminfo_VmallocUsed_bytes`	Gauge	Amount of vmalloc area used.
`lightbox_meminfo_WritebackTmp_bytes`	Gauge
`lightbox_meminfo_Writeback_bytes`	Gauge	Memory that is actively being written back to the disk.

netdev

Network devices information, fetched from linux /proc/net/dev.

Name	Type	Comments	Labels
`lightbox_netdev_receive_bytes_total`	Gauge	Total received bytes.	device
`lightbox_netdev_receive_compressed_total`	Gauge		device
`lightbox_netdev_receive_drop_total`	Gauge	Total number of packets dropped by the network device.	device
`lightbox_netdev_receive_errs_total`	Gauge	Total number of receive errors detected by the network device.	device
`lightbox_netdev_receive_fifo_total`	Gauge		device
`lightbox_netdev_receive_frame_total`	Gauge		device
`lightbox_netdev_receive_multicast_total`	Gauge		device
`lightbox_netdev_receive_packets_total`	Gauge	Total received packets.	device
`lightbox_netdev_transmit_bytes_total`	Gauge	Total transmitted bytes.	device
`lightbox_netdev_transmit_compressed_total`	Gauge		device
`lightbox_netdev_transmit_drop_total`	Gauge		device
`lightbox_netdev_transmit_errs_total`	Gauge		device
`lightbox_netdev_transmit_fifo_total`	Gauge		device
`lightbox_netdev_transmit_frame_total`	Gauge		device
`lightbox_netdev_transmit_multicast_total`	Gauge		device
`lightbox_netdev_transmit_packets_total`	Gauge	Total transmitted packets.	device

netstat

Network statistics, fetched from linux /proc/net/netstat.

Name	Type	Comments	Labels
`lightbox_netstat_Icmp6_InErrors`	Gauge
`lightbox_netstat_Icmp6_InMsgs`	Gauge
`lightbox_netstat_Icmp6_OutMsgs`	Gauge
`lightbox_netstat_Icmp_InErrors`	Gauge
`lightbox_netstat_Icmp_InMsgs`	Gauge
`lightbox_netstat_Icmp_OutMsgs`	Gauge
`lightbox_netstat_Ip6_InOctets`	Gauge
`lightbox_netstat_Ip6_OutOctets`	Gauge
`lightbox_netstat_IpExt_InOctets`	Gauge
`lightbox_netstat_IpExt_OutOctets`	Gauge
`lightbox_netstat_Ip_Forwarding`	Gauge
`lightbox_netstat_TcpExt_ListenDrops`	Gauge
`lightbox_netstat_TcpExt_ListenOverflows`	Gauge
`lightbox_netstat_TcpExt_SyncookiesFailed`	Gauge
`lightbox_netstat_TcpExt_SyncookiesRecv`	Gauge
`lightbox_netstat_TcpExt_SyncookiesSent`	Gauge
`lightbox_netstat_TcpExt_TCPSynRetrans`	Gauge
`lightbox_netstat_Tcp_ActiveOpens`	Gauge
`lightbox_netstat_Tcp_CurrEstab`	Gauge
`lightbox_netstat_Tcp_InErrs`	Gauge
`lightbox_netstat_Tcp_InSegs`	Gauge
`lightbox_netstat_Tcp_OutSegs`	Gauge
`lightbox_netstat_Tcp_PassiveOpens`	Gauge
`lightbox_netstat_Tcp_RetransSegs`	Gauge
`lightbox_netstat_Udp6_InDatagrams`	Gauge
`lightbox_netstat_Udp6_InErrors`	Gauge
`lightbox_netstat_Udp6_NoPorts`	Gauge
`lightbox_netstat_Udp6_OutDatagrams`	Gauge
`lightbox_netstat_UdpLite6_InErrors`	Gauge
`lightbox_netstat_UdpLite_InErrors`	Gauge
`lightbox_netstat_Udp_InDatagrams`	Gauge
`lightbox_netstat_Udp_InErrors`	Gauge
`lightbox_netstat_Udp_NoPorts`	Gauge
`lightbox_netstat_Udp_OutDatagrams`	Gauge

nvme-errorlog

Exposing NVMe error log statistics per device.

Name	Type	Comments	Labels
`lightbox_nvme_errorlog_entries_total`	Gauge	Sum of `error_count` across NVMe error log entries aggregated by device and status code.	serial_number,node_uuid,sct,sc,error_type,dnr
`lightbox_nvme_errorlog_last_entry_count`	Gauge	The maximum `error_count` field observed for each status code type.	serial_number,node_uuid,sct,sc,error_type,dnr

smart

Exposing disk smart statistics (for more information. See the Linux documentation for the smartctl cmd.

Name	Type	Comments	Labels
`lightbox_smart_available_spare_percent`	Gauge	Available spare [percent].	serial_number, node_uuid, type
`lightbox_smart_available_spare_threshold_percent`	Gauge	Available spare threshold [percent].	serial_number, node_uuid, type
`lightbox_smart_controller_busy_time`	Counter	Controller busy time.	serial_number, node_uuid, type
`lightbox_smart_critical_comp_temperature_threshold_celsius`	Gauge	Critical composite temperature threshold.	serial_number, node_uuid, type
`lightbox_smart_critical_comp_temperature_time`	Counter	Critical composite temperature time.	serial_number, node_uuid, type
`lightbox_smart_critical_warning`	Gauge	Number of critical warnings.	serial_number, node_uuid, type
`lightbox_smart_data_units_read`	Counter	Number of data units read.	serial_number, node_uuid, type
`lightbox_smart_data_units_written`	Counter	Number of data units written.	serial_number, node_uuid, type
`lightbox_smart_error_information_log_entries`	Counter	Number of error information log entries.	serial_number, node_uuid, type
`lightbox_smart_host_read_commands`	Counter	Number of host read commands.	serial_number, node_uuid, type
`lightbox_smart_host_write_commands`	Counter	Number of host write commands.	serial_number, node_uuid, type
`lightbox_smart_media_and_data_integrity_errors`	Counter	Number of media and data integrity errors.	serial_number, node_uuid, type
`lightbox_smart_percentage_used`	Gauge	Percentage used.	serial_number, node_uuid, type
`lightbox_smart_power_cycles`	Counter	Number of power cycles.	serial_number, node_uuid, type
`lightbox_smart_power_on_hours`	Counter	Number of hours powered on.	serial_number, node_uuid, type
`lightbox_smart_temperature_celsius`	Gauge	Temperature [celsius].	serial_number, node_uuid, type
	Counter	Number of host read commands.	serial_number, node_uuid, type
`lightbox_smart_unsafe_shutdowns`	Counter	Number of unsafe shutdowns.	serial_number, node_uuid, type
`lightbox_smart_warning_comp_temperature_threshold_celsius`	Gauge	Warning composite temperature threshold.	serial_number, node_uuid, type
`lightbox_smart_warning_comp_temperature_time`	Counter	Warning composite temperature time.	serial_number, node_uuid, type

Last updated on

Was this page helpful?