Lightbits Memory Utilization and Linux Memory Management on x86 Servers

In x86 architecture, memory is managed in pages, with the standard small page size being 4KB. These pages are mapped using a multi-level page table, which helps efficiently translate virtual addresses to physical addresses. However, frequent lookups in the page table can introduce overhead.

To optimize performance, huge pages (2MB) reduce the number of page table entries and improve Translation Lookaside Buffer (TLB) efficiency. By covering larger memory regions with a single entry, huge pages lower memory access latency and reduce page table overhead (CPU utilization).

The Linux kernel memory management typically does not allocate and map physical pages when a user-space process allocates virtual memory. Instead, the allocation and mapping of physical pages occur lazily, meaning they happen only when the memory is accessed (touched) for the first time. This demand-paging approach helps optimize memory usage and defer physical allocation until necessary.

Lightbits' software allocates most of the available system memory for metadata, which is used to determine the physical location (LBA and disk) of a block within a Lightbits volume. This metadata is divided into 9KB chunks, which are allocated at startup and managed within our own memory pool.

During this initial allocation, Linux does not immediately back these virtual memory allocations with physical pages. When these chunks are first accessed (touched), Linux could allocate huge pages (2MB), in which case multiple 9KB chunks are placed within a single 2MB region. Alternatively, Linux could allocate regular 4KB pages, in which case only 4KB is allocated initially, with additional allocations occurring as more of the chunk is accessed. Lightbits touches the header of each chunk at startup, which ensures that at least part of the chunk is allocated and mapped in physical memory early in the process. Depending on the Linux kernel’s behavior, this initial access could trigger either a 4KB or 2MB physical page allocation.