Yesterday, Facebook open sourced a suite of Linux kernel components and tools. This suite includes products that can be used for resource control and utilization, workload isolation, load balancing, measuring, monitoring, and much more. Facebook has already started using these products on a massive scale throughout its infrastructure and many other organizations are also adopting them.
The following are some of the products that they have open sourced:
Berkeley Packet Filter (BPF)
BPF is a highly-flexible Linux kernel code execution engine. It enables safe and easy modifications of kernel behaviors with custom code by allowing bytecode to run at various hook points. Currently, it is being widely used for networking, tracing and security in a number of Linux kernel subsystems.
What can you do with it?
- You can extend the Linux kernel behavior for a variety of purposes such as load balancing, container networking, kernel tracing, monitoring, and security.
- You can solve those production issues where user-space solutions alone aren’t enough by executing the user-space code in the kernel.
Btrfs
Btrfs is a copy-on-write (CoW) filesystem, which means that instead of overwriting in one place, all the updates to metadata or file data are written to a new location on the disk. Btrfs mainly focuses on fault tolerance, repair, and easy administration. It supports features such as snapshots, online defragmentation, pooling, and integrated multiple device support. It is the only filesystem implementation that works with resource isolation.
What can you do with it?
- You can address and manage large storage subsystems by leveraging features like snapshots, load balancing, online defragmentation, pooling, and integrated multiple device support.
- You can manage, detect, and repair errors with data and metadata checksums, mirroring, and file self-healing.
Netconsd (Netconsole daemon)
Netconsd is a UDP-based daemon that provides lightweight transport for Linux netconsole messages. It receives and processes log data from the Linux kernel and serves it up as a structured data. Simply put, it is a kernel module that sends all kernel log messages over the network to another computer, without involving user space.
What can you do with it?
- Detect, reorder, or request retransmission of missing messages with the provided metadata.
- Extract meaningful signal from the data logged by netconsd to rapidly identify and diagnose misbehaving services.
Cgroup2
Cgroup2 is a Linux kernel feature that allows you to group and structure workloads and also control the amount of system resources assigned to each group. It consists of controllers for memory, I/O, central processing unit, and more. Using cgroup2, you can isolate workloads, prioritize, and configure the distribution of resources.
What can you do with it?
- You can create isolated groups of processes and then control and measure the distribution of memory, IO, CPU and other resources for each group.
- You can detect resource shortages using PSI pressure metrics for memory, IO, and CPU with cgroup2.
- With cgroup2, production engineers will be able to deal with increasing resource pressure more proactively and prevent conflicts between workloads.
Along with these products, they have open-sourced Pressure Stall Information (PSI), oomd, and many others.
You can find the complete list of these products at Facebook Open Source website and also check out the official announcement.
Read Next
Facebook open sources QNNPACK, a library for optimized mobile deep learning
Facebook introduces two new AI-powered video calling devices “built with Privacy + Security in mind”
Facebook’s Glow, a machine learning compiler, to be supported by Intel, Qualcomm and others