By hook or by crook: The technology of Linux rootkits – Part 2: Kernel-mode rootkits

This is the second part of a series on Linux rootkits. The first part focused on userland rootkits: malware that hides in user-space processes by injecting code or hooking libraries using techniques like LD_PRELOAD.

While still a risk, userland rootkits are limited to compromising parts of user space, in a piecemeal manner, rather than the entire system. Kernel-mode rootkits go further, hiding within the operating system’s most privileged layer.

In this post, we explore:

• What makes kernel-mode rootkits so powerful.
• The techniques used to inject code into the kernel.
• Common hooking and hiding strategies.
• Rootkit detection challenges and possible prevention measures.

Kernel-mode rootkits

Linux is a monolithic kernel, a single executable that is loaded into memory when the system boots, and runs entirely in kernel mode with exclusive privileged access to hardware.

Whereas user-mode rootkits assert themselves by diverting code execution in user-space programs, kernel-mode rootkits hook the execution flow of kernel code. This gives them several advantages:

• System-wide impact: Hooks applied at kernel level are effective across all of user space. System call behaviour is modified for all callers regardless of how system calls are invoked.
• Deeper control: Hooks can be placed not just at the level of the system call interface, but on any function in the kernel control path of handler code, or within specific kernel subsystems, for a more precisely targeted effect.
• Data structure tampering: Kernel-mode rootkits can modify dynamic data structures that the kernel uses to represent and track the use of system resources. These so-called direct kernel object manipulations are particularly hard to detect, and because the affected data serve a bookkeeping role, can be used to hide the malware activity even from a kernel-level observer.

So what? Kernel-mode rootkits operate at the heart of the operating system, giving attackers deep, stealthy control. Once installed, they can undermine almost any detection tool or forensic method running on the compromised system itself.

Routes of access into kernel memory

User-mode processes cannot directly access kernel memory. Leaving aside kernel exploits, there aren't many supported means of injecting code into the kernel from user space. Here are three which have seen use in rootkits.

(Legacy) /dev/kmem special device file

A Linux kernel compiled with the configuration flag CONFIG_DEVKMEM provides the special device file /dev/kmem which is a raw representation of the kernel's virtual address space that can be read and written to from userland.

This technique was historically used in rootkits like SuckIt, which used /dev/kmem to patch system call table function pointers so that malicious handler code is activated. However, /dev/kmem is now disabled on most distributions, and this technique has fallen out of common use.

So what? Although largely obsolete, the /dev/kmem technique illustrates how raw access to kernel space – even when limited – can be sufficient to install stealthy hooks. Systems still exposing this interface are at risk.

Loadable kernel modules (LKM)

The primary facility provided by Linux for introducing code into the kernel is via loadable kernel modules (LKM). These are executable object files which can be loaded into a running kernel by a privileged user with the insmod command (which in turn invokes the system call init_modules()).

Upon loading, initialisation code is executed, and this is when the rootkit module attaches its hooks into the kernel.

LKMs have access to:

• Kernel memory.
• The memory of the user-space process running during the switch to kernel mode.

They can directly implement any desired hooking strategy both in the kernel and in user space.  They can also take advantage of the kernel tracing and instrumentation frameworks ftrace and kprobes to attach callbacks to predefined tracepoints and arbitrary functions. The hooking of arbitrary functions by these frameworks is ultimately implemented by inline patching of kernel code.

Many examples with source code of LKM-based rootkits can be found on this github page. Notable ones include Diamorphine, Adore-ng, Reptile, and KoviD.

So what? LKMs remain one of the most effective and widely used rootkit deployment methods. If a system permits unsigned or unverified module loading, attackers can operate at the deepest levels of the OS with full control and stealth.

eBPF (enhanced Berkeley Packet Filters)

Another possible kernel rootkit injection route is via eBPF, a virtual machine running inside the Linux kernel, that executes verified bytecode programs that can be attached and run in reaction to certain types of kernel events.

eBPF programs of Express Data Path (XDP) and Traffic Control (TC) type can filter, redirect and monitor network events at the network interface level. Those of tracepoint and kprobe types can be attached to the tracepoints and kprobes mentioned above in connection with loadable kernel modules. The bpf() system call allows for communication between eBPF programs and user space.

Although they run in kernel mode, eBPF programs are subject to restrictions. In particular:

• They cannot modify kernel code or data.
• They cannot call arbitrary kernel functions.
• Their ability to act on the execution context is limited to the use of a set of helper functions. These however enable eBPF programs to read and modify user-space memory.

As a result, eBPF programs can be used to perform process injection on user-space processes. When attached to tracepoints and kprobes, they can:

• Read and modify the arguments with which the attached-to system call or other kernel function was called with, and
• Override their return values.

This allows eBPF to be used to hook kernel functions. However, arbitrary function pointer modification and inline patching is beyond its capabilities, as is any direct kernel object manipulation.

Proof-of-concept examples of eBPF rootkits include the open source Boopkit and TripleCross. In the wild, eBPF has been seen implementing the network backdoor component of several rootkits.

So what? While more limited than LKMs, eBPF has become a widely available alternative interface for running rootkit code inside the kernel. eBPF rootkits have a lower risk of causing system instability, and are invisible to tools which check only for the presence of LKMs.

Commonly-subverted kernel sites

By deploying a kernel module or an eBPF program, an attacker can proceed to hook code and modify data structures anywhere in the kernel. Below are some parts of the Linux kernel popularly targeted by rootkits.

The system call interface

Hooking system calls at kernel level allows rootkits to subvert Linux’s supported userland-facing ABI.

Performed correctly, kernel-level hooking can be used to exhaustively filter the flow of information into user space. In the other direction, userland applications can invoke hooked system calls to activate malicious routines a rootkit has implanted in the kernel.

An LKM rootkit can hook system calls by:

• Overwriting function pointers in the system call table.
• Switching system call tables by modifying the interrupt handler’s reference.
• Performing inline patching of kernel functions implementing the system call. This is preferred because of mitigations put in place to prevent system call table pointer redirection, such as making the table read-only. Such mitigations can be overridden but doing so strongly signals malicious activity. In any case, the presence of any modified pointers in an otherwise static system call table is a closely-regarded indicator of compromise.

The hooking of system calls can also be done using eBPF, by leveraging the kernel APIs ftrace and kprobes.

Virtual File Systems

The hooking of kernel functions need not be confined to those directly involved in the handling of system calls.

A kernel subsystem popularly abused by rootkits is the Virtual File System (VFS), an abstraction layer that presents a uniform interface for userland interactions with any file system.

These include:

• Storage filesystems like EXT4 and NTFS.
• Pseudo filesystems like procfs, sysfs and devfs.

Under VFS, files are interfaces to handler functions in the kernel. Accessing a file triggers those handlers. Examples:

• /proc/modules, when read, causes the kernel to output its current list of loaded LKMs.
• /proc/sys/net/tcp provides a list of the system’s currently open TCP connections.
• Numerical subdirectories of /proc represent the PIDs of currently running processes.

Thus, by hooking file and inode operation handler functions of virtual filesystems, rootkits hide not just the presence of regular files, but also processes and their activities.

So what? If defenders rely heavily on outputs from procfs or sysfs, those views may be completely controlled and falsified by a rootkit intercepting VFS handlers.

Network Stack

Rootkits can also target the kernel’s network stack. From the initial arrival of a packet at a network interface to when it’s handed off to user space, it’s possible to hook the kernel’s processing of packets as they pass through different points of the network stack.

Hooks applied at an early stage can be used to hide traffic from tools like tcpdump. They can also be used to implement backdoors for remote access and control. A hook function can listen for incoming traffic containing a “magic packet” and if one is present execute instructions encoded in the subsequent part of the stream while hiding this traffic from the rest of the stack. Because the backdoor is dormant until the reception of the magic packet, it is passive and thus stealthy.

The kernel’s Netfilter framework allows hooks to be registered at predefined packet processing stages. eBPF-based rootkits also have the ability to attach eBPF programs to the XDP fast path, very early on at the network interface level, giving them the ability to bypass Netfilter, seccomp and most socket telemetry.

If these popular hooking sites are being watched by defenders, attackers can go deeper and hook underlying routines such as ip_rcv, the kernel’s main IPv4 packet receiving function, using ftrace/kprobes or inline patching.

Dynamic kernel data structures

Beyond hooking kernel functions, LKM rootkits can also modify kernel data structures responsible for bookkeeping in system operations such as process, memory, and filesystem management.

Because these are dynamic data structures, allocated as needed during normal runtime operation and whose values are always subject to change, malicious modifications to them are much harder to detect than kernel hooking, which mostly involves making changes in static memory sections.

The direct manipulation of dynamic kernel objects is, in many cases, essential for achieving complete concealment of system state.

For example, the VFS read handler for /proc/modules dynamically generates its list of loaded kernel modules by iterating through a linked list of such modules that is maintained by the kernel. While hooking the handler function can filter and hide the rootkit’s module from procfs and userland tools like lsmod, the module will still be visible in kernel space unless removed from the module list data structure.

While promising better concealment, the direct manipulation of kernel data structures must be performed carefully. Failure to properly remove or restore elements of these data structures in a thread-safe manner can lead to system instability and an increased likelihood of detection.

Detection and prevention

Detecting rootkits

Rootkits are precisely intended to make malware hard to find. They do this by modifying the behaviour of operating system components.

That makes detecting one a paradoxical task: how do you verify the integrity of a system using tools that might themselves be compromised?

For this reason, much of the art of rootkit detection is an exercise in cross-view analysis – comparing two different views of the system to look for discrepancies. If found, these could indicate the presence of a rootkit.

For example:

• An LD_PRELOAD rootkit might be discovered by comparing the output of ls with directory listings produced by a statically-linked busybox binary.
• A kernel-resident detector could expose userland process hiding by comparing its internal view of the process list with the output of a utility like ps.

Cross-view detection of kernel-mode rootkits ideally requires a trusted perspective from outside the compromised system altogether. Virtualised workloads can take advantage of hypervisor-based introspection, where a virtual machine monitor examines the memory of a guest OS without relying on its internal mechanisms.

So what? Without access to external or trusted sources of truth, many rootkit detection efforts are inherently limited. Hypervisor-based or offline memory analysis often becomes essential for high-assurance environments.

While bearing in mind the fact that any in-system tools used in detection could themselves be compromised, the traditional categories of malware detection approaches still apply to the detection of rootkits.

• Signature-based detection can involve looking in memory for instruction sequences of known rootkit kernel modules.
• An integrity-based approach could detect kernel hooks by looking for the presence of modifications to kernel text, such as inserted jumps at the beginning of functions, or by checking whether any jumps or function pointers executed in kernel mode are directed to addresses outside of valid kernel code regions.
• Behaviour-based methods can detect actions which almost every kernel-mode rootkit performs, such as searching for the addresses of data structures they wish to tamper with.  Here that could involve monitoring calls to the kernel function kallsyms_lookup_name that’s used by loadable kernel modules to get the addresses of unexported kernel symbols.

Rootkits often don’t do a thorough job of removing references to malware from all the kernel structures which track them. As a result, they may leave the kernel in an inconsistent state after their manipulations. This too can be detected.

So what? Even advanced rootkits often leave behind anomalies in kernel metadata. Monitoring for inconsistent process states or unbalanced kernel structures can provide valuable forensic indicators.

Preventing rootkits

At the kernel level, prevention consists foremost in restricting the means of injecting code into kernel memory. If feasible:

• Disable loadable kernel modules by recompiling the kernel with CONFIG_MODULES=n.
• Enforce loading only of signed kernel modules by enabling secure boot and setting module.sig_enforce=1 in the kernel’s command-line parameters.
• Also use Linux Security Modules like SELinux and Apparmor to restrict access to system calls used to load kernel modules and eBPF programs.

At the userland level:

• Use static linking for security-critical utilities to harden them against LD_PRELOAD-style hijacks.
• Mount /etc on a read-only filesystem.

To reduce opportunities for process injection:

• Disable ptrace(), even for root, by setting /proc/sys/kernel/yama/ptrace_scope to 2.
• Alternatively, with SELinux, set the deny_ptrace Boolean to 1 (setsebool -P deny_ptrace 1).

Finally, harden the environment against privilege escalation in the first place:

• Patch vulnerabilities promptly.
• Audit services and daemons running with elevated privileges.
• Isolate workloads using unprivileged containers or virtual machines.
• Most importantly, apply the principle of least privilege everywhere – because if the attacker never gains root in the first place, there will be no planting of rootkits.

So what? Most rootkits depend on prior root access. Strong baseline hardening and isolation practices can greatly reduce, and in many cases prevent, the opportunity for rootkits to be deployed in the first place.

Key Takeaways

• Kernel-mode rootkits are powerful because they operate with full system privileges and can modify both control flow and data inside the kernel. Changes there affect all of user space.
• Common rootkit targets include system call handlers, VFS interfaces, and dynamic data structures used for tracking modules and processes.
• Detection is most effective when comparing multiple views of the system, ideally from trusted and external sources.
• Preventing rootkits means restricting code injection into the kernel, limiting attacker privilege escalation, and applying defence-in-depth controls at both userland and kernel levels.