By hook or by crook: The technology of Linux rootkits – Part 1: Userland rootkits

Your Linux system could be compromised right now – and you might have no idea.

To check, you might begin by looking for unusual processes and network connections, using system utilities or diagnostic software. But what if these are lying to you, or are themselves being fed incomplete or false information from lower levels of the operating system?

This could be the case if your system were infected by advanced malware known as rootkits. Rootkits open up the confidence-sapping possibility that trusted diagnostic tools might be accomplices of malware.

In this two-part series on Linux rootkits, we will first look at threat actors’ motivations for developing them. We will then give a high-level overview of the features of the Linux operating system that constrain their design, leading to their classification into two kinds: userland and kernel rootkits.

This post will examine userland rootkits. The second part of this series will cover kernel rootkits and explore approaches to detection and prevention.

Why do threat actors use rootkits?

After an attacker has gained access to a system and escalated their privileges, typically by exploiting a software vulnerability or misconfiguration, their key objective is to maintain long-term, covert control over the compromised environment. A “root-kit” is what they deploy towards this end.

Rootkits are designed to maintain the stealthy presence of the attacker and to provide persistent means of retaining privileged access to the system.

• Stealth: The stealth functionality includes hiding files, hiding processes, masking resource utilisation, and masking network connections, allowing attackers to operate undetected for extended periods.
• Persistence: Rootkits can alter system behaviour to interfere with user attempts to remove them and other associated malware. Most of them also embed backdoor functionality into the system, permitting attackers to connect to it remotely, bypass authentication checks or grant processes elevated privileges. This provides threat actors with means of regaining access to the compromised system if they lose it.

Rootkits often support other post-exploitation activities, such as surveillance and credential sniffing, but these are secondary to their main purpose of maximising attackers’ control over a system while minimising the risk of discovery.

User space and the kernel in Linux

A Linux system is divided into user space and the kernel. The kernel, which is loaded at boot, handles all interactions with hardware and provides the execution environment for user programs.

User space (or userland) is where these non-kernel (“user”) processes run with limited privileges, isolated from the underlying hardware resources and from each other. In particular, each userland process is given its own virtual address space, and cannot directly access the memory of another process, nor that of the kernel.

Hardware support is needed to implement this fundamental division of privilege, and CPUs provide at least two execution modes:

• Kernel mode, in which the CPU may perform any operation.
• User mode, where privileged operations, which could alter the global state of the system, are not permitted.

When a userland process wants to access a system resource, it must make a system call to request the kernel to perform a privileged operation on its behalf. The CPU will switch to kernel mode to allow kernel routines to handle the system call before returning to user mode upon completion.

System calls thereby constitute the de facto interface between userland and the kernel, but, in most operating systems, applications are discouraged from directly making them – instead it’s expected that they’re called via high-level encapsulations in userland API libraries. Linux, as a standalone kernel project, is a unique exception. It supports its system call interface as a stable, documented and public ABI for direct use by userland applications.

This has made system calls the unrivalled target of Linux rootkits. A typical Linux system might provide around 400 of them, but because of the inherited Unix design pattern of representing system resources as files and interactions with them as read and write operations, a rootkit can accomplish much of its aims just by interfering with those system calls which perform reads and writes.

Rootkits are classified into two kinds, userland and kernel-mode, depending on which side of the system call interface they operate.

Userland rootkits

Early userland rootkits: replacing binaries

Some early userland rootkits were simply collections of programs that replaced key system binaries to hide files and processes, or implement backdoors.

For example, the source code of the ls program could be modified and recompiled to produce an altered Trojan version that does not display any file whose name began with the string rootkit_. This could then be used to hide a SUID privilege escalation backdoor executable file called rootkit_getroot.

Because the programs reside on disk, they are detectable by running file integrity checks against a database of hashes of known good system binaries.

In-memory modification of program behaviour; process injection; process hollowing

To remain hidden, rootkits avoid making changes to disk. Instead, they aim to make their modifications in volatile memory only.

Since each userland process has its own, isolated virtual address space, a userland rootkit must rely on kernel-supported features to inject malicious code into another process. Linux offers three ways to do that:

• ptrace() system call: This system call is intended for debugging purposes, and can be used to attach to a process to inspect and modify its memory and registers.
• procfs pseudo filesystem: The procfs pseudo filesystem represents processes as subdirectories and files in the /proc directory. The memory of a process can be accessed and modified by reading and writing to the file /proc/$PID/mem, where $PID is the ID of the process.
• process_vm_readv() and process_vm_writev() system calls: These system calls allow copying ranges of a remote process’ memory to and from a local buffer.

All three methods require root privileges (specifically, CAP_SYS_PTRACE capability).

By writing code into the memory of a victim legitimate process and executing it instead of what was originally there, a rootkit can run a malicious process disguised as a legitimate one. This is known as process hollowing, and can be used, for instance, to implement a stealthy command and control backdoor for persistent remote access. DEGU is an example of this approach.

So what? Any unexpected use of ptrace() or modification of another process’ memory should be a red flag to defenders. Monitoring access to /proc/$PID/mem can also help identify suspicious behaviour.

Hooking

It is not always desirable to entirely replace the code on an injected process. In particular, to implement a rootkit’s hiding functionality it’s often better to make smaller modifications which alter runtime behaviour more selectively. This allows victim processes to run normally for the most part, while diverting the execution flow for specific functions whose return values might reveal something the rootkit aims to conceal.

The diversion of the flow of code execution is referred to as hooking.

This can involve directly modifying code in memory – which in most rootkit use cases means performing inline patching of instructions in function prologues and epilogues, replacing existing opcodes with jump or trap instructions. Execution flow is thereby redirected to malicious code, which intercepts function calls to perform additional operations and modify return values.

Where functions are called indirectly through variables containing addresses of code, it’s also possible to divert execution flow towards malicious code by modifying these function pointers. This is very common due to the use of dispatch tables of function pointers at many important places in the operating system.

Dynamic linker hijacking and shared library API function hooking

Even with a way of performing process injection, separate process address spaces mean that, at a fundamental level, in-memory manipulation of userland processes needs to be done individually. This approach does not scale well for implementing a userland rootkit that affects all or most userland programs on a system.

Increased comprehensiveness relies on a key observation: to conceal malware activity from user space, rootkits need only to hook the execution paths involved in the invocation of system calls.

Shared library hooking and GOT redirection

Most user-space programs on Linux don't make system calls directly. Instead, they rely on pre-compiled functions in system libraries such as libc (the GNU C standard library), which wrap the underlying system calls in standard API functions.

Typically, library code is not incorporated into executables themselves, but is dynamically linked. This means the executable only contains symbolic references to functions in shared object files (such as libc.so), which are mapped into virtual memory only at runtime. Their addresses are unknown until then.

Dynamically linked executables contain a section known as the Global Offset Table (GOT), which holds addresses of these shared library functions. The dynamic linker updates GOT pointers during runtime, at the first call to the shared library functions.

By replacing GOT pointers with addresses pointing to malicious code, a user-space rootkit can hijack system call invocations made through shared library functions. This refactorisation amplifies the power of userland rootkits – a few changes now suffice to corrupt entire classes of programs.

LD_PRELOAD: easy system-wide hijacking

Linux provides a convenient mechanism for performing this dynamic linker hijacking. The environment variable LD_PRELOAD specifies a list of user-specified shared objects to be loaded before all others. If a malicious shared object listed in LD_PRELOAD defines functions with signatures identical to those in standard libraries, the dynamic linker will update the corresponding GOT entries to point to the malicious versions.

To apply this system-wide to all programs, the malicious libraries can be added to the configuration file /etc/ld.so.preload. This not only ensures the malicious objects are loaded with every executable but also makes the hooks persistent.

The LD_PRELOAD technique is sufficient for implementing much rootkit functionality, and is by far the most widely used technique for making user-space rootkits. Examples can be found here. Simply by hooking the libc functions open(), read(), opendir() and readdir(), for example, it's possible to hide files, processes and network connections from user applications.

Limitations of LD_PRELOAD

However, LD_PRELOAD does not work on programs that:

• Do not use shared libraries to make system calls.
• Are statically linked, meaning library code is included in the executable at compile time and no dynamic linking occurs.

Notably, it is not possible to use LD_PRELOAD to hide its shared object file from the output of ldd, which displays the shared objects used by an executable – because ldd itself is statically linked.

So what? The LD_PRELOAD mechanism remains the most accessible and commonly used entry point for userland rootkits. Its presence can be subtle but system-wide, affecting nearly all user processes. While its limitations are known, the technique remains effective – especially in the typical GNU/Linux userland, where most programs are written using standard libraries and dynamic linking is dominant.

Limitations of userland rootkits

Restricted by the structure of user space in Linux, userland rootkits face key fundamental limitations:

• They cannot hook or intercept the kernel routines which perform the work of system calls.
• They cannot tamper with internal kernel data structures.
• Their coverage cannot be exhaustive, which makes them detectable through cross-view analysis – e.g. for LD_PRELOAD rootkits by comparing outputs of statically-linked and dynamically-linked versions of the same tools.

For these reasons, kernel-mode rootkits, which we will look at in the next part of this series, offer more powerful concealment techniques.

Key takeaways

• Userland rootkits work by modifying user-space processes to hide activity and maintain persistence.
• Techniques include process injection, function hooking, and dynamic linker hijacking using LD_PRELOAD.
• These techniques often leave no permanent trace, making detection harder – but not impossible.
• Limitations of userland rootkits include their inability to access or modify kernel-level data structures.
• Detection strategies such as file integrity checks and cross-view analysis can be effective against them.
• Part 2 will examine kernel-mode rootkits, which operate below the system call interface and offer deeper concealment capabilities.