Linux Incident Response Part 1

In the past months I was working on security incidents involving Linux systems and I struggled to find good material on that topic to structure well my response plan. In fact, Linux is the investigator’s black hole, blind spot, deficiency, weakness and so on. Because there are not enough artifacts to find out to confirm theory of what exactly happened on that system. Also, there is not enough out there on Linux systems Incident Response - not enough tools that can help you perform live response, not enough write-ups on what to look for. I feel we need more on that topic in Forensics & Incident Response community. In fact, Linux powers 37% of the Internet as per current data. I start a series of articles in “how to” style to document for myself incident response on Linux systems and notably I will cover Ubuntu Server (because of its popularity).

Linux Kernel Internals

Linux is the black hole for DFIR analysts First things first, I have to prepare well of what’s coming my way in an Incident Response emergency. I need to know the common file systems that Linux might be coming with. Usually on Ubuntu Server that would be ext4, due to advanced features and speed.

Common Linux file systems comparison table:

If you’re like me coming from the Windows world note that Linux is very limited from forensic artifact perspective. You won’t be able to find juicy, meaty evidence you’re used to on a Windows system. There are certain limitations you’ll face during investigation:

No evidence of execution present on disk (Windows has Prefetch, SuperFetch, ShimCache, UserAssist, Jump Lists, to name but a few) Memory capture is impossible without installing kernel module (LiME LKM). That approach is against all forensic best practices. File system artifacts are limited (ext4, xfs, etc.), lots of metadata is missing. User activity difficult to reconstruct, command history can be tampered with (.bash_history can be modified or deleted by the user). /var/log/ directory can be modified and logs deleted by non-root users. Difficult to impossible to recover files once deleted with EXT4 file system. … and so on. It’s not all bad news though, there is a bright side to Ubuntu and Linux forensics in general. Log files are verbose enough to give you an idea of what happened on the system. On top of that, there is good open source and commercial software for file integrity and security monitoring (OSSEC, Tripwire). Good setup makes a big difference when looking to setup such system.

When analyzing supposedly compromised Ubuntu Server here is a list of things I usually look at:

Suspicious processes
Suspicious network connections
Newly created users
Dubious directories and files
Currently logged users
Bash history - /home/username/.bash_history
Cron jobs (scheduled tasks) - /etc/cron or /var/spool/cron
Malicious startup script common locations:
- /etc/inittab
- /etc/init.d
- /etc/rc.d
- /etc/init.conf
- /etc/init

Common services and their log locations, if I’m doing more targeted analysis:

Location	Information and Log Type
/var/log/messages	General messages/logs and system related stuff
/var/log/auth.log	Authentication logs
/var/log/kern.log	Kernel logs
/var/log/cron.log	Crond logs from cron jobs
/var/log/maillog	Mail server logs
/var/log/qmail/	Qmail log directory (more files exist inside this directory)
/var/log/httpd/	Apache access and error logs directory
/var/log/lighttpd/	Lighttpd access and error logs directory
/var/log/boot.log	System boot log
/var/log/mysqld.log	MySQL database server log file
/var/log/secure or /var/log/auth.log	Authentication log
/var/log/utmp or /var/log/wtmp	Login records file
/var/log/yum.log	Yum command log file.

The standard Ubuntu directory structure mostly follows the Filesystem Hierarchy Standard, which can be referred to for more detailed information.

Location	Purpose
/bin	a place for most commonly used terminal commands, like ls, mount, rm, etc.
/boot	contains files needed to start up the system, including the Linux kernel, a RAM disk image and bootloader configuration files.
/dev	contains all device files, which are not regular files but instead refer to various hardware devices on the system, including hard drives.
/etc	contains system-global configuration files, which affect the system’s behavior for all users.
/home	home sweet home, this is the place for users' home directories.
/lib	contains very important dynamic libraries and kernel modules
/media	intended as a mount point for external devices, such as hard drives or removable media (floppies, CDs, DVDs).
/mnt	a place for mount points, but dedicated specifically to “temporarily mounted” devices, such as network filesystems.
/opt	can be used to store additional software for your system, which is not handled by the package manager.
/proc	a virtual filesystem that provides a mechanism for kernel to send information to processes.
/root	the superuser’s home directory, not in /home/ to allow for booting the system even if /home/ is not available.
/run	a tmpfs (temporary file system) available early in the boot process where ephemeral run-time data is stored. Files under this directory are removed or truncated at the beginning of the boot process. (It deprecates various legacy locations such as /var/run, /var/lock, /lib/init/rw in otherwise non-ephemeral directory trees as well as /dev/.* and /dev/shm which are not device files.)
/sbin	contains important administrative commands that should generally only be employed by the superuser.
/srv	can contain data directories of services such as HTTP (/srv/www/) or FTP.
/sys	a virtual filesystem that can be accessed to set or obtain information about the kernel’s view of the system.
/tmp	a place for temporary files used by applications.
/usr	contains the majority of user utilities and applications, and partly replicates the root directory structure, containing for instance, among others, /usr/bin/ and /usr/lib.
/var	dedicated to variable data, such as logs, databases, websites, and temporary spool (e-mail etc.) files that persist from one boot to the next. A notable directory it contains is /var/log where system log files are kept.

As usual, I would focus on pivot points if there are such, if not then I will follow my approach of identifying whether the system is compromised or not.

Linux Incident Response Part 1

Linux Kernel Internals

Further reading:

Hristiyan Lazarov