The Linux
Filesystem Hierarchy
A thorough tour of the standard and operationally important Linux filesystem directories — what lives where, why it is there, and how it behaves at runtime. Covers the FHS core hierarchy plus Debian, systemd, procfs, sysfs, devtmpfs, and cgroup details that administrators encounter in practice.
What the Filesystem Hierarchy Standard Is
If you open a terminal on any Linux distribution and type ls /, you will
see roughly the same set of top-level directories regardless of whether you are on
Debian, Ubuntu, Fedora, Arch, or an Alpine container. This is not an accident. It is
the Filesystem Hierarchy Standard (FHS) — the published specification
that standardizes the placement and purpose of files and directories on Unix-like systems.
The current published release is FHS 3.0. It defines required and optional directories,
directory purposes, and placement rules for interoperable software and administration.
FHS is still version 3.0. The FHS 3.0 specification was released in
2015, and Debian Policy continues to target FHS 3.0 with explicit
Debian-specific exceptions and compatibility rules. Do not treat a website rebuild, mirror
timestamp, or local document review date as a new FHS technical release. The standard exists
for a deceptively simple reason: software can reliably locate executables, libraries, configuration, runtime state, and shared data regardless of the distribution.
When nginx needs to write a PID file, it should know to write to
/run. When a shell script needs grep, it should know to
look in /usr/bin. When a package manager installs configuration, it
goes to /etc. Without a shared convention, every piece of software would
need to know the layout of every other piece of software.
Before FHS, Linux distributions were less standardized. Linux distributions often differed in where they placed libraries, spool directories, documentation, and local software, making portable software harder to write. Programs would ship with hardcoded paths that only worked on the distribution they were built for. Work on FSSTND (the Filesystem Structure Standard) began in 1993, with its first release in 1994. The effort broadened beyond Linux and was renamed the Filesystem Hierarchy Standard (FHS) in the mid-1990s. Current Debian Policy requires package file placement to follow FHS 3.0, subject to documented Debian exceptions.
The standard distinguishes directories on two axes. The first is shareable vs.
unshareable — shareable data can be shared across machines (for example, the installed software hierarchy under /usr), whereas unshareable data is specific to one machine (for example
/etc/hostname). The second axis is static vs. variable —
static data is not expected to change during ordinary system operation and normally changes
through package deployment or administrative action; variable data changes during normal
operation (logs, PID files, mail spools).
| Shareable (across machines) | Unshareable (machine-specific) | |
|---|---|---|
| Static | /usr, /opt |
/etc, /boot |
| Variable | /var/mail, /var/spool/news(historical FHS example; modern systems rarely use Usenet) |
/run, /run/lockDebian: /var/run and /var/lock are compatibility symlinks |
Debian packages are expected to follow FHS 3.0, but Debian Policy documents specific
exceptions and compatibility rules. Important examples: /var/run must be a
symlink to /run, /var/lock must be a symlink to /run/lock,
/var/www is allowed as a common web root, and Debian packages must not install
actual payload files into /usr/local. Debian also uses multiarch library directories
such as /usr/lib/x86_64-linux-gnu.
This distinction matters for NFS and network storage: in a suitably
controlled fleet, /usr can be mounted read-only and shared because it is
designed as static, shareable data. This was a major FHS design goal, although most
modern deployments distribute matching package or image contents to each host instead.
Much of /var is variable and host-specific, but FHS also defines shareable
exceptions such as /var/mail. /etc is host-specific and normally
local to each machine.
Why the Hierarchy Is Designed This Way
Understanding why the hierarchy exists as it does makes it far easier to memorize and navigate. There are a handful of organizing principles that explain most of the seemingly arbitrary layout decisions.
Principle 1: Root Filesystem Must Be Mountable for Emergency Recovery
The root filesystem (/) is kept deliberately lean. It contains only what
is necessary to boot the system and reach a minimal operational state for recovery.
This is why /bin, /sbin, and /lib exist as
separate directories from /usr/bin and /usr/lib — historically
they were on the root partition while /usr could be on a separate, potentially
network-mounted partition. The root had to be self-sufficient without /usr.
On current Debian releases, /bin, /sbin, and /lib*
at the root are compatibility symlinks into /usr — for example
/bin points to /usr/bin. This merged-/usr layout removes the old
practical split between the root filesystem and /usr for package payloads, while
keeping the historical paths available for scripts and admin muscle memory. Treat the
/usr locations as canonical and the root-level paths as aliases.
Principle 2: Separation of Configuration from Programs from Data
The three fundamental types of files a running service needs are kept in completely separate trees:
- Programs
/usr/bin,/usr/sbin,/usr/lib— distribution software payload. Mostly static between package or image updates; suitable for checksumming and, in supported designs, read-only or shared deployment. - Configuration
/etc— durable system and host configuration. It is managed by administrators, configuration management, and package-maintenance logic rather than ordinary service runtime. - Variable and Runtime Data
/var,/run,/tmp— persistent variable state, volatile runtime state, logs, spools, PID files, caches, and temporary files. Their retention semantics differ.
This separation enables powerful operational patterns. You can back up /etc
and restore a machine's configuration without touching programs. You can reclaim expendable cache data under
/var/cache using application-supported cleanup procedures. You can mount /usr
read-only as a security measure. You can use configuration management (Ansible, Puppet)
to ensure /etc always reflects a desired state.
Principle 3: Kernel-Provided Filesystems Are Mounted Under /
Linux's "everything is a file" philosophy extends to kernel internals. The kernel exposes its state and device interfaces as filesystems:
- /procProcess and kernel information. A virtual filesystem (
procfs) whose entries are generated by the kernel on access; it has no persistent disk backing. - /sysHardware topology and driver parameters. A virtual filesystem (
sysfs) reflecting the kernel's device model; it has no persistent disk backing. - /devDevice nodes and pseudo-devices. The kernel's
devtmpfssupplies the basic device nodes;udevapplies policy such as names, permissions, ownership, and persistent symlinks. No ordinary on-disk directory contents are used while devtmpfs is mounted. - /runRuntime data for running processes. A
tmpfsmounted at boot, cleared on reboot. Replaces the old/var/run.
Principle 4: /usr Is a Secondary Hierarchy
/usr replicates the structure of the root filesystem:
/usr/bin, /usr/sbin, /usr/lib, /usr/share.
Think of it as "the installed software tree." Most static package payloads live under
/usr, while host configuration belongs in /etc and persistent
variable state belongs in /var. In principle, package-managed contents under
/usr can be reconstructed from packages without replacing configuration or
application data.
The Core Directory Tree at a Glance
This annotated tree shows every significant top-level directory. Symlinks resulting from the usr-merge are marked in amber. Virtual (kernel-provided) filesystems are marked in purple. Directories covered in depth below are marked in green.
├── bin → usr/bin (symlink, usr-merge)
├── sbin → usr/sbin (symlink, usr-merge)
├── lib → usr/lib (symlink, usr-merge)
├── lib64 → usr/lib64 on amd64 usr-merged systems; used for the ABI dynamic linker, not as Debian's general 64-bit library directory
├── boot/ — kernel images, initrd, GRUB
├── lost+found/ — ext2/3/4 fsck recovery directory, if root uses ext-family filesystem
├── dev/ — device files (udev, devtmpfs)
├── etc/ — system-wide configuration files
│ ├── apt/ — APT configuration and sources
│ ├── systemd/ — systemd unit overrides
│ ├── ssh/ — SSH server and client config
│ └── ...
├── home/ — user home directories
│ └── username/
├── media/ — removable media mount points
├── mnt/ — temporary manual mount points
├── opt/ — optional/third-party software packages
├── snap/ — snap mount points, only if snapd is installed
├── proc/ — process and kernel info (procfs, virtual)
├── root/ — root user's home directory
├── run/ — runtime data (tmpfs, cleared on reboot)
│ ├── *.pid — daemon PID files
│ └── systemd/
├── srv/ — data served by services (web, FTP)
├── sys/ — kernel device model (sysfs, virtual)
├── tmp/ — temporary files; tmpfs and cleared at reboot by default on Debian 13
├── usr/ — installed software hierarchy (mostly static; may be mounted read-only)
│ ├── bin/ — general user commands and many system utilities
│ ├── sbin/ — system admin commands
│ ├── lib/ — shared libraries, kernel modules
│ ├── lib64/ — ABI loader compatibility path on Debian; normal libraries use multiarch directories
│ ├── include/ — C/C++ header files
│ ├── local/ — locally compiled software (not packaged)
│ │ ├── bin/ — local binaries
│ │ ├── lib/ — local libraries
│ │ └── share/ — local shared data
│ ├── share/ — architecture-independent data
│ │ ├── doc/ — documentation per package
│ │ ├── man/ — man page sources
│ │ └── locale/ — internationalization data
│ └── src/ — kernel source (if installed)
└── var/ — variable data (changes during operation)
├── cache/ — application cache data
├── lib/ — persistent application state data
├── log/ — log files
├── mail/ — user mail spools
├── spool/ — print/mail/cron spool queues
└── tmp/ — temporary files intended to survive reboot; subject to age cleanup
| Directory | Type | Purpose in one sentence | Changed by |
|---|---|---|---|
| / | real | Mountpoint for root filesystem; everything hangs off here | Installer, admin |
| /bin → /usr/bin | symlink | Essential user commands available to all users | Package manager |
| /boot | real | Bootloader files, kernel images, initrd | Package manager, admin |
| /dev | virtual | Device nodes for hardware and pseudo-devices | Kernel/devtmpfs and udev |
| /etc | real | Host-specific and system-wide configuration | Admin, packages, config management |
| /home | real | User home directories, personal files and settings | Users |
| /lib → /usr/lib | symlink | Essential shared libraries needed by /bin and /sbin | Package manager |
| /media | real | Automount points for removable media (USB, CD) | udisks/udev |
| /mnt | real | Temporary mount point for manual mounts by admin | Admin |
| /opt | real | Static payload for add-on software packages | Admin, vendor installer, or package manager |
| /proc | virtual | Process and kernel state as a virtual filesystem | Kernel; selected interfaces are writable by authorized users |
| /root | real | Home directory for the root superuser | root user |
| /run | virtual | Runtime variable data; cleared on each boot (tmpfs) | Daemons, systemd |
| /sbin → /usr/sbin | symlink | Essential system administration commands | Package manager |
| /srv | real | Data served by system services (web roots, FTP) | Admin |
| /sys | virtual | Kernel device model and driver parameters | Kernel, udev |
| /tmp | runtime | Temporary files; tmpfs and cleared at reboot by default on Debian 13 | Any process, subject to permissions |
| /usr | real | Mostly static, shareable installed-software hierarchy | Package manager, admin for /usr/local |
| /var | real | Variable data: logs, spools, databases, caches | Daemons, package manager |
/ — The Root of Everything
The root directory is not just a convention — it is the mount point for the
system's root filesystem. On systems using an initramfs, the kernel first starts
early userspace from that temporary root; early userspace then mounts the real root
filesystem and switches to it. Other filesystems are subsequently grafted onto this tree. This is
why Linux paths start with /: they are absolute paths from the root
of this single unified namespace.
Unlike Windows (where each drive has its own root like C:\),
Linux has exactly one root. A USB drive does not become D:\;
it becomes /media/username/usb-label — a directory within the single tree.
A network share does not get a drive letter; it is mounted at /mnt/nfs-share.
This unified namespace simplifies everything: a path like
/var/log/nginx/error.log is unambiguous regardless of whether
/var is on a separate partition, a network filesystem, or the root partition.
FHS requires the root filesystem to be as small as possible. The real root filesystem must become available during early boot; if it is corrupt or
unavailable, normal boot cannot continue. Keeping boot-critical storage controlled reduces
recovery risk. Most distribution software and variable system data live under
/usr and /var, while user data commonly lives under
/home; any of these may be separate filesystems when the boot design supports it.
What Lives Directly in /
FHS defines a small set of standard top-level directories or symlinks and says
applications must not invent new top-level entries. Regular files directly under
/ are therefore unusual package layout, although a local administrator may
deliberately create an exceptional file such as a root-level swap file. On modern Debian
with usr-merge, several historical directories are symlinks:
lost+found is not a general-purpose storage directory. It is created at the root
of ext2/ext3/ext4 filesystems so fsck has a place to reconnect orphaned inodes
after filesystem damage or an unclean recovery. You may see /lost+found if the root
filesystem is ext-family, and you may also see /home/lost+found,
/var/lost+found, or similar when those mount points are separate ext-family filesystems.
# Show everything in /, with types
ls -la /
# See which directories are separate mountpoints
findmnt --tree
# How much space each top-level directory uses
du -sh --exclude=proc --exclude=sys --exclude=dev /* 2>/dev/null | sort -h/bin, /sbin, /lib — The Bootstrap Binaries
Historically these were distinct directories holding only the commands and libraries
needed for single-user mode recovery — when /usr was not yet
mounted. On current Debian releases they are compatibility symlinks to their /usr counterparts.
However, understanding the original intent explains what should be in
/usr/bin vs /usr/local/bin vs /opt.
What Belongs in /usr/bin vs /usr/sbin
| Directory | Intended for | Examples | Who can run |
|---|---|---|---|
| /usr/bin | User-facing commands; anyone can run | bash, ls, grep, find, cat, git, python3, curl, vim | All users |
| /usr/sbin | Commands intended primarily for system administration | fdisk, useradd, sshd, nginx, iptables | Usually executable by all users; privileged operations still require authorization |
| /usr/local/bin | Locally compiled or manually installed binaries | site-wide custom scripts, manually installed tools, locally built binaries | All users |
| /usr/local/sbin | Locally compiled admin tools | custom maintenance scripts, local backup tools | Primarily administrators; directory placement does not itself enforce privilege |
The historical bin/sbin distinction is about a command's intended administrative role,
not an access-control boundary. Debian 13 installs ss in
/usr/bin; the iproute2 package provides ip at
both /usr/bin/ip and /usr/sbin/ip for compatibility. Many
inspection operations are useful without root, while state-changing operations remain
restricted by normal Unix permissions and Linux capabilities.
/usr/lib — Shared Libraries and More
/usr/lib contains several distinct types of files that are often confused:
- .so filesShared object libraries — compiled code shared between multiple programs.
libc.so.6,libssl.so.3. Named with versioned suffixes. When a program links againstlibssl, the linker resolves it to the actual file at load time. - /usr/lib/systemd/systemd unit files — the canonical location for package-provided service units. When a package installs a service, it goes here. Do not edit these files directly — use drop-in overrides in
/etc/systemd/system/. - /usr/lib/modules/Kernel modules — loadable kernel modules (
.kofiles) organized by kernel version.modprobesearches here. Every installed kernel version has its own subdirectory here. - /usr/lib/firmware/Firmware blobs — binary firmware files loaded into hardware devices (Wi-Fi cards, GPU firmware, etc.). Separate from kernel modules.
- /usr/lib/python3/dist-packages/Debian-packaged Python libraries — Python modules installed by APT live here. Avoid using global
pip --break-system-packageson production systems; use virtual environments, pipx, or Debian packages instead. - /usr/lib/x86_64-linux-gnu/Multi-arch libraries — on multi-arch systems, libraries for each architecture are in architecture-specific subdirectories. This is Debian's multi-arch implementation.
# Find all shared libraries for a package
dpkg -L libc6 | grep '\.so'
# What shared libraries does nginx use at runtime?
ldd /usr/sbin/nginx
# List all kernel modules for the running kernel
ls /usr/lib/modules/$(uname -r)/kernel/
# Find where a shared library lives
ldconfig -p | grep libssl
# Show multi-arch library directories
dpkg-architecture -qDEB_HOST_MULTIARCH # e.g. x86_64-linux-gnu
ls /usr/lib/x86_64-linux-gnu//boot — What the Bootloader Needs
/boot contains the main boot artifacts used after firmware hands control
to the boot chain: bootloader configuration or support files, kernel images, and
initramfs images. Exact contents depend on the bootloader and layout. On UEFI systems, /boot often contains /boot/efi as a
submount point for the EFI System Partition. /boot is commonly left
unencrypted for bootloader compatibility, but it is not inherently required to be
unencrypted; supported encrypted-boot designs depend on the bootloader, firmware,
key-management method, and chosen filesystem.
update-grub.grubx64.efi and shimx64.efi. Present only when an ESP is mounted there.# List all installed kernels
ls /boot/vmlinuz*
# Which kernel are we running right now?
uname -r
# Check a config option in the running kernel
grep CONFIG_BTRFS_FS /boot/config-$(uname -r)
# Rebuild initramfs after changes that must be available in early boot
update-initramfs -u -k all
# Rebuild GRUB config (e.g. after installing a new kernel)
update-grub
# Show UEFI boot entries referencing /boot/efi
efibootmgr -vDebian normally leaves older installed kernel packages available until they are explicitly
removed, so GRUB may offer a previous kernel as a recovery option. The
linux-image-amd64 metapackage tracks the current Debian amd64 kernel ABI.
After upgrades, apt autoremove may propose obsolete kernel packages for
removal according to APT's dependency and protection rules; review the proposed list
before accepting it. /etc/kernel-img.conf does not provide a general
“number of kernels to retain” setting.
/home and /root — User Data
/home contains a subdirectory for each non-system user:
/home/alice, /home/bob, etc. Each home directory is owned by that user. On Debian 13,
adduser defaults new non-system user homes to mode 0700,
but administrators can change this with DIR_MODE in
/etc/adduser.conf or use other provisioning tools. The root user is special — root's home
directory is /root, not /home/root, because /root
must be available even if /home is on a separate, unmounted partition.
What Lives in a User's Home Directory
Applications follow a convention (formalized in the XDG Base Directory Specification) for where to store user-specific data:
| Path | XDG Variable | Purpose | Examples |
|---|---|---|---|
| ~/.config/ | $XDG_CONFIG_HOME | User-specific configuration | ~/.config/git/config, ~/.config/nvim/ |
| ~/.local/share/ | $XDG_DATA_HOME | User-specific data files | Desktop entries, app state, fonts |
| ~/.local/bin/ | — | User-installed executables | pip --user or pipx command shims, user-local scripts |
| ~/.cache/ | $XDG_CACHE_HOME | Non-essential cached data | Browser cache, apt build cache, thumbnails |
| ~/.ssh/ | — | SSH keys and config | id_rsa, authorized_keys, config |
| ~/.bashrc, ~/.bash_profile | — | Shell initialization files | Aliases, PATH additions, env vars |
Many older applications still use dotfiles directly in home
(~/.vimrc, ~/.tmux.conf, ~/.gitconfig).
Modern applications should use XDG directories, but adoption is uneven. The
$HOME directory of a long-lived Unix user is often an accumulation of
both conventions. Tools like xdg-user-dirs manage the standard user
directories (Documents, Downloads, Pictures, etc.) and create them if absent.
# List top-level dotfiles and dot-directories in your home
find "$HOME" -mindepth 1 -maxdepth 1 -name '.*' -printf '%f\n' | sort
# Show XDG base directories
echo $XDG_CONFIG_HOME # defaults to ~/.config if unset
echo $XDG_DATA_HOME # defaults to ~/.local/share
echo $XDG_CACHE_HOME # defaults to ~/.cache
# How much space is a user's home taking?
du -sh ~/
# Find large files in home directory, sorted by byte size
find "$HOME" -type f -size +100M -printf '%s\t%p\n' 2>/dev/null | sort -n | numfmt --field=1 --to=iec/tmp and /run — Ephemeral Filesystems
/tmp — Temporary Files
/tmp is the dumping ground for short-lived temporary files. Any user or process can
write here, but the directory is protected by the sticky bit. On new Debian 13 (Trixie)
installations, /tmp is mounted as a tmpfs by default and is therefore
RAM-backed and cleared at reboot. This applies to upgraded systems too, starting at the first reboot after the upgrade — pre-existing files in /tmp are not deleted but become hidden beneath the new mount. Systems that already define /tmp in /etc/fstab are unaffected; systemctl mask tmp.mount restores a plain directory.
- !World-writable with sticky bit:
/tmphas permissions1777. The sticky bit means that while anyone can create files, an entry can normally be removed or renamed only by its owner, the directory owner, or a privileged process. Without the sticky bit, any user could delete any other user's temp files. - !Never store sensitive data in /tmp without setting restrictive permissions: since all users can list the directory, filenames are visible to anyone. Use
mktempto create uniquely named files, or use/tmpsubdirectories owned exclusively by your process. - →tmpfs vs disk: A tmpfs
/tmpconsumes memory/swap and is capped by systemd policy, commonly up to half of physical RAM by default. Large scratch workloads should use a disk-backed location such as/var/tmp, a service-specific path under/var/cache, or an explicitly configured mount. Override withsystemctl edit tmp.mount, a dedicated/etc/fstabentry, or masktmp.mountto disable the tmpfs mount.
/var/tmp — Longer-Lived Temporary Storage
/var/tmp is for temporary files that should survive a reboot. It is the better
location for large scratch files, resumable jobs, and tools that need temporary state across
restarts. On new Debian 13 installations, systemd-tmpfiles deletes files in
/tmp after 10 days since last use and files in /var/tmp
after 30 days. Because /tmp is a tmpfs by default, its contents also
disappear at reboot. Systems upgraded from Debian 12 receive an opt-out file at
/etc/tmpfiles.d/tmp.conf that preserves the old no-age-cleanup behavior
until the administrator removes or edits it.
/run — Runtime State
/run was introduced (replacing the older /var/run) to address
a bootstrapping problem: /var might be on a separate partition that isn't
mounted during early boot, but daemons need to write PID files and Unix sockets before
/var is available. /run is always a tmpfs mounted
early in boot. On Debian, /var/run and /var/lock are compatibility
symlinks to /run and /run/lock.
docker group normally permits controlling this socket and is effectively root-equivalent on the host. Prefer rootless Docker or another rootless container engine where that security boundary matters./run/NetworkManager/, while systemd-networkd uses paths such as /run/systemd/netif/.# See all PID files for running services
ls /run/*.pid 2>/dev/null
# Check SSH daemon's PID file
cat /run/sshd.pid
ps -p $(cat /run/sshd.pid)
# Your user's runtime directory (Wayland sockets, etc.)
ls /run/user/$(id -u)/
# Check tmpfs mounts
mount -t tmpfs/media, /mnt, /srv, /opt
/media — Removable Media
/media is where udisks2 and desktop environments mount
removable storage. When you plug in a USB drive, it appears at
/media/username/label-or-uuid. On headless servers without a desktop,
removable media is rarely relevant — use /mnt instead.
/mnt — Administrator Mounts
/mnt is the traditional mount point for temporary mounts by the
system administrator. It has no further structure imposed by FHS. Common usage:
mounting a USB drive for a rescue operation (mount /dev/sdb1 /mnt),
mounting an NFS share temporarily, or mounting an ISO image. Because FHS reserves /mnt for temporary administrator mounts, prefer a
service- or data-specific mountpoint for permanent filesystems. Local policy may still use
named subdirectories such as /mnt/data, but that is an administrative convention,
not an FHS-defined permanent layout.
/srv — Service Data
/srv is intended for data served by the system's services.
The FHS says: "data for services provided by this system." A web server serving
a site could serve from /srv/www/example.com/. An FTP server could
serve from /srv/ftp/. In practice, many Debian packages use
/var/www for web content by default (Nginx, Apache), and
/srv is often empty on fresh installs. However, using /srv
is cleaner when you control the configuration — it keeps service data separate from
variable runtime data in /var.
/opt — Optional Software
/opt is intended for add-on application packages, commonly from third-party
vendors. Such software may be installed manually or by a vendor-supplied Debian package;
use of /opt does not by itself mean the files are outside dpkg/APT management. The key characteristic of software in /opt is that its static package
payload is grouped under a registered subdirectory such as
/opt/vendor/package/. FHS does not require every file to remain
inside that directory: host-specific configuration belongs under
/etc/opt/<subdir>/, and variable data belongs under
/var/opt/<subdir>/. Small integration files may also appear in
standard locations when required.
- Google Chrome installs to /opt/google/chrome/
- Zoom installs to /opt/zoom/
- Microsoft Edge installs to /opt/microsoft/msedge/
- JetBrains tarballs are unpacked by hand, conventionally to /opt/<product>-<version>/
- Static payload grouped below /opt
- Config may use /etc/opt; variable data may use /var/opt
- Binary → /usr/bin/
- Config → /etc/
- Data → /usr/share/
- Docs → /usr/share/doc/
- Uninstall requires package manager
Not every third-party runtime uses /opt. Java is the classic
counter-example: Debian's openjdk-* packages and Oracle's own
.deb builds install under /usr/lib/jvm/<jdk>/, registered
with the alternatives system, while Oracle's RPM builds use /usr/java/.
Treat /opt/<vendor>/ as a convention that vendors may follow, not a rule
the FHS enforces on them.
Snap and Flatpak Paths on Debian Desktops
/snap, /var/lib/snapd/, /var/lib/flatpak/, and
~/.local/share/flatpak/ are not core FHS directories and are
not normally present on a minimal Debian server. They appear when an administrator installs
Snap or Flatpak support, usually on desktop systems. Snap commonly exposes mounted revisions
below /snap/<name>/<revision>, while system-wide Flatpak deployments live
under /var/lib/flatpak/ and per-user deployments under
~/.local/share/flatpak/. A root-level /flatpak directory is not the
usual Debian convention unless a local administrator deliberately creates one.
# List what's installed in /opt
ls /opt/
# Mount an NFS share temporarily
mount -t nfs server:/export/data /mnt
# Check if /media has any mounted removable drives
ls /media/
findmnt -R /media/etc — The Configuration Universe
/etc is the system's configuration brain. The name stands for
et cetera in the original Unix (a catch-all), though it is now
retroactively expanded as "Editable Text Configuration." Everything here is
primarily text files intended to be read and modified by system administrators.
Compiled binaries do not belong here, but executable text scripts and drop-in snippets do: for
example, /etc/network/if-up.d/, /etc/cron.*, and maintainer/admin hooks.
The fundamental contract of /etc is this: programs read durable
configuration from /etc; volatile state belongs elsewhere. Daemons should not store
runtime state in /etc; they use /run for PID files and sockets and
/var for persistent state. Admin tools, package maintainer scripts, and configuration
management systems may intentionally modify /etc because that is how desired system
configuration is recorded.
The Most Critical Files in /etc
x and the hash is stored in /etc/shadow. The file is world-readable, so password hashes should not be stored here.shadow group. Contains password fields that commonly use a format such as $algorithm$salt$hash. This is the file attackers want in a breach.hostnamectl set-hostname newname.localhost; a local hostname mapping is
common but may instead be supplied by DNS or an NSS module such as nss-myhostname.
Local entries can override DNS according to /etc/nsswitch.conf.ls -l /etc/resolv.conf before editing because manual changes may be overwritten.pam_env module for PAM-managed login sessions. It is not a universal source for every process or service. Syntax is simple assignments rather than general shell code.The /etc Subdirectory Structure
| Directory | Contains | Key files |
|---|---|---|
| /etc/apt/ | APT package manager configuration | sources.list, sources.list.d/, apt.conf.d/, preferences.d/, keyrings/ — put per-repository keys in /etc/apt/keyrings/ and point at them with Signed-By: in a deb822 .sources file; the older trusted.gpg.d/ still works but trusts each key for every repository |
| /etc/systemd/ | systemd configuration and unit overrides | system.conf, journald.conf, system/ (drop-in units), network/ (networkd) |
| /etc/ssh/ | SSH server and client configuration | sshd_config, sshd_config.d/, ssh_config, ssh_host_*_key |
| /etc/nginx/ or /etc/apache2/ | Web server configuration | nginx.conf, sites-available/, sites-enabled/ (symlinks), conf.d/ |
| /etc/cron.d/ | System cron job definitions | One file per package, same format as crontab but with username field |
| /etc/cron.daily/ | Executable scripts intended for daily execution through cron/anacron; some packages use systemd timers instead | Examples vary by installation; names normally have no filename extension |
| /etc/pam.d/ | PAM authentication module stacks | sshd, login, sudo, common-auth, common-session |
| /etc/security/ | PAM security module configuration | limits.conf (ulimits), access.conf (login restrictions) |
| /etc/default/ | Default values for init scripts and system tools | grub, keyboard, locale, useradd, ntp |
| /etc/sysctl.d/ | Kernel parameter overrides (applied at boot) | Local *.conf snippets. On Debian 13, systemd-sysctl no longer reads /etc/sysctl.conf at boot; the vendor defaults now ship as /usr/lib/sysctl.d/50-default.conf from the linux-sysctl-defaults package (Recommended by systemd, so installed by default). Keep local overrides here, numbered 60–90 so they win. |
| /etc/modprobe.d/ | Kernel module load-time options | blacklist.conf, options for specific drivers |
| /etc/udev/rules.d/ | udev device naming rules | Persistent network device names, custom device permissions |
| /etc/logrotate.d/ | Log rotation rules per package | nginx, syslog, apt — define retention, compression, rotation frequency |
| /etc/profile.d/ | Shell environment snippets for all login users | Sourced by /etc/profile. Add PATH entries, env vars here. |
| /etc/network/ | Legacy ifupdown network configuration | interfaces — static IP config, bonding (ifupdown/legacy style) |
| /etc/NetworkManager/ | NetworkManager configuration | NetworkManager.conf, system-connections/ (per-connection profiles) |
The /etc/default/ Pattern
Debian packages may use /etc/default/packagename files to supply
package-specific defaults without editing an init script or generated configuration.
These files are often shell-style variable assignments, but they have effect only when
the relevant package, init script, or service unit explicitly reads them; systemd does
not automatically read every file in /etc/default.
# /etc/default/grub — controls GRUB generation
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_RECOVERY="true"
# After editing, always run:
update-grub
# /etc/default/keyboard — keyboard layout
XKBMODEL="pc105"
XKBLAYOUT="us"
XKBVARIANT=""
XKBOPTIONS=""
# Apply keyboard changes
dpkg-reconfigure keyboard-configurationThe Drop-in Override Pattern (.d/ directories)
Many /etc configurations use a drop-in directory pattern:
instead of one monolithic file, there is a main file and a .d/
directory whose contents are automatically included. The benefit: packages can add their
own configuration snippets without overwriting yours, and you can add your own without
touching the package-provided file.
# apt sources — main file AND a drop-in directory
cat /etc/apt/sources.list # main file
ls /etc/apt/sources.list.d/ # .sources (deb822, preferred) and legacy .list entries
# sshd — main config AND override directory (Bookworm+)
cat /etc/ssh/sshd_config
ls /etc/ssh/sshd_config.d/ # drop-in overrides
sudo /usr/sbin/sshd -T # merged result: first value seen wins
# systemd unit override for nginx (do not edit /usr/lib/systemd)
mkdir -p /etc/systemd/system/nginx.service.d/
cat > /etc/systemd/system/nginx.service.d/override.conf <<'EOF'
[Service]
LimitNOFILE=65536
EOF
systemctl daemon-reload && systemctl restart nginx
# Check merged effective unit config after overrides
systemctl cat nginxProtecting /etc with Version Control
Because /etc is machine-specific configuration, it is an excellent
candidate for version control. The tool etckeeper automatically
initializes a Git repository in /etc and commits changes whenever
packages are installed or upgraded:
apt install etckeeper
etckeeper init # initialize git repo in /etc
etckeeper commit "Initial /etc state"
git -C /etc log --oneline # see all changes over time
git -C /etc diff HEAD~1 # what changed in the last operation/var — Everything That Changes
/var contains data that changes continuously during the system's operation.
Unlike /etc (changes only when an admin acts intentionally), /var
is written to by daemons, package managers, and application processes constantly. On servers it is often useful to isolate selected high-growth subtrees such as
/var/log, database data, or container storage on separate filesystems.
This helps contain capacity failures, although a full /var can still break
services. PID files under the tmpfs-backed /run are not themselves stored
on the root disk.
/var/log — The System's Journal
/var/log is where traditional log files live. On minimal systemd-based Debian
installations, journald may be the primary log source; files such as syslog,
auth.log, and kern.log appear when rsyslog or another syslog daemon is
installed and configured. Understanding both models is essential for diagnosing problems:
journalctl.journalctl _COMM=sshd, journalctl -u ssh, or PAM-related journal entries.journalctl -k.dpkg.log.* files and APT history when answering “what changed?”Log Rotation with logrotate
Log files would grow unbounded without rotation. On systemd-based Debian,
logrotate is normally invoked by logrotate.timer; cron-based
installations may invoke it daily instead. It reads /etc/logrotate.conf
and package or local rules under /etc/logrotate.d/.
# Example: /etc/logrotate.d/nginx
/var/log/nginx/*.log {
daily
missingok
rotate 52 # with daily rotation, keep up to 52 old daily files
compress # gzip old logs
delaycompress # keep yesterday's log uncompressed
notifempty
create 0640 www-data adm
sharedscripts
postrotate
[ -f /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
endscript
}
# Test logrotate configuration without actually rotating
logrotate --debug --force /etc/logrotate.d/nginx
# View status of last rotation
cat /var/lib/logrotate/status/var/lib — Persistent Application State
/var/lib is where applications store their persistent state data —
things that must survive reboots but are not configuration files that humans edit.
This is a critical distinction:
- /var/lib/dpkg/The dpkg database — the authoritative record of every installed package, its files, version, and installation status. If this is corrupted, the package manager cannot function. The most critical subtree:
/var/lib/dpkg/info/contains per-package file lists, pre/postinstall scripts, and md5sums. - /var/lib/apt/APT's cache of package metadata — the lists of available packages downloaded from repositories.
apt updaterefreshes this.apt-get cleanclears downloaded .deb files from/var/cache/apt/archives/but not this. - /var/lib/postgresql/PostgreSQL cluster data — Debian-created clusters normally live below this directory. Administrators may relocate
data_directory, and PostgreSQL tablespaces can place selected objects elsewhere, so confirm with the running server configuration before treating this path as complete. - /var/lib/mysql/MySQL/MariaDB database files — tablespaces, redo/undo data, internal metadata, and other engine-managed files. Exact formats vary substantially by server and version; do not manipulate them as ordinary files while the database is running.
- /var/lib/docker/Docker's data root — all container layers (overlay filesystem), volumes, networks, and metadata for the default rootful Docker daemon. Can consume enormous amounts of space. Manageable with
docker system prune. Access to the rootful Docker daemon socket is effectively root-equivalent; rootless Docker stores state under the user's home/XDG data locations instead and reduces host-root exposure. - /var/lib/systemd/systemd persistent state — random seed, catalog/database state, and other systemd-managed persistent data. Persistent journal files, when enabled, live under
/var/log/journal/, not here.
/var/cache — Expendable Cache Data
/var/cache is intended for expendable data that applications can regenerate
or reacquire. This placement is a design contract, not permission to delete arbitrary
contents while applications are running.
That does not make an indiscriminate rm -rf /var/cache/* a safe live-system
procedure: stop the owning service or use its supported cleanup command, and confirm
that an application has not misused the directory for non-cache state.
| Directory | What it caches | How to clear | Typical size |
|---|---|---|---|
| /var/cache/apt/archives/ | Downloaded .deb package files | apt clean |
Can reach several GB after many installs |
| /var/cache/man/ | Compiled ("cat") man page cache | mandb rebuilds the index; cat pages regenerate on read |
Usually small — Debian does not pre-generate cat pages, so this is often near-empty and rarely exceeds a few MB |
| /var/cache/fontconfig/ | Font metadata index | fc-cache -f |
Small |
/var/spool — Job Queues
/var/spool holds data awaiting processing — jobs queued for later
execution or transmission. Key subdirectories:
- /var/spool/cron/crontabs/Per-user crontab files. Each file is named after the user it belongs to. Edited by
crontab -e, never directly. - /var/mail/Primary FHS location for user mailboxes in mbox format. One file per user.
/var/spool/mailmay exist as a compatibility symlink or legacy path. - /var/spool/cups/Print job queue. Jobs waiting to be printed, held jobs, and the print history.
# Find what is consuming space in /var
du -sh /var/*/ 2>/dev/null | sort -h
# Check the dpkg package database
ls /var/lib/dpkg/info/nginx* # see all dpkg files for nginx
cat /var/lib/dpkg/info/nginx.list # every file owned by the nginx package
# Clear apt package cache to free space
apt clean # removes downloaded .deb files from /var/cache/apt/archives
apt autoclean # removes only outdated .deb files
# Check journal disk usage
journalctl --disk-usage
journalctl --vacuum-size=500M # limit journal to 500 MB
journalctl --vacuum-time=30d # remove journal older than 30 days
# See docker space usage
docker system df
docker system prune -a # destructive: review unused objects first; does not remove volumes unless requested/usr — The Installed Software Universe
/usr is the largest directory on a typical Debian system and contains
virtually all distribution-installed software. Historically, /usr began as a
user hierarchy on early Unix systems; today it is best understood as the shareable, mostly
read-only software hierarchy. The popular expansion "Unix System Resources" is a useful
mnemonic, not the original meaning. On hardened systems /usr may be mounted
read-only; normal package operations are then performed through controlled remounts or image
updates.
/usr/share — Architecture-Independent Data
/usr/share contains the data portion of installed packages — things that
are the same regardless of CPU architecture (text, images, icons, translations, documentation):
| Path | Contents | Notes |
|---|---|---|
| /usr/share/doc/ | Package documentation, changelogs, copyright files | One subdirectory per package. copyright files required by Debian policy. |
| /usr/share/man/ | Man page sources (troff format) | Organized by section: man1/, man2/, ..., man8/. Compressed as .gz. |
| /usr/share/locale/ | Compiled gettext translation catalogs (.mo files) | Subdirectory per locale, then LC_MESSAGES/ per package. |
| /usr/share/fonts/ | System-wide fonts | Subdirectories by format: truetype/, opentype/, X11/. User fonts go in ~/.local/share/fonts/. |
| /usr/share/applications/ | Desktop entry files (.desktop) | Defines application launchers for desktop environments. Name, icon, exec command, categories. |
| /usr/share/icons/ | Icon themes | hicolor/ (all lowercase) is the fallback theme. Theme subdirectories contain size-specific PNGs and SVGs. |
| /usr/share/bash-completion/ | Architecture-independent shell completion data | Loaded by the bash-completion framework to provide programmable tab completion for commands. This directory is unrelated to systemd unit files. |
| /usr/share/zoneinfo/ | Timezone data files | Binary format timezone rules for every region. /etc/localtime is a symlink into here. |
| /usr/share/ca-certificates/ | CA certificate trust store source | Package-provided certificate sources live here; selection is controlled by /etc/ca-certificates.conf. Local CA certificates normally go under /usr/local/share/ca-certificates/. |
| /usr/share/perl5/ or /usr/share/python3/ | Architecture-independent language library files | Architecture-independent interpreter data may live here. Debian-packaged Python modules normally use /usr/lib/python3/dist-packages/; compiled extensions use architecture-specific library directories. |
Compressed Documentation, less, and lesspipe
Debian stores many package documents and man-page sources compressed under
/usr/share/doc/ and /usr/share/man/. The pager less
can be integrated with helper scripts such as lesspipe through the
LESSOPEN and LESSCLOSE environment variables. When configured, this
allows commands such as less /usr/share/doc/package/changelog.gz or
less /var/log/syslog.1.gz to transparently display decompressed text without
manually running zcat. For administration, this explains why the same file may
appear compressed on disk but readable directly through the pager.
# Inspect how less is configured in your shell
echo "$LESSOPEN"
echo "$LESSCLOSE"
# Read compressed package docs without manually decompressing
less /usr/share/doc/bash/changelog.Debian.gz
less /var/log/syslog.1.gz/usr/local — Your Own Software
/usr/local mirrors the /usr structure but is reserved for
software you install yourself, outside of Debian package payloads. Debian
packages must not place regular files there, though maintainer scripts may create empty
administrative directories below standard /usr/local subdirectories. It is the
right place for:
- →Compiled-from-source programs whose build system uses the conventional
/usr/localprefix - →Go binaries deliberately deployed site-wide to
/usr/local/bin; an ordinary user-levelgo installnormally uses$GOBINor$GOPATH/bin - →Custom shell scripts that should be available to all users
- →Locally managed language runtimes or tools when intentionally installed system-wide; for Python applications and CLI tools, prefer virtual environments or
pipxrather than global pip installation
/usr/local/bin appears before /usr/bin in the
default PATH on Debian. This means a binary you install in /usr/local/bin
will shadow a package-provided version in /usr/bin of the same name. This
is intentional — your locally compiled version takes precedence. Be careful: if you install
a custom python3 to /usr/local/bin/, it can shadow Debian's
expected interpreter path for users and scripts that rely on PATH. Prefer virtual
environments or pipx for Python command-line tools.
/usr/src — Kernel Source
/usr/src holds source code for the system. On a typical Debian server it
often contains kernel header trees (linux-headers-*) and may also contain
source trees installed by DKMS or local administrators. Kernel headers provide the
interfaces needed to build external modules such as DKMS-managed drivers. If you install the versioned full kernel source package (for Debian 13, for example
linux-source-6.12), it installs a compressed archive such as
/usr/src/linux-source-6.12.tar.xz; extract it yourself when needed.
# How much space does /usr occupy?
du -sh /usr
# Where does a specific command live?
which nginx # /usr/sbin/nginx
type -a ls # shows all instances including shell builtins
whereis grep # binary, man page, and source locations
# Which package installed a file?
dpkg -S /usr/bin/grep # → grep: /usr/bin/grep
# List all files installed by a package
dpkg -L nginx | head -30
# Find all .desktop launcher files
ls /usr/share/applications/
# Check timezone data
ls -la /etc/localtime # symlink → /usr/share/zoneinfo/...
timedatectl # current timezone and NTP status/proc — The Kernel's Process and State Interface
/proc is one of the most important directories in the Linux filesystem —
and it contains zero actual files on disk. Every "file" you read from
/proc is generated on-the-fly by the kernel in response to your read
system call. The kernel uses this virtual filesystem to expose its internal state to
userspace through a file-oriented interface. Access is still constrained by permissions,
capabilities, namespaces, and procfs mount options.
Many process and memory tools — including ps, top,
free, and much of lsof — read extensively from
/proc. Networking tools are more mixed: modern ss primarily
uses netlink interfaces, while compatibility and fallback paths may read procfs.
Understanding /proc lets you inspect a large part of the same raw process
and kernel state directly.
The /proc/PID Namespace — One Directory Per Process
The most distinctive feature of /proc: for every running process, there
is a directory named after its PID containing detailed information about that specific
process:
├── cmdline — full command line (null-byte separated arguments)
├── comm — short command name (15 chars max)
├── environ — environment variables (null-byte separated)
├── exe — symlink to the executable binary on disk
├── cwd — symlink to current working directory
├── root — symlink to process root (/ unless chrooted)
├── status — human-readable state: Name, Pid, VmRSS, threads
├── stat — machine-readable state: used by top/ps internally
├── statm — memory use in pages: size, resident, shared
├── maps — all memory-mapped regions (libraries, heap, stack)
├── smaps — detailed per-mapping memory accounting
├── limits — current ulimits: soft and hard limits
├── oom_score — OOM killer score; higher = killed first
├── io — I/O accounting: bytes read/written, syscall counts
├── fd/ — directory of symlinks: one per open file descriptor
├── fdinfo/ — additional info for each fd: flags, position
├── net/ — per-process network namespace view
├── ns/ — namespace entries: mnt, net, pid, uts, ipc, user
└── task/ — one subdirectory per thread (TID)
/proc/self — The Current Process Shortcut
/proc/self is a symbolic link that always resolves to the /proc/<PID>
directory of the process that is looking at it. This is why /dev/stdin,
/dev/stdout, and /dev/stderr can point through
/proc/self/fd/: the target adapts to the calling process. It is heavily used by
shells, runtimes, debuggers, container tools, and scripts that need to refer to their own
file descriptors or namespace context without knowing their numeric PID in advance.
# Read the command line of PID 1 (systemd)
cat /proc/1/cmdline | tr '\0' ' ' # replace nulls with spaces
# What executable is running as PID 1234?
readlink /proc/1234/exe
# What files does process 1234 have open?
ls -la /proc/1234/fd
# See environment variables of a running process
cat /proc/1234/environ | tr '\0' '\n'
# Memory usage of a specific process
cat /proc/1234/status | grep -E 'VmRSS|VmSize|Threads'
# Is the process in a namespace (e.g. container)?
ls -la /proc/1234/ns/ # compare inode numbers between processes
ls -lai /proc/1/ns # PID 1 (host) namespaces
ls -lai /proc/1234/ns # different inodes = different namespaces = container
# /proc/self always means "this process"
readlink /proc/self
ls -l /proc/self/fdFiles such as /proc/<PID>/cmdline, /proc/<PID>/environ,
and /proc/<PID>/fd/ can reveal arguments, environment variables, open files,
and sometimes secrets. On multi-user systems, consider a restrictive hidepid=
mount option for /proc and avoid placing secrets in command-line arguments or
environment variables.
Key System-Wide /proc Files
| File | What it contains | Used by |
|---|---|---|
| /proc/cpuinfo | CPU model, cores, flags, MHz, cache sizes. One block per logical CPU. | lscpu, nproc, htop |
| /proc/meminfo | Complete memory stats: MemTotal, MemFree, MemAvailable, Buffers, Cached, SwapTotal, HugePages. MemAvailable is the number to watch, not MemFree. | free, vmstat, htop |
| /proc/loadavg | 1, 5, 15-minute load averages + running/total threads + last PID created. | uptime, top |
| /proc/uptime | Two numbers: seconds since boot, seconds system was idle. Divide the second by nproc to get average idle. | uptime command |
| /proc/mounts | Mounted filesystems visible in the reader's current mount namespace, with options. It is live state; /etc/fstab is persistent configuration and may not match it. | mount, findmnt |
| /proc/net/tcp | TCP sockets visible in the current network namespace, encoded in hex: local address:port, remote address:port, state, and inode. /proc/net/tcp6 and UDP counterparts also exist. | Direct diagnostics; legacy or fallback tooling |
| /proc/net/if_inet6 | IPv6 interface addresses. /proc/net/dev shows per-interface TX/RX counters. | Direct diagnostics; legacy tooling |
| /proc/sys/ | Kernel tunables and state. Many entries are writable and change live kernel behavior, but not all /proc/sys files are writable. sysctl reads and writes the writable ones; persistent settings live in /etc/sysctl.d/. | sysctl |
| /proc/interrupts | Interrupt counts per CPU per IRQ. Shows which hardware is generating interrupts and which CPUs are handling them. | irqtop, perf |
| /proc/cmdline | The kernel command line as passed by GRUB. Shows boot parameters, root device, and any custom parameters you added in /etc/default/grub. | systemd, custom scripts |
| /proc/version | Kernel version and build information, including compiler details. It overlaps with but is not identical to uname -a. | Build identification and diagnostics |
| /proc/filesystems | All filesystems currently supported by the running kernel (both compiled-in and loaded as modules). | mount |
| /proc/modules | Currently loaded kernel modules: name, memory size, use count, dependencies. lsmod formats this table. | lsmod |
| /proc/buddyinfo | Memory fragmentation data — how many free memory blocks of each power-of-two size exist per NUMA zone. Used for advanced memory diagnostics. | memory analysis tools |
/proc/sys — Live Kernel Tuning
/proc/sys is unique because it exposes live kernel tunables as files.
Many entries are writable, and writing a value to those files immediately
changes a live kernel parameter. Other entries are read-only kernel state, so do not assume
every path under /proc/sys accepts writes. This is how sysctl works:
# Read a kernel parameter directly
cat /proc/sys/net/ipv4/ip_forward # 0 = disabled, 1 = enabled
# Enable IP forwarding immediately (for routing/NAT)
echo 1 > /proc/sys/net/ipv4/ip_forward # takes effect instantly
# Equivalent sysctl command:
sysctl -w net.ipv4.ip_forward=1
# Important /proc/sys paths:
# net/ipv4/tcp_syncookies — SYN flood protection
# net/ipv4/conf/all/rp_filter — reverse path filtering
# vm/swappiness — swap aggressiveness
# vm/dirty_ratio — % of RAM before writes are flushed
# kernel/dmesg_restrict — restrict dmesg to root only
# kernel/kptr_restrict — hide kernel pointers in /proc
# kernel/randomize_va_space — address space layout randomization (ASLR)
# fs/file-max — max total open file descriptors system-wide
# Make sysctl changes permanent
echo 'net.ipv4.ip_forward = 1' > /etc/sysctl.d/99-routing.conf
sysctl --system # apply sysctl.d files from all supported configuration directories
# ── Debian 13 note ──────────────────────────────────────────────
# systemd-sysctl no longer reads /etc/sysctl.conf at boot. Vendor
# defaults now come from /usr/lib/sysctl.d/50-default.conf, shipped
# by linux-sysctl-defaults (Recommended by systemd → installed by
# default). Put local persistent settings in /etc/sysctl.d/*.conf.
#
# Caveat: procps `sysctl --system` DOES still read /etc/sysctl.conf
# last, after the *.d directories. So a stale /etc/sysctl.conf can
# silently apply on a manual `sysctl --system` but NOT at boot —
# a divergence worth checking after a bookworm → trixie upgrade.
dpkg -l linux-sysctl-defaults # confirm the package is present
cat /usr/lib/sysctl.d/50-default.conf # review the new vendor defaults
test -s /etc/sysctl.conf && echo 'leftover settings — migrate to /etc/sysctl.d/'/sys — The Kernel's Device Model
/sys is newer than /proc and was designed to address
/proc's organizational chaos. While /proc grew organically
and contains a mix of process info and kernel state, /sys has a structured layout built around the kernel device model. Registered devices,
drivers, buses, classes, and selected kernel subsystems expose objects and attributes here;
sysfs is extensive, but it is not a promise that every kernel object has a userspace entry.
/sys is mounted as sysfs — another virtual filesystem with
no disk backing. udev, the device manager, reads extensively from
/sys to learn about newly detected hardware and decide how to name
and configure device files in /dev.
The Top-Level /sys Structure
├── bus/ — organized by hardware bus type
│ ├── pci/ — all PCI/PCIe devices and drivers
│ ├── usb/ — all USB devices and hubs
│ ├── scsi/ — SCSI devices and storage handled through the SCSI layer, including many SATA/SAS devices; NVMe has its own bus/class
│ ├── i2c/ — I2C bus devices (temperature sensors, etc.)
│ └── platform/ — platform/embedded devices
├── block/ — all block (storage) devices
│ ├── sda/ — first SATA/SAS disk
│ └── nvme0n1/ — first NVMe drive
├── class/ — organized by device class/function
│ ├── net/ — network interfaces
│ ├── block/ — flat class view of block devices, including partitions; not identical in scope/layout to /sys/block
│ ├── input/ — keyboards, mice, touchscreens
│ └── hwmon/ — hardware monitoring sensors
├── devices/ — complete device tree, mirrors hardware topology
├── firmware/ — UEFI/ACPI firmware interfaces
│ └── efi/ — UEFI variables, Secure Boot state
├── kernel/ — kernel parameters and subsystems
│ ├── debug/ — debugfs mount point, if mounted; ftrace, tracing, zswap diagnostics
│ └── mm/ — memory management parameters
├── module/ — per-module parameters and reference counts
└── power/ — system power state (suspend, hibernate)
/sys/kernel/debug — debugfs
/sys/kernel/debug is the usual mount point for debugfs, a kernel
debugging filesystem. It is not part of ordinary application configuration and is commonly
accessible only to root. Depending on kernel configuration and mounted subsystems, it may
expose tracing interfaces such as ftrace, scheduler/debug information, zswap diagnostics,
driver-specific debug files, and other low-level kernel internals. Treat it as a diagnostic
surface: useful during performance and kernel investigations, but inappropriate as a stable
scripting API.
# Is debugfs mounted?
findmnt /sys/kernel/debug
# Common tracing/debug locations when available
ls /sys/kernel/debug/tracing 2>/dev/null
ls /sys/kernel/debug/zswap 2>/dev/nullPractical /sys Operations
── Network Interface Attributes ──
# Get the MAC address of eth0
cat /sys/class/net/eth0/address
# Read the interface operational state (text such as up, down, dormant, unknown)
cat /sys/class/net/eth0/operstate
# See link speed in Mb/s when the driver exposes this attribute
cat /sys/class/net/eth0/speed
# TX/RX bytes since boot (useful for network accounting)
cat /sys/class/net/eth0/statistics/rx_bytes
cat /sys/class/net/eth0/statistics/tx_bytes
── Block Device Attributes ──
# Is sda rotational? (0 = SSD/NVMe/non-rotational, 1 = HDD/rotational)
cat /sys/block/sda/queue/rotational
# Disk scheduler for sda
cat /sys/block/sda/queue/scheduler
# Change only to a scheduler shown as available in the preceding output:
echo none > /sys/block/sda/queue/scheduler # example; valid choices depend on kernel and device
# Get disk size in 512-byte sectors, then convert
cat /sys/block/sda/size
echo $(($(cat /sys/block/sda/size) * 512 / 1024 / 1024 / 1024)) GiB
── Hardware Sensors (hwmon) ──
# Find temperature sensors
for f in /sys/class/hwmon/hwmon*/temp*_input; do
echo "$(cat "$(dirname "$f")/name"): $(cat "$f") millidegrees C / $(($(cat "$f")/1000))°C"
done
── CPU Power Management ──
# Current CPU frequency of core 0
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
# CPU governor (performance/powersave/schedutil)
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
── UEFI Secure Boot State ──
cat /sys/firmware/efi/efivars/SecureBoot-* # raw bytes (5th byte: 1=enabled)
[ -d /sys/firmware/efi ] && echo "UEFI boot" || echo "BIOS boot"Using /sys for Device Discovery
The /sys/devices tree is the authoritative hardware topology map.
Tools like lspci, lsusb, and udevadm info
read from here:
# Get all attributes of a specific device (the udev way)
udevadm info --attribute-walk --name=/dev/sda
# Find the /sys path for a device
udevadm info --query=path --path=/sys/class/net/eth0
# → /devices/pci0000:00/0000:00:1f.6/net/eth0
# What kernel driver is bound to the first NVMe namespace?
readlink -f /sys/class/nvme/nvme0/device/driver
# List all PCI devices with their driver
for d in /sys/bus/pci/devices/*; do
echo "$(basename $d): $(cat $d/class 2>/dev/null) driver=$(basename $(readlink $d/driver 2>/dev/null || echo none))"
done/sys/fs/cgroup — cgroups v2 Resource Control
/sys/fs/cgroup is the control-group filesystem. On modern systemd-based Debian
systems it is normally the unified cgroup v2 hierarchy. cgroups organize
processes into a tree and let the kernel account for, limit, and delegate resources such as
CPU, memory, I/O, process counts, and pressure-stall information. You will see systemd
slices, scopes, and services represented here because systemd is the normal manager of the
host cgroup tree.
Do not casually write to random files under /sys/fs/cgroup on a systemd host.
Prefer systemd resource-control settings such as CPUQuota=,
MemoryMax=, TasksMax=, IOWeight=, and
systemctl set-property. Direct cgroup-file manipulation is useful for learning
and special container/runtime work, but systemd may overwrite or reorganize unmanaged changes.
# Confirm cgroup v2 is mounted
findmnt /sys/fs/cgroup
stat -fc %T /sys/fs/cgroup # cgroup2fs = unified cgroup v2
# See the cgroup of the current shell and PID 1
cat /proc/self/cgroup
cat /proc/1/cgroup
# Inspect systemd's slice/service tree
systemd-cgls
systemctl show ssh.service -p ControlGroup -p MemoryCurrent -p TasksCurrent
# Apply a supported systemd resource limit
systemctl set-property ssh.service MemoryMax=512M/dev — Everything Is a File
/dev is where Linux's "everything is a file" philosophy becomes
most visible. Many hardware devices and kernel-provided pseudo-devices expose a
device node in this directory, but not every item visible in sysfs
has a corresponding /dev node. Reading or writing a device node invokes
the associated kernel driver's interface; the semantics depend on that device and may
represent data transfer, control operations, or both.
Device files are of two types. Block devices (like disks) transfer data in chunks; they support seeking to any position. Character devices (like terminals and serial ports) expose stream- or record-oriented device I/O and generally do not provide block-device-style random access. Transfers may still occur in buffers rather than literally one byte per system call. Other special filesystem object types include FIFOs and Unix-domain sockets, which provide IPC through the filesystem namespace; they are not device nodes.
Device Naming Conventions
| Device | Type | Naming | Examples |
|---|---|---|---|
| SCSI-family disks (including many SATA/SAS/USB devices) | Block | /dev/sdX (a, b, c...) | /dev/sda, /dev/sdb, /dev/sda1 (partition 1) |
| NVMe disks | Block | /dev/nvmeXnY (controller, namespace) | /dev/nvme0n1, /dev/nvme0n1p1 (partition 1) |
| Virtual disks (VM) | Block | /dev/vdX (virtio) or /dev/xvdX | /dev/vda on KVM/QEMU VMs |
| Loop devices | Block | /dev/loopX | /dev/loop0 for a mounted ISO image |
| LVM logical volumes | Block | /dev/mapper/vg-lv or /dev/vg/lv | /dev/mapper/vg--debian-root |
| Terminals | Character | /dev/ttyX (physical), /dev/pts/X (pseudo) | /dev/tty1, /dev/pts/0 (SSH session) |
| Serial ports | Character | /dev/ttySX or /dev/ttyUSBX | /dev/ttyS0 (COM1), /dev/ttyUSB0 (USB serial) |
| USB devices | Character | /dev/bus/usb/XXX/YYY | Raw USB device access; usually accessed via libusb |
| GPU/graphics | Character | /dev/dri/cardX, /dev/dri/renderDX; vendor drivers may add other nodes | /dev/dri/card0, /dev/dri/renderD128, proprietary NVIDIA /dev/nvidia* |
| Sound | Character | /dev/snd/ | /dev/snd/pcmC0D0p (playback), /dev/snd/controlC0 |
| CDROM/optical | Block | /dev/srX | /dev/sr0 for first optical drive |
| MD RAID | Block | /dev/mdX | /dev/md0, /dev/md/hostname:0 |
Special Pseudo-Devices
Several device files in /dev are not backed by hardware at all — they
are kernel interfaces that produce or consume data in special ways:
command 2>/dev/null.dd if=/dev/zero of=/dev/sdb is destructive, but it is not a guaranteed secure erase for SSDs, thin-provisioned storage, or remapped sectors; use device secure erase, blkdiscard, or crypto-erase when appropriate.getrandom() or a well-maintained cryptographic library so initialization and error handling are explicit./dev/urandom after initialization and mainly differs during very early boot before the generator is ready. Kept for compatibility.CONFIG_DEVMEM/CONFIG_STRICT_DEVMEM, architecture rules, lockdown mode, and capabilities. It is not normally controlled through /etc/modprobe.d.udev — How Devices Get Their Files
Most device nodes are supplied initially by the kernel through devtmpfs
and then managed by udev, the userspace device manager. Administrators can
create special nodes with mknod in exceptional cases, but normal hardware
discovery is automatic. When the kernel detects hardware at boot or hot-plug time, it
emits a uevent that udev receives. udev
then consults its rules files in /etc/udev/rules.d/ and
/usr/lib/udev/rules.d/ to determine:
- 1What permissions, ownership, and tags to apply
- 2What persistent symlinks to create (for example
/dev/disk/by-id/and/dev/disk/by-uuid/) - 3What device properties or network-interface naming policy to apply
- 4What additional actions or notifications to trigger; long-running work should be delegated to services rather than performed inside a udev rule
# List all block devices with their device nodes
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
# Find disk by UUID (stable across device reordering)
ls -la /dev/disk/by-uuid/
# Find disk by label
ls -la /dev/disk/by-label/
# Find disks by hardware identity (usually preferable for device identity)
ls -la /dev/disk/by-id/
# Find disks by connection topology (changes if the device moves to another port)
ls -la /dev/disk/by-path/
# Get udev attributes for a specific device
udevadm info --query=all --name=/dev/sda
# Monitor udev events in real time (plug in USB to see this fire)
udevadm monitor --property
# Test a udev rule without reloading
udevadm test /sys/block/sda
# Generate random bytes using /dev/urandom
dd if=/dev/urandom bs=32 count=1 2>/dev/null | base64 # 256-bit random key
# See your current open terminals
ls -la /dev/pts/
# What process has /dev/sda open?
fuser /dev/sda
lsof /dev/sdaEvery device file has a major number (identifies the driver) and a
minor number (identifies the specific device instance). The kernel uses
this pair to route I/O to the correct driver. You can see them with
ls -la /dev/sda — the two numbers before the date are major and minor.
mknod creates device files manually (rarely needed; udev handles this
automatically). The mapping of major numbers to drivers is in
/proc/devices.
Troubleshooting Using the Filesystem Hierarchy
Knowing where things live transforms troubleshooting from guesswork into systematic investigation. Every problem has a trail in the filesystem.
Scenario: Service Won't Start
- 1Check journald logs:
journalctl -u servicename -n 50 --no-pager— the first place to look. - 2Check /var/log/: many services write their own logs here even on systemd systems.
ls -lt /var/log/ | head -10shows recently modified logs. - 3Check /etc/ config syntax: most services support a config test:
nginx -t,apache2ctl -t(equivalentlyapache2ctl configtest), andsshd -t. Run the sshd test as root and by full path, because/usr/sbinis normally absent from an unprivilegedPATHand the test also reads the host keys:sudo /usr/sbin/sshd -t. Success is silent with exit status 0; failures name the file and line on stderr. - 4Check runtime directories and sockets: inspect
systemctl status,systemctl show -p RuntimeDirectory, and relevant paths under/run. Missing directories, stale sockets, or incorrect permissions can prevent startup; the journal should normally record the failure. - 5Check /var/lib/servicename/: a corrupt database or state file prevents startup. For Debian-packaged PostgreSQL, cluster data normally lives below
/var/lib/postgresql/<major>/<cluster>/; a mismatchedPG_VERSION, ownership problem, or incomplete upgrade can prevent startup.
These are two different modes, not variants of the same check. sshd -t is test mode:
it checks the validity of the configuration file and the sanity of the keys, prints nothing on
success, and returns a usable exit status. sshd -T is extended test mode: it validates
and then writes the effective configuration to stdout. Use -t as the gate before
reloading, and -T when you need to know which value actually won.
Avoid the idiom sshd -T 2>&1 | head as a syntax check. In a pipeline the exit status
is that of head, so $? is almost always 0 and any &&
guard built on it is broken; head also truncates an alphabetically ordered dump, showing
the first directives rather than the interesting ones.
# Gate a reload on a real exit status
sudo /usr/sbin/sshd -t && systemctl reload ssh
# Validate a candidate file before installing it
sudo /usr/sbin/sshd -t -f /tmp/sshd_config.new
# Effective configuration after sshd_config.d/ drop-ins are merged
sudo /usr/sbin/sshd -T | grep -Ei 'permitrootlogin|passwordauth|pubkeyauth|port'
# Match blocks are only applied when a connection spec is supplied
sudo /usr/sbin/sshd -T -C user=deploy,host=jump.example.net,addr=10.0.0.5The drop-in directory makes -T more valuable than it used to be. Because
/etc/ssh/sshd_config carries an Include /etc/ssh/sshd_config.d/*.conf line
near the top, and sshd keeps the first value it sees for most keywords, a file such as the
50-cloud-init.conf shipped by many cloud images can silently override what you just
edited in the main file. Only the merged dump shows the winner.
Scenario: System Boot Failure
- 1/etc/fstab error: a wrong UUID, unavailable device, or unsuitable mount option can delay boot or enter emergency mode. From rescue media or an emergency shell, inspect
/etc/fstab, compare identifiers withblkid, and usefindmnt --verify. - 2/boot corrupt: kernel or initrd missing. Mount root, run
update-initramfs -u -k allandupdate-grubfrom chroot. - 3Encrypted-root or storage mapping failure: verify
/etc/crypttab, required packages, key sources, and the initramfs. After correcting early-boot mappings, rebuild withupdate-initramfs -u -k all. - 4Systemd unit failure:
systemctl list-units --failedfrom rescue mode. Look in/etc/systemd/system/for broken overrides.
Scenario: Disk Full
- 1/var/log/ overgrowth: logrotate misconfigured or a service logging excessively. Identify with
du -ahx /var/log | sort -h | tail -30. Use logrotate, journald vacuuming, or a deliberate archive/removal procedure; do not blindly delete active logs. - 2/var/cache/apt/archives/: accumulated .deb files. Clear with
apt clean. - 3/var/lib/docker/: inspect with
docker system df -v. Prune only objects confirmed unused.docker system prune -a --volumesis destructive and may remove unused volumes containing application data. - 4/tmp tmpfs full: a process may be writing large temporary files. Inspect with
du -ahx /tmp 2>/dev/null | sort -h | tail -30andlsof +D /tmpwhere practical. - 5Inode exhaustion: filesystem has bytes free but no inodes. A directory with millions of small files. Find with
df -ihand then narrow down withfind /var -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head.
Scenario: Permission Denied
- 1Check file ownership:
ls -la /path/to/file. Expected: config files in /etc owned root, data in /var/lib/service owned by that service's user. - 2Check group membership:
id usernameandgetent group groupname. Services often require membership in groups likessl-cert, but be careful with powerful groups:dockercan be root-equivalent on a rootful Docker host, anddiskcan permit raw block-device access. - 3Check ACLs: standard permission bits may not show additional ACL entries or the ACL mask that limits effective permissions.
getfacl /path/to/file. - 4Check AppArmor:
aa-statusandjournalctl -k | grep DENIED. AppArmor confines processes to specific filesystem paths defined in profiles under/etc/apparmor.d/.
# Complete filesystem investigation toolkit
# What is using all the inodes on this filesystem?
df -ih
find /var -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -10
# Is a deleted file still holding disk space? (open but unlinked)
lsof +L1 # show files with 0 hard links (deleted but still open)
# Fix: restart the service holding the file open (it will release the space)
# Parse and verify /etc/fstab without mounting filesystems
findmnt --verify --verbose --tab-file /etc/fstab
# After review, test mountable entries; -f performs a fake mount where supported
mount -a -f -v
# Check AppArmor denials in the kernel log
journalctl -k --grep="apparmor.*DENIED"
# Which service user owns files in /var/lib?
ls -la /var/lib/ | awk '{print $3, $9}' | sortQuick Reference Cheatsheet
Where Does It Belong?
| If you have... | It goes in... | Because... |
|---|---|---|
| A config file for a service | /etc/service-name/ | Configuration is human-edited, machine-specific |
| A log file from a running service | /var/log/ | Variable runtime data; survives reboots |
| A PID file for a daemon | /run/ | Runtime state; cleared on reboot (tmpfs) |
| A Unix domain socket | /run/service/ | Sockets are per-session runtime objects |
| A database's actual data files | /var/lib/service/ | Persistent application state, not config |
| A cache that can be deleted | /var/cache/service/ | Expendable derived data, can be regenerated |
| A binary you compiled from source | /usr/local/bin/ | Locally installed, not managed by package manager |
| A third-party self-contained app | /opt/app-name/ | Self-contained; does not scatter files everywhere |
| A Snap or Flatpak app | /snap, /var/lib/snapd, /var/lib/flatpak, or ~/.local/share/flatpak | Application-bundle frameworks use their own non-core-FHS deployment roots |
| Data served by a web/FTP service | /srv/ or /var/www/ | FHS: srv for served data; /var/www is common Debian default |
| A short-lived temporary file your script creates | /tmp/ (use mktemp) |
Cleared on reboot; world-writable with sticky bit |
| A temporary file that must survive reboot | /var/tmp/ | Longer-lived temporary storage; cleaned by age policy, not simply by reboot |
| Per-user config for an app | ~/.config/app/ | XDG base directory spec; user-specific |
| Per-user cache data | ~/.cache/app/ | XDG cache dir; safe to delete |
Key Files Every Admin Needs to Know
| File | What it does | How to safely edit |
|---|---|---|
| /etc/passwd | User accounts (not passwords) | vipw, usermod, or other account-management tools |
| /etc/shadow | Password hashes | passwd, chage, or vipw -s as appropriate |
| /etc/sudoers | sudo authorization | visudo — validates syntax before saving |
| /etc/fstab | Filesystem mount table | Run findmnt --verify; then test deliberately with mount -a when safe |
| /etc/crypttab | Encrypted device unlock table | Text editor; rebuild initramfs when the mapping is needed in early boot (for example root or an early-mounted filesystem) |
| /etc/hosts | Static hostname resolution | Text editor; immediate effect |
| /etc/apt/sources.list.d/*.sources | APT repositories in the preferred deb822 format; legacy /etc/apt/sources.list and *.list are also supported |
Text editor; run apt update after |
| /etc/ssh/sshd_config | SSH server settings | Test with sudo /usr/sbin/sshd -t; then systemctl reload ssh on Debian
(the unit is ssh.service with sshd.service as an alias; RHEL-family systems
use sshd only). Reload sends SIGHUP and leaves established sessions untouched. Verify with a
second connection before closing the first, and check
systemctl is-enabled ssh.socket — under socket activation the listening port comes from the
socket unit, not from Port in sshd_config |
| /etc/default/grub | GRUB boot parameters | Text editor; run update-grub after |
| /proc/sys/net/ipv4/ip_forward | IP routing enabled/disabled | sysctl -w or echo; persist in /etc/sysctl.d/ |
| /proc/sys/kernel/randomize_va_space | Address Space Layout Randomization setting | Read with sysctl kernel.randomize_va_space; avoid disabling except for controlled debugging |
Virtual Filesystem Quick Reference
| Filesystem | Mount point | Type | What it shows | Writable? |
|---|---|---|---|---|
| procfs | /proc | proc | Process state + kernel internals | Mostly read-only; selected /proc/sys entries writable |
| sysfs | /sys | sysfs | Hardware topology + driver params | Some attributes |
| devtmpfs | /dev | devtmpfs | Device nodes | Kernel/devtmpfs and udev; privileged users can create nodes explicitly |
| tmpfs | /run | tmpfs | Runtime PID files, sockets | Yes (daemons) |
| tmpfs | /tmp | tmpfs on new Debian 13 installs | Temporary files | Yes (all users) |
| cgroup2fs | /sys/fs/cgroup | cgroup2 | Resource control hierarchy | Yes (root/systemd) |
| debugfs | /sys/kernel/debug | debugfs | Advanced kernel debug info | Some (root) |
| pstore | /sys/fs/pstore | pstore | Previous crash/oops messages | Read-mostly |
Programs live in /usr — read-only, installed by packages, shared across users.
Configuration lives in /etc — machine-specific, edited by humans and automation, backed up carefully.
Variable state lives in /var — grows and shrinks during operation; high-growth subtrees are often isolated on servers.
Runtime state lives in /run — in RAM, always gone after reboot, never back up.
Kernel interfaces live in /proc, /sys, /dev — generated by the kernel, never on disk.
When you cannot find a file, ask: is it config? Data? Cache? Runtime? That narrows it to one of these trees immediately.
Primary Sources and Scope
This guide describes FHS 3.0 as applied to Debian 13 (Trixie), with Linux-kernel and systemd behavior where FHS itself does not specify implementation details. Package paths and defaults can still vary by architecture, installed packages, local policy, containers, and upgrades from earlier Debian releases.
- FHSFilesystem Hierarchy Standard 3.0 — normative hierarchy and placement rules.
- DebianDebian Policy, chapter 9 — Debian's FHS exceptions,
/run,/usr/local, multiarch, and mail-path rules. - TrixieDebian 13 release notes —
/tmptmpfs, age cleanup, usrmerge-related notes, and Debian 13 sysctl behavior. - systemdDebian systemd file-hierarchy manual and related Trixie man pages — runtime directories, unit locations, tmpfiles, cgroups, and resource control.
- Kernel
No comments:
Post a Comment