Linux Server Hardening: CIS, eBPF, and CrowdSec

A fresh Ubuntu 24.04 image is not insecure. It is convenient, which is a different thing. It ships with SSH open to the world, an unpartitioned /tmp, password authentication available, and a sysctl set tuned for a laptop rather than a machine with a public address.

Linux server hardening is the work of removing that convenience in the right order. Get the order wrong and you lock yourself out of a box at 11pm. Get it right and the same box survives a scanner, a credential stuffer, and a dropped web shell without anyone being paged.

This is the sequence that holds up for a small fleet: a two-person team running forty client sites, some on VPS instances, a couple on dedicated hardware. Nothing here needs an enterprise contract.

What actually gets exploited on a default install

Most compromises on a stock server follow one of three paths, and none of them involves a memory-corruption zero-day against the kernel.

The first is credentials. SSH with password authentication enabled answers every attempt, and distributed attempts arrive from thousands of addresses, each trying once or twice. A local tool that counts failures per source address never fires, because no single address is guilty of enough failures. The address rotation is the attack, not a side effect of it.

The second is writable-and-executable storage. A web app that lets an attacker write a file into /tmp or /dev/shm has, on a default mount, also given them a place to run it. Droppers love /dev/shm for exactly this reason: it is a tmpfs, so it survives long enough to run and leaves nothing on disk to scan.

The third is the service you forgot. A database that only listens on localhost is fine. A database that was briefly bound to 0.0.0.0 during a migration and never reverted is a data breach waiting for a port scan.

Hardening is mostly about making each of those paths boring. You cannot patch your way out of a wrong default.

Mounts and sysctls first, because they cannot lock you out

Start with the controls that fail closed. A bad mount option or a bad sysctl can break an application, but neither will stop you from logging in to fix it.

Anywhere untrusted code can write needs to be unexecutable. In /etc/fstab:

# Temporary filesystems: no devices, no setuid, no execution
tmpfs  /tmp      tmpfs  rw,nosuid,nodev,noexec,relatime,size=1G   0 0
tmpfs  /var/tmp  tmpfs  rw,nosuid,nodev,noexec,relatime,size=512M 0 0
tmpfs  /dev/shm  tmpfs  rw,nosuid,nodev,noexec,relatime,size=256M 0 0

noexec is the one that changes outcomes. A PHP shell that can write a file can no longer run it. The other two close smaller doors: nosuid stops a dropped binary from carrying privilege bits, and nodev stops a device node from appearing in a scratch directory.

The sizes are illustrative rather than prescribed. tmpfs lives in RAM, so those numbers are memory budgets as much as disk quotas. Size /dev/shm to the largest shared-memory consumer on the box, because PostgreSQL parallel workers and some browser runtimes will fail with an undersized tmpfs.

The other caveat: some installers unpack a helper into /tmp and execute it. When that happens, point the installer at a different working directory rather than removing noexec permanently. For Debian-family package tooling, TMPDIR=/var/cache/apt/archives/partial apt install ... usually does it. Fix the tool, not the control.

Next, the network stack. A drop-in file keeps this manageable:

# /etc/sysctl.d/60-hardening.conf
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.log_martians = 1
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
kernel.sysrq = 0

Apply with sysctl --system, then confirm one value survived: sysctl kernel.kptr_restrict should return 2. Files in /etc/sysctl.d/ are read in lexical order and later files win, so a package drop-in numbered above yours will quietly override it. The number in the filename is load-bearing.

rp_filter = 1 is strict reverse-path filtering. On a host with asymmetric routing, strict mode drops legitimate packets; use 2 (loose) there. On a normal web server, strict is correct and blocks spoofed source addresses.

SSH: make a stolen key worth less

Changing the SSH port is not hardening. It removes noise from your logs and nothing else. The controls that matter are authentication method, key type, and who is allowed to log in at all.

Put the settings in a drop-in rather than editing sshd_config in place, so a package upgrade can never silently revert them:

# /etc/ssh/sshd_config.d/00-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitEmptyPasswords no
PubkeyAuthentication yes
AuthenticationMethods publickey
MaxAuthTries 3
MaxSessions 4
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
PermitUserEnvironment no
AllowUsers deploy ops

Two lines do most of the work. AuthenticationMethods publickey removes every password path, including the keyboard-interactive fallback that often survives a PasswordAuthentication no in the wrong place. AllowUsers turns a valid credential for a forgotten account into a non-event.

Use Ed25519 keys, and prefer a hardware-backed sk variant for anyone with production access. The private key for an ed25519-sk credential lives on the token and requires a physical touch, so a copied ~/.ssh directory is worth nothing on its own [2]:

ssh-keygen -t ed25519-sk -O resident -O verify-required -C "[email protected]"

-O resident stores the credential on the device so it can be discovered on another machine. -O verify-required asks for the token PIN on every use, which is the difference between "possession" and "possession plus knowledge".

Before you reload anything, validate:

sudo sshd -t && sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|allowusers'

Then keep your current session open, and open a second one. Only close the first when the second works. That habit has saved more production hosts than any rollback plan.

Two more things while you are here. Put administrative SSH behind a private overlay instead of the public interface; the WireGuard vs Tailscale comparison covers the trade-offs for a small team. And if the box is a hypervisor or a database host, harden the account layer with pam_faillock as a backstop for console and su attempts:

# /etc/security/faillock.conf
deny = 5
fail_interval = 900
unlock_time = 1800

Lockouts from pam_faillock apply to local logins and sudo, not to key-based SSH, so this does not create a second way to lock yourself out remotely. It does slow down an attacker who already has a shell.

Kernel-side visibility without the auditd tax

Static configuration describes what should happen. It cannot tell you that nginx just spawned /bin/sh, which is the single most useful signal on a web server.

auditd can watch syscalls, and for a narrow rule set it is fine. Widen the rules and the cost shows up as I/O and CPU on the busiest machine you own, precisely when you cannot afford it. eBPF takes a different route: small verified programs run inside the kernel at tracepoints, write events into a ring buffer, and never copy the payload into user space unless a rule asked for it [4]. That is why an eBPF agent can watch every execve on a busy host while a comparable auditd rule set struggles.

Cilium's Tetragon is the tool I reach for, because enforcement and observation share one policy file. A tracing policy that kills shells spawned by a service account:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "kill-shell-from-web"
spec:
  kprobes:
    - call: "sys_execve"
      syscall: true
      args:
        - index: 0
          type: "string"
      selectors:
        - matchArgs:
            - index: 0
              operator: "Prefix"
              values:
                - "/bin/sh"
                - "/bin/bash"
                - "/usr/bin/curl"
                - "/usr/bin/wget"
                - "/usr/bin/nc"
          matchActions:
            - action: Sigkill

That policy is a starting point, not a finished control. It will also kill legitimate cron wrappers that shell out to curl, and it does nothing about a reverse shell opened by a Python process that never calls execve. Run it in Post mode first, collect a week of events, and only then decide which paths are genuinely never expected on that host. Falco is the other common choice, with a large default rule set and a different operational shape; the EDR, XDR, and MDR guide explains where host-level tracing fits against agent-based endpoint products, and Tetragon's documentation is the reference for policy syntax [5].

The limitation worth saying out loud: kernel telemetry tells you what happened. It does not tell you whether it mattered. Someone has to own the alerts, or the agent becomes another dashboard nobody reads.

CrowdSec: the blocklist is the product

fail2ban reads a log, counts failures per address, and adds a firewall rule. That model works against a single noisy attacker and fails against a rotating botnet, because the count never reaches the threshold.

CrowdSec separates detection from enforcement [3]. Parsers turn log lines into events. Scenarios decide whether a sequence of events is an attack. Decisions land in a local database, and a bouncer applies them to nftables, iptables, or a reverse proxy. The part that changes the economics is the community blocklist: when enough independent instances report the same address, it propagates to everyone.

Install on Ubuntu from CrowdSec's repository, with the nftables bouncer:

curl -s https://install.crowdsec.net | sudo sh
sudo apt-get install -y crowdsec crowdsec-firewall-bouncer-nftables
sudo cscli collections install crowdsecurity/linux crowdsecurity/nginx
sudo systemctl reload crowdsec

Then confirm it is actually parsing something, because a bouncer with no decisions is a firewall rule that never fires:

sudo cscli metrics show acquisition
sudo cscli decisions list
sudo cscli alerts list --limit 5

The false-positive risk is real and worth planning for. Community blocklists contain addresses that were abusive for someone else, and shared egress ranges (corporate NAT, carrier-grade NAT, some cloud egress pools) can put a legitimate user behind a banned address. Whitelist your own ranges and any partner integrations on day one, and check cscli decisions list before you conclude that a login failure is the user's fault.

Codify it, or rebuild it by hand next quarter

Every control above is one rebuild away from disappearing. A new VPS spins up from the same stock image, and the hardening lives in a shell history that nobody saved.

Two mechanisms fix that. The first is a configuration management playbook that applies the same files you tested by hand:

- name: Hardened Linux baseline
  hosts: all
  become: true
  tasks:
    - name: Install unattended security updates
      ansible.builtin.apt:
        name: unattended-upgrades
        state: present
        update_cache: true

    - name: Apply kernel sysctl baseline
      ansible.builtin.copy:
        src: files/60-hardening.conf
        dest: /etc/sysctl.d/60-hardening.conf
        mode: "0644"
      notify: Reload sysctl

    - name: Apply SSH hardening drop-in
      ansible.builtin.copy:
        src: files/00-hardening.conf
        dest: /etc/ssh/sshd_config.d/00-hardening.conf
        mode: "0600"
      notify: Validate and reload sshd

  handlers:
    - name: Reload sysctl
      ansible.builtin.command: sysctl --system
    - name: Validate and reload sshd
      ansible.builtin.shell: sshd -t && systemctl reload ssh

The second is a scanner that checks the result against the benchmark instead of against your memory. CIS publishes distribution-specific benchmarks, and the Ubuntu benchmark page lists the current version and the tooling that reads it [1]. oscap can evaluate a host against a SCAP profile and produce a machine-readable report, which is what turns "we hardened it" into a number you can track between quarters. Expect the first scan to be uncomfortable. Level 1 findings are usually a long list of small things, and fixing them in one pass is how teams break production.

Automating this is also how you make the hardening survive a rebuild onto dedicated hardware or a fresh VPS: the playbook is the artifact, the server is disposable.

Where teams get this wrong

Three patterns show up again and again. Linux server hardening fails on operational habits far more often than on missing controls.

They harden one server and call it a standard. The fleet drifts within a month, and the drift is invisible until an incident review.

They enforce before they observe. Killing every execve of /bin/bash on day one takes down the backup agent, the monitoring plugin, and one cron job nobody documented.

They collect telemetry with no owner. An eBPF agent writing to a local file that nobody reads is worse than no agent, because it creates the belief that someone is watching. If you would rather not staff that rotation, managed monitoring is the honest alternative, and the wider control set sits under cybersecurity services.

The order matters more than the tooling. Mounts and sysctls, then SSH, then kernel visibility, then a blocklist, then automation. Each step is testable on its own, and none of them requires you to trust a dashboard you have not read. That sequence is the whole of Linux server hardening; everything else is a variant of it.