Text Processing

Text Processing on the Command Line

Almost everything a LAMP server tells you arrives as text: access and error logs, php.ini, .env files, composer.json. A handful of small tools joined with the pipes of Redirection and Pipes answer questions such as "who is hammering the login page?" in one line, over SSH, on a server with no GUI at all.

The examples read a real Apache 129 access log. The script below fills it with about 500 requests: varied pages and browsers, some missing files and a burst from a vulnerability scanner. A local route makes the kernel treat the whole documentation range 203.0.113.0/24 (RFC 5737) as its own, so curl 3,008 --interface can send each request from a different client address. It runs for about three and a half minutes and prints nothing.

traffic.sh: generating varied traffic against the local ApacheShell
sudo ip route add local 203.0.113.0/24 dev lo    # borrow a documentation range as client IPs
ips=(10 10 10 10 10 21 21 21 34 34 47 58)         # repeats skew the traffic toward a few
pages=(/ / / / / /index.html /index.html /?page=2 /?utm_source=newsletter
       /icons/ubuntu-logo.png /icons/ubuntu-logo.png /favicon.ico /about.php)
agents=("Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0"
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/140.0.0.0 Safari/537.36"
        "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
        "curl/8.18.0")
probes=(/wp-login.php /.env /xmlrpc.php /phpmyadmin/ /.git/config /server-status)
hit() { curl -s -o /dev/null --interface "203.0.113.$1" -A "$2" "http://127.0.0.1$3"; }
for i in $(seq 440); do
  hit "${ips[RANDOM % 12]}" "${agents[RANDOM % 4]}" "${pages[RANDOM % 13]}"
  sleep "0.$(( RANDOM % 10 ))"
  if (( i == 220 )); then                          # a vulnerability scanner arrives
    for j in $(seq 60); do hit 99 "python-requests/2.32.5" "${probes[j % 6]}"; done
  fi
done
sudo ip route del local 203.0.113.0/24 dev lo

Subsections