Archive · Linux

Text Processing: grep, sed, awk and Pipes

Chain small tools together to answer questions about logs and configuration files.

1 min read

  • Ubuntu 18.04.2 LTS
  • GNU grep 3.1
  • GNU awk 4.1

Unix tools do one job and read text from standard input, so they combine with |.

# the ten client IPs with most requests in an nginx access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head

grep -i "error" /var/log/syslog | tail -20     # case-insensitive match
grep -rn "listen 80" /etc/nginx/               # recursive, with line numbers
sed -i 's/PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config
cut -d: -f1,7 /etc/passwd                      # user and login shell
Tool Best at
grep Selecting lines
sed Search and replace, line edits
awk Working with columns
sort, uniq Ordering and counting

Test a sed -i edit without -i first, or use -i.bak to keep a backup copy of the file.