Text processing with sed and awk

sed and awk are the two workhorses of text on the command line. sed rewrites lines as they pass through. awk splits each line into fields. Then you can pick, test or total them. Together they turn a wall of output into the few numbers or lines you actually want.

What you'll be able to do

• Replace text with sed. Control how many times each line is touched.
• Delete lines that match a pattern. Print only a range of lines.
• Pull single fields out of a line with awk.
• Read colon-separated data with awk -F.
• Total a column. Build a pipeline over a log file.

Step 1: Create a file and substitute text with sed

Start with a small file. The heredoc writes it in one go, and cat shows what landed.

cat > scores.txt << 'EOF'
alice 90 maths
bob 75 physics
carol 88 chemistry
dave 60 maths
EOF
cat scores.txt

alice 90 maths
bob 75 physics
carol 88 chemistry
dave 60 maths

The s command substitutes. It takes a pattern, a replacement and flags. By default it replaces the first match on each line.

sed 's/maths/MATHS/' scores.txt

alice 90 MATHS
bob 75 physics
carol 88 chemistry
dave 60 MATHS

Every line containing maths changed. Now watch the g flag. It replaces every match on a line, not only the first.

printf 'cat dog cat\ndog cat dog\ncat cat cat\n' > words.txt
sed 's/cat/CAT/' words.txt

CAT dog cat
dog CAT dog
CAT cat cat

Only the first cat on each line changed. Add g and every one changes.

sed 's/cat/CAT/g' words.txt

CAT dog CAT
dog CAT dog
CAT CAT CAT

Step 2: Delete and print lines with sed

The d command deletes every line that matches a pattern.

sed '/maths/d' scores.txt

bob 75 physics
carol 88 chemistry

The -n flag turns off the default printing of every line. Pair it with the p command to print only the lines you ask for. Here that is a range.

sed -n '2,3p' scores.txt

bob 75 physics
carol 88 chemistry

Step 3: Pick fields with awk

awk splits each line on whitespace. The pieces are numbered $1, $2 and so on. The special name $NF means the last field.

awk '{print $1}' scores.txt

alice
bob
carol
dave

awk '{print $NF}' scores.txt

maths
physics
chemistry
maths

When the fields are separated by something other than spaces, set the separator with -F. This file uses colons.

printf 'web1:10.0.0.11:8080\nweb2:10.0.0.12:8080\ndb1:10.0.0.21:5432\n' > servers.txt
awk -F: '{print $2}' servers.txt

10.0.0.11
10.0.0.12
10.0.0.21

Step 4: Total a column with awk

The END block runs once, after the last line. That is where a total belongs.

printf '12\n7\n30\n5\n' > numbers.txt
awk '{sum+=$1} END {print sum}' numbers.txt

54

The first block adds each line's value into sum. The END block prints it once the file runs out. Add a counter in the first block, and divide in the END block. The same shape gives you an average.

Step 5: A realistic pipeline over a log file

Here is a small access log. Each line records an address, a request and a status code.

cat > access.log << 'EOF'
10.0.0.4 - - "GET /index.html" 200
10.0.0.7 - - "GET /about.html" 404
10.0.0.4 - - "GET /style.css" 200
10.0.0.9 - - "GET /missing.png" 404
10.0.0.4 - - "GET /app.js" 200
10.0.0.7 - - "GET /old.html" 404
EOF

Now chain the tools. Take the first field, then count how often each value repeats. Sort the result so the busiest address is on top.

awk '{print $1}' access.log | sort | uniq -c | sort -rn

      3 10.0.0.4
      2 10.0.0.7
      1 10.0.0.9

Quick reference

Command	What it does
sed 's/old/new/' file	Replace the first match on each line
sed 's/old/new/g' file	Replace every match on each line
sed '/pattern/d' file	Delete lines that match the pattern
sed -n '2,3p' file	Print only lines 2 to 3
sed -i 's/old/new/g' file	Edit the file in place
awk '{print $1}' file	Print the first field of each line
awk '{print $NF}' file	Print the last field of each line
awk -F: '{print $2}' file	Split on colons and print the second field
awk '{sum+=$1} END {print sum}' file	Add up a column and print the total

Next steps

• Environment variables and PATH to control the tools your pipeline runs.
• Archives and compression to pack the files you produce.
• Pipe Dreams to build longer pipelines of your own.