Pipes and redirection

Every command reads something and writes something. Pipes and redirection let you control both ends. One command's output can then feed straight into the next. This is where the terminal beats any app. You build a small tool out of parts that already exist.

What you'll be able to do

• Send one command's output into another with a pipe.
• Chain two and three commands into a single line.
• Write output into a file with > and append with >>.
• Hide error messages with 2>/dev/null.
• Pick columns, sort, count and edit text with cut, sort, uniq and sed.

Step 1: Make a file to work with

This command writes a small access log. It uses redirection, which this lesson covers below. For now, read it as "put the text between the markers into access.log". There is no output, and the prompt returns.

$ cd ~
$ cat > access.log << 'EOF'
10.0.0.4 GET /index.html 200
10.0.0.7 GET /about.html 200
10.0.0.4 POST /login 302
10.0.0.9 GET /missing.html 404
10.0.0.7 GET /index.html 200
10.0.0.4 GET /index.html 200
EOF

Each line has four parts: a client address, a method, a path, and a status code.

Step 2: What a pipe is

The pipe symbol | takes the output of the command on its left and feeds it to the command on its right. Start with a count and no pipe.

$ wc -l access.log
6 access.log

Now the same count, by piping cat into wc.

$ cat access.log | wc -l
6

cat prints the file. The pipe hands that text to wc, and wc counts the lines. The number is the same, but wc never saw a file name, so it prints only the count.

Step 3: Chaining commands

cut picks one field from every line. -d ' ' sets the space as the separator. -f1 asks for the first field, the client address.

$ cut -d ' ' -f1 access.log
10.0.0.4
10.0.0.7
10.0.0.4
10.0.0.9
10.0.0.7
10.0.0.4

Now chain three. sort groups identical addresses. uniq -c then collapses each group and counts it.

$ cut -d ' ' -f1 access.log | sort | uniq -c
      3 10.0.0.4
      2 10.0.0.7
      1 10.0.0.9

uniq only compares neighbours, which is why sort must come first. Add sort -n to order the results by count.

$ cut -d ' ' -f1 access.log | sort | uniq -c | sort -n
      1 10.0.0.9
      2 10.0.0.7
      3 10.0.0.4

Four commands, one line, and a ranked list of your busiest clients.

Step 4: Writing output to files

A single > sends a command's output into a file, replacing whatever was there before. Two >> append instead, adding to the end.

$ echo "first line" > demo.txt
$ echo "second line" >> demo.txt
$ cat demo.txt
first line
second line

Step 5: Hiding errors

Commands write errors to a second stream, separate from normal output. You can see it when a command fails.

$ ls no-such-file
ls: cannot access 'no-such-file': No such file or directory

The 2> redirects that error stream, numbered 2, into a file. The file /dev/null throws away anything sent to it.

$ ls no-such-file 2>/dev/null

Nothing is printed now. This helps when you search widely and expect harmless errors.

$ find ~ -name "*.log" 2>/dev/null
/data/data/com.termux/files/home/app.log

Step 6: Sorting and counting numbers

sort puts lines in order. By default it compares them as text, so 10 comes before 2. Add -n to sort by numeric value instead.

$ printf '10\n2\n1\n30\n2\n' > numbers.txt
$ sort -n numbers.txt
1
2
2
10
30

Add -u to drop duplicate lines as well.

$ sort -n -u numbers.txt
1
2
10
30

Step 7: Editing text with sed

sed edits a stream of text. The form s/old/new/g replaces text. The trailing g means every match on a line.

$ sed 's/line/LINE/g' demo.txt
first LINE
second LINE

Leave off the g and only the first match on each line changes.

$ sed 's/line/LINE/' demo.txt
first LINE
second line

sed prints its result to the screen. It does not change the file unless you redirect the output.

Step 8: A realistic one-liner

Which pages were requested, and how often? Chain cut, sort, uniq -c and sort -n on the third field, the path.

$ cut -d ' ' -f3 access.log | sort | uniq -c | sort -n
      1 /about.html
      1 /login
      1 /missing.html
      3 /index.html

Read it from the top. One request each for two pages, one failed request, and three hits on the home page.

Now find every failed request. grep keeps the lines with a 404, and cut shows only the path.

$ grep "404" access.log | cut -d ' ' -f3
/missing.html

That is the shape of the whole lesson. Small commands, joined by pipes, answering a real question.

Quick reference

Command	What it does
A | B	Feed the output of A into B
cut -d DELIM -f N FILE	Print field N, split on the delimiter
sort FILE	Sort lines as text
sort -n FILE	Sort lines as numbers
sort -u FILE	Sort and drop duplicates
uniq -c	Count runs of identical adjacent lines
sed 's/OLD/NEW/g' FILE	Replace every match in a stream
CMD > FILE	Write output to a file, replacing it
CMD >> FILE	Append output to a file
CMD 2>/dev/null	Discard error messages

Next steps

You now have the core of the command line. Put the whole set to work in a real investigation with Logs Never Lie. To look up a command you half remember, revisit Reading and searching text. If paths still trip you up, refresh Navigating the filesystem.