uniq

Collapse or count repeated lines that sit together

About

Removes consecutive duplicate lines, or reports how many times each run repeats. It only sees neighbours, so feed it ordered input. Handy for frequency counts and for cleaning logs.

Install it

Included with Termux

Run this inside Termux. If the package is already present, pkg will say so and do nothing.

Commands worth knowing

6 of the invocations you will actually reach for. Tap the copy button to put one on your clipboard.

uniq ips.txt Drop consecutive duplicate lines, keeping one copy of each run.
uniq -c ips.txt Prefix each distinct line with how many times it appeared in a row.
uniq -d ips.txt Print only the lines that were repeated.
uniq -u ips.txt Print only the lines that appeared exactly once.
uniq -i names.txt Treat lines that differ only in case as duplicates.
uniq -f1 log.txt Skip the first field when deciding whether two lines match.

Worked example

Run uniq -c ips.txt and this is what you should see.

termux
$ uniq -c ips.txt
      3 203.0.113.7
      1 198.51.100.2
      2 192.0.2.55

Developed by

GNU Project .