uniq
Text processing
Usage
sort [file] | uniq -cCollapses only runs of identical neighbouring lines, not duplicates spread through the file, so it is nearly always used after sort.
Common flags
| Flag | Meaning |
|---|---|
| -c | Prefix each line with how many times it repeated |
| -d | Only the lines that appear more than once |
| -u | Only the lines that appear exactly once |
| -i | Ignore case |
| -f | Skip the first N fields before comparing |
Examples
sort access.log | uniq -c | sort -rn | headThe most repeated lines.
uniq -d list.txtJust the duplicates.
These were built to be piped together. Three joined commands are usually shorter than one that does everything.
How to read this
- Square brackets [ ] mark a part you may leave out.
- An ellipsis … means you can list more than one.
- Flags are case-sensitive — in some commands -r and -R do different things.
Questions
QWhat does uniq do?
Collapses only runs of identical neighbouring lines, not duplicates spread through the file, so it is nearly always used after sort.
QHow do I type it?
sort [file] | uniq -c — square brackets mark the parts you can leave out.
QHow many flags are worth knowing?
5 are listed here; the full set is in man uniq. This command sits under Text processing.
Related commands
man page: man uniq