uniq
Text processing
Usage
sort [file] | uniq -cCollapses only runs of identical neighbouring lines, not duplicates spread through the file, so it is nearly always used after sort.
Common flags
| Flag | Meaning |
|---|---|
| -c | Prefix each line with how many times it repeated |
| -d | Only the lines that appear more than once |
| -u | Only the lines that appear exactly once |
| -i | Ignore case |
| -f | Skip the first N fields before comparing |
Examples
sort access.log | uniq -c | sort -rn | headThe most repeated lines.
uniq -d list.txtJust the duplicates.
These were built to be piped together. Three joined commands are usually shorter than one that does everything.
How to read this
- Square brackets [ ] mark a part you may leave out.
- An ellipsis … means you can list more than one.
- Flags are case-sensitive — in some commands -r and -R do different things.
Questions
Q. What does uniq do?
Collapses only runs of identical neighbouring lines, not duplicates spread through the file, so it is nearly always used after sort.
Q. How do I type it?
sort [file] | uniq -c — square brackets mark the parts you can leave out.
Q. How many flags are worth knowing?
5 are listed here; the full set is in man uniq. This command sits under Text processing.
Related commands
man page: man uniq