Linux uniq Command

In Linux we use “uniq” command to filters out the repeated lines in a file. or to remove duplicates we need the uniq command.

By default, uniq command removes any duplicate lines that are adjacent to each other.

We can also use uniq command to filters out the adjacent matching lines from the input file and writes the filtered data to the output file.

uniq Command Syntax

~$  uniq [OPTION] [INPUT[OUTPUT]]

Option	Description
-c –-count	It tells how many times a line was repeated by displaying a number as a prefix with the line.
-d –-repeated	It only prints the repeated lines and not the lines which aren’t repeated.
-D –-all-repeated[=METHOD]	It prints all duplicate lines and METHOD can be any of the following
-f N –-skip-fields(N)	It allows you to skip N fields(a field is a group of characters; delimited by whitespace) of a line before determining the uniqueness of a line.
-i –-ignore case	By default comparisons done are case sensitive but with this option case insensitive comparisons can be made.
-s N –-skip-chars(N)	It doesn’t compare the first N characters of each line while determining uniqueness. This is like the -f option; but it skips individual characters rather than fields.
-u –-unique	It allows you to print only unique lines.
-z –-zero-terminated	It will make a line end with 0 bytes (NULL); instead of a newline.
-w N –-check-chars(N)	It only compares N characters in a line.
––help	It displays a help message and exit.
–-version	It displays version information and exit.
none	Do not delimit duplicate lines at all. This is the default.
prepend	Insert a blank line before each set of duplicated lines.
separate	Insert a blank line between each set of duplicated lines.

uniq -c option

uniq -c option shows total of each repeated Number of Lines.

~$  uniq -c music.txt

uniq -d option

uniq -d option just print the repeated lines only one time.

~$  uniq -d music.txt

uniq -D option

uniq -D option print all repeated lines.

~$  uniq -D music.txt

uniq -u option

uniq -u option print only the unique lines.

~$  uniq -u music.txt

uniq -f N option

uniq -f N option allows the N (Numbers) fields to be skipped while comparing the uniqueness of the lines.

~$  uniq -f 3 music.txt

# This option is useful when the lines are numbered as shown in the example.

music.txt
1.Oh-ooh-whoa-oh-oh-oh-oh
2.Oh-ooh-whoa-oh-oh-oh-oh
3.Oh-ooh-whoa-oh, oh-oh-oh-oh
4.You know you love me (yo), I know you care (uh-huh)
5.Just shout whenever (yo), and I'll be there (uh-huh)
6.You are my love (yo), you are my heart (uh-huh)
7.And we will never, ever, ever be apart (yo, uh-huh)
8.Are we an item? (Yo) girl, quit playin' (uh-huh)

uniq -i option

uniq -i option is used to make the comparison case-insensitive.

~$  uniq -i music.txt

uniq -w option

uniq -w option is a way to skipping characters, we can also ask uniq to limit the comparison to a set number of characters.

~$  uniq -w 3 music.txt

uniq -z option

uniq -z option gives a NULL terminated output instead of newline terminated (default option).

~$  uniq -z music.txt

Remove duplicate lines and output to a new file

To send only one copy of each line from list (which is typically sorted) to output file music3.txt

~$  uniq list music3.txt