Used with its -u option the sort command removes duplicate lines from a file. For example:

   sort -u -o mail_labels addresses

This removes all duplicate lines from the file addresses and places the output in the file mail_labels.

Duplicate lines do not have to be consecutive for the sort command to identify and remove them.