How to remove duplicate lines from a list without losing order

A mailing list, an export of IDs from two tables, task lists merged together — duplicates creep in unnoticed, and the consequences can sting: an email sent twice or a doubled entry in a report. Cleaning a list takes seconds once you know three nuances.

Nuance one: case

Are “Ivanov@mail.ru” and “ivanov@mail.ru” one address or two? For email — one; for passwords and product codes — definitely two. That's why the tool has a case-sensitivity switch: turn it off for addresses and domains, on for codes and passwords.

Nuance two: invisible spaces

A space or tab at the end of a line (everywhere after copying from Excel) makes the line “different”, and the duplicate survives the cleanup. The right order of operations: trim the edges first, then remove duplicates. The duplicate remover has a dedicated trim checkbox that does this automatically before comparing.

Nuance three: line order

The quick fix — piping a list through sort | uniq on the command line — re-sorts everything. That often breaks the task: mailing order, log chronology, task priorities. That's why the online tool preserves order by default: second and later occurrences are removed, and the first stays where it was.

After cleanup, the statistics are worth a glance: how many lines there were and how many remain. The difference is exactly the number of duplicates; if it's suspiciously large (half the list, say), look for the source of the repeats instead of just mopping up. Counting the length and makeup of a list is easy in the word counter, and finding the overlaps between two lists — in the list comparison tool.