Remove Duplicate Lines
Deduplicate thousands of lines in one pass, order preserved.
Three different jobs hiding under one name
'Remove duplicates' means one of three things, and they produce different results. Decide which you need before running anything:
| Operation | On a b a c a | Use it when |
|---|---|---|
| Deduplicate, keep first | a b c | You want one copy of each value and care about order |
| Keep only unique lines | b c | You want values that appear exactly once — finding singletons |
| Deduplicate and count | a ×3, b ×1, c ×1 | You are auditing frequency, not cleaning output |
Decide the comparison rules first
- Exact match —
Aandaare different values and both survive. - Case-insensitive —
A,aandAcollapse to whichever appeared first, which is usually what you want for email addresses and domain names. - Trim first — leading and trailing spaces are invisible in a spreadsheet and cause duplicates that look identical on screen. Trim, then compare.
- Ignore blank lines — otherwise every blank line collapses into a single blank line, which is rarely what you meant.
Case-insensitive deduplication is locale-dependent for non-ASCII. In Turkish, the lowercase of I is ı, not i, so a naive fold merges the wrong pairs. For Latin text with accents, Unicode normalisation should be applied before comparing.
Order, memory and very large lists
Order-preserving deduplication needs to remember every line it has already seen, so memory grows with the number of unique lines, not the total. A million-line list with a million distinct values is the worst case; a million-line list with fifty distinct values is trivial. If your input is at that scale, sort first and then collapse adjacent duplicates, which uses almost no memory.
Where this comes up in practice
- Cleaning an email or subscriber export before an import that rejects duplicates.
- Collapsing a log filtered by keyword into one line per distinct error message.
- De-duplicating a keyword research list that merged two sources.
- Removing repeated URLs from a crawl or sitemap export.
How to use it
- Paste one item per line.
- Choose exact, case-insensitive or trimmed comparison.
- Decide whether to keep the first occurrence or keep only lines that appear once.
- Copy the clean list.
Worth knowing
- Output order always matches input order when keeping the first occurrence.
- Memory use scales with the count of unique lines, not total lines.
- Trimming before comparison is what catches spreadsheet-pasted lists.
- Blank-line handling is a separate decision from duplicate handling.
Limitations
- Case-insensitive comparison is locale-sensitive for non-ASCII text.
- The tool does not normalise Unicode forms unless trimming is enabled alongside.
- Fuzzy or near-duplicate matching is not performed — this is exact matching only.
- Extremely large unique line sets are bounded by browser memory.
Frequently asked questions
Does it keep the first or the last occurrence?
How do I remove duplicates ignoring case?
A@Example.com and a@example.com then collapse to whichever appeared first.