🧹 Remove Duplicate Lines

Remove duplicate lines from your text while preserving order. Keep only the first occurrence of each unique line. Perfect for cleaning up lists, data sets, and removing repetitive entries.

Enter Your Text (one item per line):

Ignore Case

📌 Unique Lines:

💡 How it works:

Paste your text with one item per line
Click "Remove Duplicates" to process
First occurrence of each line is kept
Order is preserved from original text
Enable "Ignore Case" to treat "Apple" and "apple" as duplicates
Statistics show duplicates removed

📝 Examples:

Input (With Duplicates):

apple banana apple cherry banana date apple

Output (Duplicates Removed):

apple banana cherry date

Statistics:

Original Lines: 7
Unique Lines: 4
Duplicates Removed: 3

Input (Case-Sensitive - Without Ignore Case):

Hello hello HELLO World

Output (Case-Sensitive - Treats as Different):

Hello hello HELLO World

Output (Case-Insensitive - With Ignore Case):

Hello World

📚 Use Cases:

Clean up email lists or contact databases
Remove duplicate entries from data exports
Organize lists and eliminate repetitive items
Prepare data for analysis and reporting
Deduplicate log files or transaction records
Clean up word lists or dictionaries
Remove redundant entries from merged datasets

ℹ️ Note: This tool preserves the order of first occurrence. Duplicates are identified based on exact line matching (or case-insensitive if that option is enabled).

First occurrence wins

Lines are walked from top to bottom against a set of everything seen so far. A line whose key is already in the set is counted and dropped; anything new is kept in the position it already had:

key = ignoreCase ? line.toLowerCase() : line
seen.has(key) ? drop it : keep the line and remember the key

So the copy that survives is always the earliest one, with its original spelling. Feed it Hello, hello, HELLO with Ignore Case ticked and you get a single Hello - the later capitalisations vanish rather than overwrite it. There is no keep-the-last option; reverse your list before pasting if the last entry is the one that matters.

Matching is exact, character for character

Invisible whitespace defeats it. apple and apple with a trailing space are two different lines. Run the text through Remove Extra Spaces first if it came from a spreadsheet or a PDF.
Blank lines are lines. The first empty line is kept and every later one is removed as a duplicate, which quietly closes up the gaps between paragraphs.
Ignore Case is plain lowercasing. Fine for ASCII lists and email addresses; it will not equate the German Straße with STRASSE, or a curly apostrophe with a straight one.
Nothing is sorted. Order is left exactly as pasted - send the result through Sort Lines if you want it alphabetical as well as unique.

Careful with Ignore Case on email lists. Domains are case-insensitive but the part before the @ is not, so J.Smith@example.com and j.smith@example.com are in principle two different mailboxes.

Frequently asked questions

Does this keep the first or the last copy of a duplicate?

The first. Everything after it is discarded and the surviving line stays put, so a chronological list keeps its earliest entries.

Why are two lines that look identical both still there?

Something invisible differs - a trailing space, a tab, a non-breaking space from Word, or a curly quote where the other line has a straight one. Comparison is character by character, so any of those makes the lines distinct.

How do I remove duplicates in Excel instead?

Select the column and use Data, Remove Duplicates, or write =UNIQUE(A:A) in a spare cell. Pasting a column here is quicker for a one-off and leaves the workbook untouched.