How do I remove duplicate lines in a text file?

Open the file, copy its text, paste it into the box above, then copy the result or download it as a .txt file. Nothing is uploaded, so it is safe for lists of emails, names or other private data. In a terminal, “sort -u file.txt” removes duplicates but sorts the list; “awk '!seen[$0]++' file.txt” keeps the original order.

Does it keep the first or the last duplicate?

The first by default, so the list keeps the order in which each line first appeared. Choose “Keep last occurrence” to keep the final copy instead, in the position where it last appears.

Is the comparison case-sensitive?

By default yes, so “Banana” and “banana” are different lines. Turn on “Ignore case” to treat them as the same; the line you keep is output exactly as you typed it. Case folding does not depend on your language setting, so a capital dotted İ is never matched with I, while “ß” matches “SS” and “ss”.

What happens to blank lines and spaces?

With “Trim spaces” on (the default), spaces and tabs at the start and end of a line are ignored when comparing and are removed from the result. Blank lines (empty or only spaces) can be treated like any other line, so only one remains; ignored, so all of them stay where they are; or removed entirely.

Does it work with Hebrew, Arabic, Chinese or accented text?

Yes. Lines are compared as Unicode text, and with “Compare Unicode forms as equal” on (the default) an accented letter typed as one character matches the same letter typed as a base letter plus a combining accent. The boxes detect text direction per line, so right-to-left text such as Hebrew and Arabic reads correctly, and sorting uses your browser’s dictionary rules for its language.

What is natural sort order?

Natural order compares runs of digits as numbers instead of letter by letter, so item2 comes before item10 (plain A to Z puts item10 first because “1” sorts before “2”). Numbers with leading zeros count by value, so item02 and item2 are tied and then ordered by their text.

Will it handle a very long list?

Yes. Lists of 100,000 lines are processed in a fraction of a second on an ordinary computer; for very large pastes the result updates a moment after you stop typing so the page stays responsive.

Are Windows and Mac line endings kept?

The result uses the same line ending as your text: Windows (CRLF) if your first line break is CRLF, otherwise a plain line feed. A final line break is kept only if your text had one.

Is my list saved or sent anywhere?

No. Everything runs in your browser and the text is never uploaded, stored or logged. It is gone when you close or reload the page. The only thing the site records, if you allow analytics, is that a button was used, never any of the text.

Remove Duplicate Lines

Paste a list and every repeated line is removed at once, keeping the first copy of each (or the last, if you prefer). The example list has 8 lines; with the default settings 3 duplicates go and 5 lines remain: apple, banana, cherry, Banana, date. Turn on “Ignore case” and “Banana” joins “banana”, leaving 4. You can also drop blank lines, list only the lines that repeat, and sort what is left A to Z, naturally (item2 before item10), by length or at random. Your list never leaves the browser.

Updated 2026-10-03Runs in your browser: nothing you enter is uploadedHow this tool works

Your listStays in your browser

Replace the example: paste or type one item per line. The result updates as you type.

Your list is processed on this device and never uploaded. It is not saved either: reload the page and it is gone.

Options

Ignore caseTrim spacesOnly duplicated linesCompare Unicode forms as equal

Blank linesWhich copy to keepSort

Result

8 lines in, 5 lines out, 3 duplicates removed.

apple
banana
cherry
Banana
date

Counts

Lines in: 8
Lines out: 5
Unique lines: 5
Duplicates removed: 3

Most repeated lines

apple3×
cherry2×

What each option does

The same messy list, “pear, Pear, apple, pear , (blank), apple, (blank), fig”, with one option changed at a time.
Option	Result	Effect
Default (trim on, case-sensitive)	pear, Pear, apple, (blank), fig	Spaces around a line are ignored; “pear” and “Pear” stay separate.
Ignore case	pear, apple, (blank), fig	“Pear” counts as “pear”.
Keep the last copy	Pear, pear, apple, (blank), fig	Each line is output where it appears last.
Remove blank lines	pear, Pear, apple, fig	Blank and whitespace-only lines are dropped.
Only duplicated lines	pear, apple, (blank)	Each repeated line once; lines that appear once are left out.
Do not trim	pear, Pear, apple, pear , (blank), fig	“pear” and “pear ” are different lines.

The sort orders

Sorting “item10, Item2, item1, item02, ábaco, abc”.
Sort	Result	How it works
A to Z	ábaco, abc, item02, item1, item10, Item2	Dictionary order for your browser’s language: accents sort with their letter, and a lowercase letter comes just before its capital.
Z to A	Item2, item10, item1, item02, abc, ábaco	The same order, backwards.
Natural	ábaco, abc, item1, item02, Item2, item10	Digit runs are compared as numbers, so item2 comes before item10. Leading zeros do not change the value; ties fall back to plain text order.
Shortest first	abc, Item2, item1, ábaco, item10, item02	By number of characters; lines of equal length keep their order.
Reverse	abc, ábaco, item02, item1, Item2, item10	Flips the list upside down.
Shuffle	different every time	A random order from your browser’s cryptographic random number generator (a Fisher–Yates shuffle).

How to remove duplicate lines from a list

Paste your list. Replace the example with your own text, one item per line. The result appears as you type.
Choose what counts as a duplicate. Turn on “Ignore case” to treat Apple and apple as the same, leave “Trim spaces” on to ignore spaces at the ends of lines, and pick whether blank lines are kept, treated as one line or removed.
Pick which copy to keep. Keep the first occurrence (the default) or the last. “Only duplicated lines” instead lists just the lines that appear more than once.
Sort if you like. Leave the original order, or sort A to Z, Z to A, naturally, by length, reversed or shuffled.
Copy or download. Copy the result to the clipboard or download it as a .txt file. The counts and the most repeated lines are shown beside it.

Frequently asked questions

How do I remove duplicate lines in a text file?: Open the file, copy its text, paste it into the box above, then copy the result or download it as a .txt file. Nothing is uploaded, so it is safe for lists of emails, names or other private data. In a terminal, “sort -u file.txt” removes duplicates but sorts the list; “awk '!seen[$0]++' file.txt” keeps the original order.
Does it keep the first or the last duplicate?: The first by default, so the list keeps the order in which each line first appeared. Choose “Keep last occurrence” to keep the final copy instead, in the position where it last appears.
Is the comparison case-sensitive?: By default yes, so “Banana” and “banana” are different lines. Turn on “Ignore case” to treat them as the same; the line you keep is output exactly as you typed it. Case folding does not depend on your language setting, so a capital dotted İ is never matched with I, while “ß” matches “SS” and “ss”.
What happens to blank lines and spaces?: With “Trim spaces” on (the default), spaces and tabs at the start and end of a line are ignored when comparing and are removed from the result. Blank lines (empty or only spaces) can be treated like any other line, so only one remains; ignored, so all of them stay where they are; or removed entirely.
Does it work with Hebrew, Arabic, Chinese or accented text?: Yes. Lines are compared as Unicode text, and with “Compare Unicode forms as equal” on (the default) an accented letter typed as one character matches the same letter typed as a base letter plus a combining accent. The boxes detect text direction per line, so right-to-left text such as Hebrew and Arabic reads correctly, and sorting uses your browser’s dictionary rules for its language.
What is natural sort order?: Natural order compares runs of digits as numbers instead of letter by letter, so item2 comes before item10 (plain A to Z puts item10 first because “1” sorts before “2”). Numbers with leading zeros count by value, so item02 and item2 are tied and then ordered by their text.
Will it handle a very long list?: Yes. Lists of 100,000 lines are processed in a fraction of a second on an ordinary computer; for very large pastes the result updates a moment after you stop typing so the page stays responsive.
Are Windows and Mac line endings kept?: The result uses the same line ending as your text: Windows (CRLF) if your first line break is CRLF, otherwise a plain line feed. A final line break is kept only if your text had one.
Is my list saved or sent anywhere?: No. Everything runs in your browser and the text is never uploaded, stored or logged. It is gone when you close or reload the page. The only thing the site records, if you allow analytics, is that a button was used, never any of the text.

How it works

Text is split into lines on LF, CRLF or CR. Each line gets a comparison key: the line itself, trimmed if “Trim spaces” is on, case-folded if “Ignore case” is on (upper-cased and then lower-cased, which is independent of your language and matches “ß” with “SS”), then normalized to Unicode NFC if that option is on. Lines with equal keys are duplicates, found in one pass with a hash map, so the time grows in step with the length of the list. The kept line is output as typed (trimmed if trimming is on). “Duplicates removed” counts every extra copy, so a line that appears three times adds two. A blank line is one that is empty or has only whitespace. Sorting runs after duplicates are removed and uses the browser’s Intl.Collator for your language; natural order turns on its numeric option, and shuffle uses crypto.getRandomValues with rejection sampling so every order is equally likely.

Sources

All tools Home & DIY calculators