Paste a list and get one copy of each line. Keep the first or the last copy, choose whether capitalization and edge spaces count, drop blank lines, or sort the result.
This tool runs entirely in your browser. What you enter is never sent anywhere.Your input is never sent anywhere.
DetailsOptions
How duplicate lines are found and which copy is kept
The tool reads your text one line at a time and checks each line against the lines before it. By default the first copy of a line is kept where it is, and every later copy is removed, wherever it sits in the list. Nothing has to be sorted first. Switch Keep to Last copy if the newest entry is the one you want.
Two lines count as the same only if they match exactly. You can loosen that with two options: ignore spaces around lines, and turn off Match case so that Apple and apple are one line. Blank lines can be dropped, and the result can be sorted A → Z or Z → A. The result updates as you type.
Lists that collect repeats
Email and contact lists from several sources
When you merge lists, the same address shows up more than once. Dedupe before you send so no one gets the same message twice.
Keyword and URL lists
Exports from SEO or ad tools often repeat entries. One copy of each keyword or URL gives you a list you can count and filter.
Log lines and IP addresses
If each line holds only the value you care about, such as an IP address, you get the unique values. Lines with a timestamp each count as different.
Config files and code lists
Repeated class names, keys, or import lines are easy to miss by eye. Paste the list in and see how many repeats there were.
How to dedupe a list in 3 steps
- Paste your list into the Input list box, with one item on each line.
- Set the options: Remove blank lines, Ignore spaces around lines, Match case, Keep, and Sort result.
- The result appears as you type. Check the Original, Removed, and Remaining counts, then click Copy result.
Note: There is no run button. The result updates every time you change the text or an option.
What counts as a duplicate
- Exact match: two lines are the same only if every character matches. A single extra space makes a line different unless Ignore spaces around lines is on.
- First or last copy: by default the first occurrence stays and later ones are removed. With Keep set to Last copy, the last occurrence stays at its own position and the earlier ones are removed.
- Match case: on by default. With it off, lines are compared in lowercase, and the line you get back keeps the capitalization of its first copy.
- Ignore spaces around lines: spaces and tabs at both ends of each line are ignored when lines are compared. The result keeps the original text of each line, so indentation and trailing spaces are not removed.
- Sort: digits inside a line are compared by value, so 2 comes before 10. The sort runs after duplicates are removed.
- Counts: Removed is the number of lines in minus the number of lines out, so it includes blank lines that were dropped.
Common lists and what to set for each
A starting point for the options. Change them to match your data.
| List | Suggested options | Why |
|---|---|---|
| Email addresses | Ignore spaces on, Match case off | The same address is often typed with different capitalization or a stray space. |
| Keywords | Ignore spaces on, Match case off | Keywords that differ only by capitalization count once. |
| URLs | Ignore spaces on, Match case on | Paths are case-sensitive on many servers, so keep the case. |
| Log lines or IP addresses | Ignore spaces on | Lines must match exactly, so a line with a different timestamp is a different line. |
| Code names and config keys | Match case on, Ignore spaces off | Names are usually case-sensitive, and indentation can matter in config files. |
| Rows where the newest entry should win | Keep: Last copy | The last copy of each line stays, so a later update replaces an earlier one. |
| An alphabetical list of unique items | Sort result: A → Z | Dedupes first, then sorts the remaining lines. |
| A list where the order matters | Sort result: Keep order | Each kept line stays at its own position in the list. |
Duplicates anywhere in the list
The tool removes a repeat whether it sits right below the first copy or a thousand lines later, and it does not need the list to be sorted.
Your data stays in your browser
The tool runs as JavaScript on the page and has no upload step, so customer lists and logs are not sent to a server.
Deduplicating without sort | uniq or a spreadsheet
How this compares with other ways to remove repeated lines, and what to check when repeats survive.
Why uniq alone leaves repeats behind
The Unix uniq command removes only repeated lines that sit next to each other. That is why it is usually run after sort, and sorting changes the order of your list.
This tool compares each line with all the lines before it, so it removes repeats anywhere in the list and leaves the order alone unless you pick a sort option.
Near-duplicates: spaces, capitalization, and hidden characters
Lines that look the same can differ by a trailing space, a capital letter, or a character you cannot see. Ignore spaces around lines and Match case deal with the first two.
A double space inside a line, a different punctuation mark, or a hidden character such as a zero-width space still makes lines different. If repeats survive, check for those.
First copy or last copy
Which copy to keep depends on your list. For a list of unique values, the first copy is usually fine. If the list is in time order and a later line replaces an earlier one, such as an updated row or a newer entry, choose Last copy.
Either way, the kept line stays at its own position in the list. It is not moved to the top or the bottom.
Why duplicates survive, and other surprises
A trailing space keeps two lines apart
Two entries look the same on screen, but one ends with a space, so the tool treats them as different lines and both stay in the result.
💡 See the solution▼
Apple and apple both stay
Match case is on by default, so lines that differ only in capitalization are treated as different lines.
💡 See the solution▼
The wrong copy was kept
The tool keeps the first copy by default. If the later line is the updated one, such as a newer row in an export, the older copy ends up in the result.
💡 See the solution▼
Lines that are only partly the same are not removed
Two log lines with the same message but different timestamps are different lines, so both stay.
💡 See the solution▼
Duplicate line remover FAQ
Q.Which copy does it keep, and does the order change?
A. By default it keeps the first copy of each line and removes the later ones. Under Keep, choose Last copy to keep the final one instead. The kept line stays at its own position in the list, so the order changes only if you choose A → Z or Z → A under Sort result. The sort is applied after the duplicates are removed.
Q.Can it treat Apple and apple as the same line?
A. Yes. Turn off Match case. The tool then compares lowercase versions of the lines. The line you get back is the copy you chose to keep, with its own capitalization. With Match case on, which is the default, Apple and apple are two different lines and both stay.
Q.Why are lines that look the same still in the result?
A. The tool compares whole lines character by character, so a difference you cannot see still makes two lines different. Spaces at the start or end of a line are the usual cause, and Ignore spaces around lines (on by default) takes care of those. A double space inside a line, a different punctuation mark, or a hidden character such as a zero-width space also makes lines different.
Q.Does Ignore spaces around lines change my text?
A. No. The option only affects the comparison. The result keeps the original text of each line it keeps, including any indentation or trailing spaces. For example, if apple and apple followed by a space are both in the list, they count as one line, and you get back the copy you chose to keep exactly as you typed it.
Q.What happens to blank lines?
A. A blank line is an empty line or a line with only spaces or tabs. With Remove blank lines on, which is the default, they are dropped. With it off, blank lines are compared like any other line: if Ignore spaces around lines is on, all blank lines count as one and a single blank line is kept. Removed blank lines are included in the Removed count.
Q.How does the sort work?
A. A → Z and Z → A sort the remaining lines using the browser's English collation. Digits inside a line are compared by their value, so 2 comes before 10 and item2 comes before item10. The sort runs after the duplicates are removed. If you leave the sort on Keep order, the lines stay in the order of the copies you kept.
Q.Does it compare whole lines or the words inside them?
A. Whole lines only. It does not remove repeated words inside a line, and it does not show you which lines were duplicated. Lines that are only partly the same, such as the same address with a different timestamp, count as different lines.
Q.Is there a limit on how many lines I can paste?
A. The tool sets no line limit. It runs in your browser and uses your device's memory. The result is recalculated every time you change the text or an option, so a very large list can feel slower on an older device. If the page slows down, paste the list in smaller parts.
Q.Is my list sent to a server?
A. No. The tool processes your text with JavaScript in your browser, and the page has no upload step. Your text is held only in the page while it is open, and closing or reloading the page clears it.
Your feedback
makes this tool better.
Tell us which features you'd like, what should be fixed, or which new tool you'd like to see. An engineer reads every submission and uses it to guide development.