Skip to main content
ToolNestora

Remove Duplicate Lines

Paste a list with one item per line and get back the same list with repeated lines removed. You decide whether upper and lower case count as different, whether spaces at the ends of lines matter, what happens to blank lines, which copy of a repeated line stays and whether the result is sorted. The tool also shows how many lines were removed and which were repeated.

Remove duplicate lines from a list

Turn off to treat “Apple” and “apple” as the same line.

Also removes those spaces from the result.

Blank lines
Keep which copy
Order
Lines in
0
Lines kept
0
Duplicates removed
0

How to use it

  1. Paste your list

    Put one item on each line. Email addresses, URLs, keywords, names and log lines all work.

  2. Choose how lines are compared

    Turn off “Match upper and lower case exactly” to treat Apple and apple as the same line, and turn on the space option to ignore spaces at the start and end of lines.

  3. Decide about blank lines and order

    Leave blank lines alone, keep a single blank line, or remove them all. Keep the first or last copy of repeated lines, and keep the original order or sort A to Z or Z to A.

  4. Copy the result

    The cleaned list appears below with the counts of lines read, kept and removed. Copy it or download it as a text file.

Example: cleaning a keyword list

A list of 1,200 keywords exported from two sources has many overlaps and different capitalisation. Turn off case matching, ignore spaces at the ends of lines, and choose “Remove all” for blank lines. The result keeps the first spelling of each keyword and the “Lines that were repeated” list shows which entries were doubled up.

Limits and limitations

  • Only whole lines are compared. Two lines that differ by even one character, or by internal spaces, are treated as different.
  • Case-insensitive matching uses simple lower-casing, so it does not treat different spellings of the same word in other alphabets as equal.
  • If you keep the last copy, lines appear in the position of that last copy.
  • Sorting uses natural order for numbers (2 before 10) and ignores case, which can differ from a spreadsheet's order.
  • The line-break style (Windows or Unix) of the input is kept in the output.

Frequently asked questions

Does it change the order of my lines?

Not unless you choose to sort. By default the first copy of each line keeps its place and later copies are removed.

How are blank lines treated?

By default they are left alone, so spacing between groups survives. You can keep one blank line or remove all of them. A line containing only spaces counts as blank.

How many lines can it handle?

It has been tested in automated tests with 200,000 lines. Much larger lists depend on your device's memory and may make the page slow.

Is my list uploaded?

No. Your list is processed by JavaScript running in your browser and is not sent to a server. The page itself still loads from a web server like any website. This site can load advertising or analytics scripts, but only after you accept them in the cookie notice. They are separate from the tools, which never pass the files or text you work with to them.

More tools