Remove Duplicate Lines

Clean repeated lines.

Loading tool…

What counts as a duplicate

Removing duplicates means keeping the first occurrence of each distinct line and discarding the rest. The interesting question is what makes two lines identical. Trailing whitespace is invisible but makes lines differ. So does case, so Admin and admin are two entries unless you say otherwise. So do different line endings when a file has passed between Windows and Unix systems.

Those invisible differences are why a deduplication that appears to have failed usually has not — the lines really are different, just not in ways you can see. Trimming whitespace and normalising case before comparing catches almost all of it. It is also worth deciding whether you want deduplication to preserve the original order, which is usually what people expect, or to sort as it goes.

Why we built this tool

Merged mailing lists, exported spreadsheets, concatenated log files, and combined keyword lists all arrive with repetition. So does any list assembled by hand over time.

Those lists are typically email addresses, customer records, or URLs — personal data in most jurisdictions, and precisely what you should not paste into an unfamiliar website. Deduplication is a hash-set operation that takes microseconds, so there is no technical reason for it to involve a server. Ours runs entirely in your browser: nothing transmitted, nothing retained, no line limit, and it keeps working with the network off.

Tips and common mistakes

  • Trim whitespace and normalise case before deduplicating, or near-identical entries survive.
  • Preserve the original order unless you specifically want the list sorted.
  • Check the removal count against your expectation — a suspiciously low number usually means invisible differences.
  • Deduplicating email lists before a send avoids messaging the same person twice.

Open Remove Duplicate Lines with a link

Assistants and scripts can open Remove Duplicate Lines with its input already in the link, after the #, which never leaves your device:

  • #text= — the text to work on

Example: https://mygadgets.ink/tools/remove-duplicate-lines/#text=... · also compressed gz fields and ?url=. How tool links work

Quick start: using Remove Duplicate Lines

  1. Paste your list into the input above.
  2. Decide whether comparison should ignore case and surrounding whitespace.
  3. Choose whether to preserve the original order or sort the output.
  4. Run the deduplication and check how many lines were removed.
  5. Copy the cleaned list.

Order the result with Sort Lines, or tidy stray whitespace first using Remove Extra Spaces.

What de-duplication does

The tool scans your list line by line and removes any line that has already appeared, keeping only the first occurrence of each unique line. A list of 500 entries with 80 repeats becomes a clean list of 420 uniques. It is one of those quiet tasks that is tedious by hand and instant with a tool.

The settings that decide what counts as a duplicate

"Duplicate" is less obvious than it sounds, and these options change the outcome:

  • Case sensitivity. Is Apple the same as apple? Case-sensitive treats them as different; case-insensitive treats them as the same. For most real lists — emails, names — case-insensitive is what you want, because Bob@x.com and bob@x.com are the same person.
  • Whitespace. A line with a trailing space is technically different from one without. Trimming whitespace before de-duplicating catches "invisible" duplicates that otherwise slip through — clean with Remove Extra Spaces first.
  • Blank lines. Decide whether empty lines should collapse to one or be removed entirely.

De-duplicate, then sort (or the reverse)

De-duplication and sorting are natural partners. A common workflow:

  1. Trim whitespace so near-identical lines match.
  2. Normalize case with Convert Case if case variants should count as the same.
  3. Remove duplicates here.
  4. Sort the clean list with Sort Lines for a tidy, ordered, unique result.

Common uses

  • Clean a mailing list of repeated addresses.
  • De-duplicate merged data from several sources.
  • Collapse repeated log lines.
  • Tidy a keyword or tag list.

Related Text Tools

Browse all Text Tools →