Duplicate Line Remover

A Python tool with a Flask web interface designed to remove duplicate entries from line break delimited lists while maintaining the relative position of blank lines in the input text. Originally created to clean up OneTab exports (maintaining tab groups), it can be used for any text list with similar requirements.
Explore the docs »

· Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Contributing
5. License
6. Contact
7. Acknowledgments

About The Project

Built With

• 
• 

Getting Started

This guide provides instructions on setting up and using the "Duplicate Line Remover" tool locally.

Prerequisites

To run this project, you'll need:

• Python (version 3.11.0 recommended)
• Flask (for the web application)

Note: I've added .python-version to .gitignore to ensure compatibility across different Python versions. For reference, the version used during development was 3.11.0.

You may want to set up a virtual environment (venv) to manage dependencies, for exmaple:

python -m venv env

source env/bin/activate

Install Flask:

pip install flask

Installation

Clone the repo:

git clone https://github.com/itsmejayd/filter-duplicate-lines.git

cd filter-duplicate-lines

Usage

1. Removing Duplicate Lines:

• Use remove_duplicates(input_file, output_file) function from filter_duplicate_lines.py to process your input file and generate a cleaned output file without duplicate lines.

Example usage:

from filter_duplicate_lines import remove_duplicates

input_file = "your_input_file_name.txt"
output_file = "cleaned_output.txt"

remove_duplicates(input_file, output_file)

2. Using the Web Application:

• Start the Flask application by running python app.py in your terminal.
• Access the application in your web browser at http://localhost:5000.
• Upload a text file or paste text into the text box, then click download button to remove duplicates and download the cleaned file.

Contributing

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

See the open issues for a full list of proposed features (and known issues).

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Joseph Dwyer - [email protected] - Twitter

Acknowledgments

• OneTab Browser Extension: A wonderful tool I love, inspired me to make my tool specifically to increase OneTab's functionality and to share it for others to use.

(back to top)