data-deduplication

Data deduplication engine, supporting optional compression and public key encryption.

Official Repository of "LLM × DATA" Survey Paper

🚢 Data Toolkit for Sailor Language Models

Self-contained C# library for data deduplication using Sqlite

Fast and efficient content-defined chunking for data deduplication. Java implementation of FastCDC as library.

Optimal distributed data deduplication and supervised learning pipeline using Apache Spark

A JAVA project that splits data using hashing techniques and removes duplicate blocks to save cloud storage. This project also uses the CloudSim framework for cloud storage simulation.

Sievio turns GitHub, local repos, and web PDFs into clean JSONL for LLM pretraining, fine-tuning, and RAG. It offers structure-aware chunking, reliable Unicode decoding, pluggable QC and safety checks, plus optional dataset cards and deduplication.

PolyDeDupe: Multi-Lingual Data Deduplication

A pure-JS, content-addressed, copy-on-write virtual filesystem for the browser, featuring: deduplication, filesystem universes (snapshots), events, and optional asynchronous sync.

Enterprise-grade SaaS platform for importing, cleaning, and managing large-scale mailing lists with advanced deduplication and enrichment.

Entify is an early open-source workspace for entity resolution, record linkage, and data deduplication. It helps teams profile messy datasets, configure Splink-powered matching workflows, and review explainable match clusters.

Production-oriented file deduplication and data-protection platform with tenant isolation, SHA-256 exact deduplication, MinHash/LSH and dHash near-duplicate detection, DLP, AES-256-GCM encryption, RBAC, and audit logging.

Offline CSV encoding, duplicate and formula-risk checkers, plus a $9 local cleanup tool and fixed-scope services.

JSON + CSV to PostgreSQL pipeline with three-level deduplication, validation and idempotent re-runs, three analytical queries, and a /companies page on Next.js App Router

Fellow is a package for creating people that can be unified by their shared values via a singleton list on the class

Этот проект представляет собой мощный инструмент для поиска и анализа дублирующихся файлов в указанной директории. Программа позволяет эффективно выявлять одинаковые файлы на основе их содержимого, используя алгоритм хеширования SHA-256. Она поддерживает настройку параметров, таких как минимальный размер файла для проверки и игнорирование определен

Scholarship Finder is a full-stack web application that collects, organizes, and displays government scholarship opportunities through web scraping, data normalization, deduplication, and validation. Built with React, TypeScript, FastAPI, PostgreSQL, Scrapy, and Docker, it enables students to search, filter, and explore scholarship details, eligibi