• Sadhik M S
• Eldo P Elias
• Surekha Mariam Varghese

Semantic duplicates in databases represent today an important data quality challenge which leads to bad decisions. In large databases, sometimes find ourselves with tens of thousands of duplicates, which necessitates an automatic deduplication. Deduplication is a capacity optimization technology that is being used to dramatically improve storage efficiency. For this, it is necessary to detect duplicates, with a fairly reliable method to find as many duplicates as possible and powerful enough to run in a reasonable time. In proposed system, introduce an effective duplicate detection method for automatic deduplication of text files and repeated strings. This will be working with the dataset from WordNet. WordNet is a lexical database for the English language. It groups English words into sets of synonyms called synsets, provides short definitions and usage examples, and records a number of relations among these synonym sets or their members. In the proposed system check the strings or text files that are semantically similar. If they are semantically similar, then remove the string and maintain only one copy of the data. To achieve this, KMP algorithm and Levenshtein Distance method used. These algorithms give better results than those of known methods, with a lesser complexity.

Deduplication, KMP algorithm, Levenshtein Distance, WordNet

"Deduplication in Databases using Pattern Matching", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.5, Issue 3, page no.281-283, March-2018, Available :http://www.jetir.org/papers/JETIR1803054.pdf

"Deduplication in Databases using Pattern Matching", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.5, Issue 3, page no. pp281-283, March-2018, Available at : http://www.jetir.org/papers/JETIR1803054.pdf