Add Semantic Deduplication

[sidjha1]

Add a semantic deduplication operator. The dedup is performed based on semantic similarity via embedding model. Pairs of elements whose similarity exceed threshold are considered duplicates.

Example

data = {
    "Text": [
        "Probability and Random Processes",
        "Optimization Methods in Engineering",
        "Digital Design and Integrated Circuits",
        "Computer Security",
        "I don't know what day it is",
        "I don't know what time it is",
        "Harry potter and the Sorcerer's Stone",
    ]
}
df = pd.DataFrame(data)
df = df.sem_index("Text", "index_dir").sem_dedup("Text", threshold=0.815)
print(df)

Will print

                                    Text
1    Optimization Methods in Engineering
4            I don't know what day it is
6  Harry potter and the Sorcerer's Stone