Jaccard Similarity for Texts in a pandas DataFrame
One way to speed up the process could be parallel processing using Pandas on Ray.
You can try NLTK implementation of jaccard_distance for jaccard similarity. I couldn't find any significant improvement in processing time though(for calculating similarity), may work out better on a larger dataset.
Tried comparing NLTK implementation to your custom jaccard similarity function (on 200 text samples of average length 4 words/tokens)
NTLK jaccard_distance:
CPU times: user 3.3 s, sys: 30.3 ms, total: 3.34 sWall time: 3.38 sCustom jaccard similarity implementation:
CPU times: user 3.67 s, sys: 19.2 ms, total: 3.69 sWall time: 3.71 s