Applying Spacy Parser to Pandas DataFrame w/ Multiprocessing

Spacy is highly optimised and does the multiprocessing for you. As a result, I think your best bet is to take the data out of the Dataframe and pass it to the Spacy pipeline as a list rather than trying to use .apply directly.

You then need to the collate the results of the parse, and put this back into the Dataframe.

So, in your example, you could use something like:

tokens = []lemma = []pos = []for doc in nlp.pipe(df['species'].astype('unicode').values, batch_size=50,                        n_threads=3):    if doc.is_parsed:        tokens.append([n.text for n in doc])        lemma.append([n.lemma_ for n in doc])        pos.append([n.pos_ for n in doc])    else:        # We want to make sure that the lists of parsed results have the        # same number of entries of the original Dataframe, so add some blanks in case the parse fails        tokens.append(None)        lemma.append(None)        pos.append(None)df['species_tokens'] = tokensdf['species_lemma'] = lemmadf['species_pos'] = pos

This approach will work fine on small datasets, but it eats up your memory, so not great if you want to process huge amounts of text.