Python NLTK: Bigrams trigrams fourgrams

If you apply some set theory (if I'm interpreting your question correctly), you'll see that the trigrams you want are simply elements [2:5], [4:7], [6:8], etc. of the token list.

You could generate them like this:

>>> new_trigrams = []>>> c = 2>>> while c < len(token) - 2:...     new_trigrams.append((token[c], token[c+1], token[c+2]))...     c += 2>>> print new_trigrams[('are', 'you', '?'), ('?', 'i', 'am'), ('am', 'fine', 'and')]

Try everygrams:

from nltk import everygramslist(everygrams('hello', 1, 5))

[out]:

[('h',), ('e',), ('l',), ('l',), ('o',), ('h', 'e'), ('e', 'l'), ('l', 'l'), ('l', 'o'), ('h', 'e', 'l'), ('e', 'l', 'l'), ('l', 'l', 'o'), ('h', 'e', 'l', 'l'), ('e', 'l', 'l', 'o'), ('h', 'e', 'l', 'l', 'o')]

Word tokens:

from nltk import everygramslist(everygrams('hello word is a fun program'.split(), 1, 5))

[out]:

[('hello',), ('word',), ('is',), ('a',), ('fun',), ('program',), ('hello', 'word'), ('word', 'is'), ('is', 'a'), ('a', 'fun'), ('fun', 'program'), ('hello', 'word', 'is'), ('word', 'is', 'a'), ('is', 'a', 'fun'), ('a', 'fun', 'program'), ('hello', 'word', 'is', 'a'), ('word', 'is', 'a', 'fun'), ('is', 'a', 'fun', 'program'), ('hello', 'word', 'is', 'a', 'fun'), ('word', 'is', 'a', 'fun', 'program')]

I do it like this:

def words_to_ngrams(words, n, sep=" "):    return [sep.join(words[i:i+n]) for i in range(len(words)-n+1)]

This takes a list of words as input and returns a list of ngrams (for given n), separated by sep (in this case a space).