Avoid Twitter API limitation with Tweepy

For anyone who stumbles upon this on Google, tweepy 3.2+ has additional parameters for the tweepy.api class, in particular:

• wait_on_rate_limit – Whether or not to automatically wait for rate limits to replenish
• wait_on_rate_limit_notify – Whether or not to print a notification when Tweepy is waiting for rate limits to replenish

Setting these flags to True will delegate the waiting to the API instance, which is good enough for most simple use cases.

The problem is that your try: except: block is in the wrong place. Inserting data into the database will never raise a TweepError - it's iterating over Cursor.items() that will. I would suggest refactoring your code to call the next method of Cursor.items() in an infinite loop. That call should be placed in the try: except: block, as it can raise an error.

Here's (roughly) what the code should look like:

# above omitted for brevityc = tweepy.Cursor(api.search,                       q=search,                       include_entities=True).items()while True:    try:        tweet = c.next()        # Insert into db    except tweepy.TweepError:        time.sleep(60 * 15)        continue    except StopIteration:        break

This works because when Tweepy raises a TweepError, it hasn't updated any of the cursor data. The next time it makes the request, it will use the same parameters as the request which triggered the rate limit, effectively repeating it until it goes though.

Just replace

api = tweepy.API(auth)

with

api = tweepy.API(auth, wait_on_rate_limit=True)