In what way is grequests asynchronous? In what way is grequests asynchronous? python python

In what way is grequests asynchronous?


.map() is meant to run retrieval of several URLs in parallel, and will indeed wait for these tasks to complete (gevent.joinall(jobs)) is called).

Use .send() instead to spawn jobs, using a Pool instance:

req = grequests.get('http://www.codehenge.net/blog', hooks=dict(response=print_res))job = grequests.send(req, grequests.Pool(1))for i in range(10):    print i

Without the pool the .send() call will block still, but only for the gevent.spawn() call it executes.


If you don't want to use grequests you can just implement requests with callbacks using requests + the threading module from the standard library. It's actually really easy, and if all you want to do is send requests with callbacks the API is nicer than the one provided by grequests.

from threading import Threadfrom requests import get, post, put, patch, delete, options, headrequest_methods = {    'get': get,    'post': post,    'put': put,    'patch': patch,    'delete': delete,    'options': options,    'head': head,}def async_request(method, *args, callback=None, timeout=15, **kwargs):    """Makes request on a different thread, and optionally passes response to a    `callback` function when request returns.    """    method = request_methods[method.lower()]    if callback:        def callback_with_args(response, *args, **kwargs):            callback(response)        kwargs['hooks'] = {'response': callback_with_args}    kwargs['timeout'] = timeout    thread = Thread(target=method, args=args, kwargs=kwargs)    thread.start()

You can verify that it works like an AJAX calls in JS: you send a request on another thread, do some stuff on the main thread, and when the request returns you invoke a callback. This callback just prints out the response content.

async_request('get', 'http://httpbin.org/anything', callback=lambda r: print(r.json()))for i in range(10):    print(i)


Create a list of requests and then send them with .imap:

event_list = [grequests.get(url_viol_card, params={"viol": i},              session=session) for i in print_ev_list]for r in grequests.imap(event_list, size=5):    print(r.request.url)
  • session is requests.Session() object (optional)
  • size=5 send 5 requests simultaneously: as soon as one of them is completed the next one is sent
  • In this example when any request (unordered) is completed its url is printed