How to perform multiple Guzzle requests at the same time?
An update related to the new GuzzleHttp guzzlehttp/guzzle
Concurrent/parallel calls are now run through a few different methods including Promises.. Concurrent Requests
The old way of passing a array of RequestInterfaces will not work anymore.
See example here
$newClient = new \GuzzleHttp\Client(['base_uri' => $base]); foreach($documents->documents as $doc){ $params = [ 'language' =>'eng', 'text' => $doc->summary, 'apikey' => $key ]; $requestArr[$doc->reference] = $newClient->getAsync( '/1/api/sync/analyze/v1?' . http_build_query( $params) ); } $time_start = microtime(true); $responses = \GuzzleHttp\Promise\unwrap($requestArr); //$newClient->send( $requestArr ); $time_end = microtime(true); $this->get('logger')->error(' NewsPerf Dev: took ' . ($time_end - $time_start) );
Update:As suggested in comments and asked by @sankalp-tambe, you can also use a different approach to avoid that a set of concurrent request with a failure will not return all the responses.
While the options suggested with Pool is feasible i still prefer promises.
An example with promises is to use settle and and wait methods instead of unwrap.
The difference from the example above would be
$responses = \GuzzleHttp\Promise\settle($requestArr)->wait();
I have created a full example below for reference on how to handle the $responses too.
require __DIR__ . '/vendor/autoload.php';use GuzzleHttp\Client as GuzzleClient;use GuzzleHttp\Promise as GuzzlePromise;$client = new GuzzleClient(['timeout' => 12.0]); // see how i set a timeout$requestPromises = [];$sitesArray = SiteEntity->getAll(); // returns an array with objects that contain a domainforeach ($sitesArray as $site) { $requestPromises[$site->getDomain()] = $client->getAsync('http://' . $site->getDomain());}$results = GuzzlePromise\settle($requestPromises)->wait();foreach ($results as $domain => $result) { $site = $sitesArray[$domain]; $this->logger->info('Crawler FetchHomePages: domain check ' . $domain); if ($result['state'] === 'fulfilled') { $response = $result['value']; if ($response->getStatusCode() == 200) { $site->setHtml($response->getBody()); } else { $site->setHtml($response->getStatusCode()); } } else if ($result['state'] === 'rejected') { // notice that if call fails guzzle returns is as state rejected with a reason. $site->setHtml('ERR: ' . $result['reason']); } else { $site->setHtml('ERR: unknown exception '); $this->logger->err('Crawler FetchHomePages: unknown fetch fail domain: ' . $domain); } $this->entityManager->persist($site); // this is a call to Doctrines entity manager}
This example code was originally posted here.
From the docs:http://guzzle3.readthedocs.org/http-client/client.html#sending-requests-in-parallel
For an easy to use solution that returns a hash of request objects mapping to a response or error, see http://guzzle3.readthedocs.org/batching/batching.html#batching
Short example:
<?php$client->send(array( $client->get('http://www.example.com/foo'), $client->get('http://www.example.com/baz'), $client->get('http://www.example.com/bar')));
Guzzle 6.0 has made sending multiple async requests very easy.
There are multiple ways to do it.
You can create the async requests and add the resultant promises to a single array, and get the result using the settle()
method like this:
$promise1 = $client->getAsync('http://www.example.com/foo1');$promise2 = $client->getAsync('http://www.example.com/foo2');$promises = [$promise1, $promise2];$results = GuzzleHttp\Promise\settle($promises)->wait();
You can now loop through these results and fetch the response using GuzzleHttpPromiseall
or GuzzleHttpPromiseeach
. Refer to this article for further details.
In case if you have an indeterminate number of requests to be sent(say 5 here), you can use GuzzleHttp/Pool::batch()
.Here is an example:
$client = new Client();// Create the requests$requests = function ($total) use($client) { for ($i = 1; $i <= $total; $i++) { yield new Request('GET', 'http://www.example.com/foo' . $i); }};// Use the Pool::batch()$pool_batch = Pool::batch($client, $requests(5));foreach ($pool_batch as $pool => $res) { if ($res instanceof RequestException) { // Do sth continue; } // Do sth}