Making multiple HTTP requests asynchronously

I just saw this, a year and a bit later, but hopefully not too late for some googler...

Typhoeus by far the best solution for this. It wraps libcurl in a really elegant fashion. You can set the max_concurrency up to about 200 without it choking.

With respect to timeouts, if you pass Typhoeus a :timeout flag, it will just register a timeout as the response... and then you can even put the request back in another hydra to try again if you like.

Here's your program rewritten with Typhoeus. Hopefully this helps anybody who comes across this page later!

require 'typhoeus'urls = [  'http://www.google.com/',  'http://www.yandex.ru/',  'http://www.baidu.com/']hydra = Typhoeus::Hydra.newsuccesses = 0urls.each do |url|    request = Typhoeus::Request.new(url, timeout: 15000)    request.on_complete do |response|        if response.success?            puts "Successfully requested " + url            successes += 1        else            puts "Failed to get " + url        end    end    hydra.queue(request)endhydra.run puts "Fetched all urls!" if successes == urls.length

Here's an example using threads.

require 'net/http'urls = [  {'link' => 'http://www.google.com/'},  {'link' => 'http://www.yandex.ru/'},  {'link' => 'http://www.baidu.com/'}]urls.each do |u|  Thread.new do    u['content'] = Net::HTTP.get( URI.parse(u['link']) )    puts "Successfully requested #{u['link']}"    if urls.all? {|u| u.has_key?("content") }      puts "Fetched all urls!"      exit    end  endendsleep

I have written an in-depth blog post about this topic which includes an answer that is somewhat similar to the one August posted - but with a few key differences:1) Keeps track of all thread references in "thread" array.2) Uses "join" method to tie up threads at the end of program.

require 'net/http'# create an array of sites we wish to visit concurrently.urls = ['link1','link2','link3']  # Create an array to keep track of threads.threads = []urls.each do |u|    # spawn a new thread for each url  threads << Thread.new do  Net::HTTP.get(URI.parse(u))    # DO SOMETHING WITH URL CONTENTS HERE    # ...    puts "Request Complete: #{u}\n"  endend# wait for threads to finish before ending program.threads.each { |t| t.join }puts "All Done!"  

The full tutorial (and some performance information) is available here: https://zachalam.com/performing-multiple-http-requests-asynchronously-in-ruby/