How can I automate running commands remotely over SSH to multiple servers in parallel?
Often, I'll just use the original Tcl version of Expect. You only need to have that on the local machine. If I'm inside a program using Perl, I do this with Net::SSH::Expect. Other languages have similar "expect" tools.
The issue of how to run commands on many servers at once came up on a Perl mailing list the other day and I'll give the same recommendation I gave there, which is to use gsh
:
http://outflux.net/unix/software/gsh
gsh is similar to the "for box in box1_name box2_name box3_name
" solution already given but I find gsh to be more convenient. You set up a /etc/ghosts file containing your servers in groups such as web, db, RHEL4, x86_64, or whatever (man ghosts) then you use that group when you call gsh.
[pdurbin@beamish ~]$ gsh web "cat /etc/redhat-release; uname -r"www-2.foo.com: Red Hat Enterprise Linux AS release 4 (Nahant Update 7)www-2.foo.com: 2.6.9-78.0.1.ELsmpwww-3.foo.com: Red Hat Enterprise Linux AS release 4 (Nahant Update 7)www-3.foo.com: 2.6.9-78.0.1.ELsmpwww-4.foo.com: Red Hat Enterprise Linux Server release 5.2 (Tikanga)www-4.foo.com: 2.6.18-92.1.13.el5www-5.foo.com: Red Hat Enterprise Linux Server release 5.2 (Tikanga)www-5.foo.com: 2.6.18-92.1.13.el5[pdurbin@beamish ~]$
You can also combine or split ghost groups, using web+db or web-RHEL4, for example.
I'll also mention that while I have never used shmux, its website contains a list of software (including gsh) that lets you run commands on many servers at once. Capistrano has already been mentioned and (from what I understand) could be on that list as well.