Awk match() - multiple per line
That is correct. AWK regular expressions don't have flags.
Also, there's no built-in support for having match
look for the second or later matches.
Only the gsub
and gensub
functions have that.
I'd try something like this:
gensub(/.*<a href=\"([^\"]+)/, "\1%", "g")last = split($0, "%", arr)delete arr[last]
where %
is a string that you can guarantee won't be found in the input.
The text-mode browser lynx might be a better tool for harvesting URLs. It's -dump
flag writes formatted output to standard output. At the end, you'll find a numbered list of every visible and hidden link on that page. (Or file. It accepts URLs or filenames as arguments.)
$ lynx -dump http://www.stackoverflow.com [snip]References Visible links 1. http://stackoverflow.com/opensearch.xml 2. http://stackoverflow.com/feeds 3. http://stackexchange.com/ 4. http://stackoverflow.com/users/login 5. http://careers.stackoverflow.com/ 6. http://chat.stackoverflow.com/[snip] 676. http://creativecommons.org/licenses/by-sa/3.0/ 677. http://blog.stackoverflow.com/2009/06/attribution-required/ Hidden links: 678. http://www.peer1.com/stackoverflow 679. http://creativecommons.org/licenses/by-sa/3.0/