Using Bash to cURL a website and grep for keywords

Here's how I would do it:

#!/bin/bashkeywords="$(<./keywords.txt)"while IFS= read -r url; do    curl -L -s "$url" | grep -ioF "$keywords" |        while IFS= read -r keyword; do            echo "$url: $keyword"        donedone < ./url_list.txt

What did I change:

• I used $(<./keywords.txt) to read the keywords.txt. This does not rely on an external program (cat in your original script).
• I changed the for loop that loops over the url list, into a while loop. This guarentees that we use Θ(1) memory (i.e. we don't have to load the entire url list in memory).
• I remove /dev/null from grep. greping from /dev/null alone is meaningless, since it will find nothing there. Instead, I invoke grep with no arguments so that it filters its stdin (which happens to be the output of curl in this case).
• I added the -o flag for grep so that it outputs only the matched keyword.
• I removed the subshell where you were capturing the output of curl. Instead I run the command directly and feed its output to a while loop. This is necessary because we might get more than keyword match per url.