Can I use WGET to generate a sitemap of a website given its URL?
wget --spider --recursive --no-verbose --output-file=wgetlog.txt http://somewebsite.comsed -n "s@.\+ URL:\([^ ]\+\) .\+@\1@p" wgetlog.txt | sed "s@&@\&@" > sedlog.txtThis creates a file called sedlog.txt that contains all links found on the specified website. You can use PHP or a shell script to convert the text file sitemap into an XML sitemap. Tweak the parameters of the wget command (accept/reject/include/exclude) to get only the links you need.