uniq | sort does not work: uniq removes contiguous duplicates.
uniq | sort
uniq
The correct way is sort | uniq or better sort -u. Because only one process is spawned.
sort | uniq
sort -u
uniq needs its input sorted, but you sorted after uniq. Try:
$ sort -u < url.txt > urls.txt
Try something like this:
cat url.txt | sort | uniq