Java ThreadPool usage

I am not sure I understand your design, but this may be a job for a Semaphore

One option is to make "frontier" a blocking queue, So any thread trying to "get" from it will block. As soon as any other URLCrawler puts objects into that queue, any other threads will be automatically notified (with the object dequeued)

I think use of wait/notify is justified in this case. Can't think of any straight forward way to do this using j.u.c.
In a class, let's call Coordinator:

private final int numOfCrawlers;private int waiting;public boolean shouldTryAgain(){    synchronized(this){        waiting++;        if(waiting>=numOfCrawlers){            //Everybody is waiting, terminate            return false;        }else{            wait();//spurious wake up is okay            //waked up for whatever reason. Try again            waiting--;            return true;        }    }public void hasEnqueued(){    synchronized(this){        notifyAll();    }} 

then,

ExecutorService exec = Executors.newFixedThreadPool(numberOfCrawlers);while(true){    URL url = frontier.get();    if(url == null){        if(!coordinator.shouldTryAgain()){            //all threads are waiting. No possibility of new jobs.            return;        }else{            //Possible that there are other jobs. Try again            continue;        }    }    exec.execute(new URLCrawler(this, url));}//while(true)