Infiniteurl is a webapp that continually generates new URLs. Use it on a host that has been dns wildcarded (i.e. Its been set so that any subdomain will return the domains IP). Default settings put up a seed page of 200 URLs. Each URL is on a subdomain of the accessed domain. Each URL points at a page that will have ten new URLs in it; each access gets a new set of ten new URLs. Every ten seconds a new subdomain will be introduced. Edit the JSP page infiniteurls.jsp to change rate of new server introduction and the number of links per page.
To use, checkout this project and build with maven using the 'war' target to generate the war file in the target directory. Place the webapp under the tomcat, Jetty, etc., webapp directory. Point the crawler at the deployed webapp and let it run. This webapp is used testing Heritrix.