If pages arent fast or links are broken i move the url to a different file checked less frequently, when checking those i move what fails to the next file and check that on an even lower frequency (if at all)

Yacy is also fun be it a tad weird.