Should be easy to do some sort of recursive loop or an iterative queue so that onyl the root URL needs to be listed.
There is already a URL set to prune the spider Having a content based match (SHA256) would also reduce the chance of getting caught in a spider redirect trap.
Should be easy to do some sort of recursive loop or an iterative queue so that onyl the root URL needs to be listed.
There is already a URL set to prune the spider Having a content based match (SHA256) would also reduce the chance of getting caught in a spider redirect trap.