discoverValidSitemaps can hang indefinitely #3412
Description
The function performs network requests to robots/sitemaps. If any of those requests stalls the promise never resolves and the async iterator blocks crawler startup.
{{ message }}
The function performs network requests to robots/sitemaps. If any of those requests stalls the promise never resolves and the async iterator blocks crawler startup.