This lesson's interactive features are locked, please to keep using them

Max Pages

As it stands, our crawler is only useful on fairly small sites. Sites with thousands or hundreds of thousands of pages take too long to crawl.

Google's crawlers run concurrently on fleets of servers to crawl the majority of the internet. You might not have that kind of budget for this project.

Let's add a maxPages setting so that we can crawl even gigantic websites and have our tool automatically stop when it's done a reasonable amount of work.

Assignment

Add a maxPages: number field to your ConcurrentCrawler class. This will be the maximum number of unique pages to crawl.
Add a shouldStop: boolean field and an allTasks: Set<Promise<void>> to your ConcurrentCrawler to help coordinate stopping all concurrent tasks.
When creating new crawl tasks, add them to the allTasks set, and remove them when they complete. This can be done with .finally().
Update your addPageVisit method to properly stop the crawler when the limit is reached:
- Check if shouldStop is already true and return false immediately if so.
- Check if the number of unique pages visited has reached maxPages. If so:
  - Set shouldStop = true
  - Print "Reached maximum number of pages to crawl."
  - Return false
Update your crawlPage function to check shouldStop at the very beginning and return immediately if it's true.
Make both the maxConcurrency and maxPages settings configurable via command-line arguments.

Now you should be able to run your program with this syntax:

# usage: npm run start <URL> <maxConcurrency> <maxPages>
npm run start https://learnwebscraping.dev/practice/ecommerce/ 3 10

Make sure that your program prints a line to the console each time you crawl a page, as well as once per page in the report. This will help you see what your crawler is doing, and ensure you can kill it with Ctrl+C if it's stuck in a loop or spamming requests.

Run and submit the CLI tests.