Concurrency

We will be using the p-limit package to help us add concurrency to our project.

1. Install the p-limit package:

   npm install p-limit
   

   • Follow the instructions in the p-limit documentation to import the package.

2. Create a class named ConcurrentCrawler to manage shared state for your crawler. Create a constructor to initialize the following fields:

   • baseURL (the starting URL)
   • pages (our Record of page visit counts)
   • limit (a function created with pLimit(maxConcurrency))

3. Add a method named addPageVisit to your ConcurrentCrawler:

   private addPageVisit(normalizedURL: string): boolean
   

   • Take a normalized URL
   • Update the pages object like you did before
   • Return true if it's the first time visiting the page, otherwise return false

4. Make your old getHTML function a method of ConcurrentCrawler. Keep the same error, status, and header checks:

   private async getHTML(currentURL: string): Promise<string>
   

   • Wrap the same fetch logic and error handling we had before in our new limit function and return the result:

     return await this.limit(async () => {
       // same fetch logic and error handling as before
       // return res.text() here
     });
     

   Wrapping our getHTML logic inside the limit function, gives us control over how many concurrent fetches run at once. This can be adjusted by changing the value of maxConcurrency. Keeping this low helps us avoid hammering the target server with too many requests.

5. Update your old crawlPage function and make it a method of ConcurrentCrawler:

   private async crawlPage(currentURL: string): Promise<void>
   

   • Call your new addPageVisit method, if it is not a new page return early
   • Get the HTML from the current URL using your new getHTML method
   • Get all the URLs from the response body HTML
   • Create an array of promises for each URL by calling this.crawlPage(nextURL)
   • Use Promise.all() to await all the concurrent crawl promises

6. Add a method named crawl to ConcurrentCrawler that starts by calling crawlPage with the baseURL and returns the pages

7. Create a function named crawlSiteAsync that:

   • Creates a ConcurrentCrawler instance
   • Calls and awaits the crawl() method
   • Returns the final pages Record

8. Test it (I tested it manually again). I recommend setting your maxConcurrency to 1 to start to just make sure everything still works. Then bump it up to 3, 5, or 10 to see it run faster.

• Make sure you're not crawling the same page multiple times. That's why my addPageVisit method returns a boolean: to indicate if it's the first time we've seen the page.
• Ensure you are only crawling URLs that belong to the same domain.
• Use Promise.all() to handle multiple concurrent requests efficiently