Adding Recursion

1. Create a crawlPage function that uses recursion to traverse a website. Recursion is useful because we want to continue to crawl each URL we find on a page until we've crawled every page on the site. Here's my function signature:

async function crawlPage(
  baseURL: string,
  currentURL: string,
  pages: Record<string, number>,
);

• currentURL is the current URL we're crawling.
• baseURL is the root URL of the website we're crawling.

In the first call to crawlPage(), currentURL will just be a copy of the baseURL, but as we make further fetch requests to all the URLs we find on the baseURL, the currentURL value will change while the base stays the same.

The pages object will be used to keep track of the number of times we've seen each internal link. This function needs to always return an updated version of this object.

This is a good use case for default parameters:

function crawlPage(
  baseURL: string,
  currentURL: string = baseURL,
  pages: Record<string, number> = {},
);

Here's my pseudocode:

• Make sure the currentURL is on the same domain as the baseURL. If it's not, just return the current pages. We don't want to crawl the entire internet, just the domain in question.
• Get a normalized version of the currentURL.
• If the pages object already has an entry for the normalized version of the current URL, just increment the count and return the current pages.
• Otherwise, add an entry to the pages object for the normalized version of the current URL, and set the count to 1.
• Get the HTML from the current URL using our getHTML function, and add a print statement so you can watch your crawler in real-time.
• Assuming all went well with the fetch request in the new function, get all the URLs from the response body HTML.
• Recursively crawl each URL you found on the page and update the pages to keep an aggregate count.
• Finally, return the updated pages object.

2. Call crawlPage in the main function instead of getHTML and log the returned pages.
3. Test your program by running it against https://learnwebscraping.dev/practice/ecommerce/.

   npm run start https://learnwebscraping.dev/practice/ecommerce/
   

4. When you're satisfied that everything is working, you can move on.