This lesson's interactive features are locked, please to keep using them

Collecting Page Data

Now we have a crawler that can run efficiently, with options to tune concurrency and set a limit on the number of pages to crawl.

But you may have noticed that what the crawler is doing isn't very useful. It recursively fetches pages and... increments counters to record how many times it hit each page? What's the point of that?

Fortunately, we already wrote the function that we need in order to have our crawler collect actual data: extractPageData, from back in the first chapter.

What we need to do now is to update the pages member of our ConcurrentCrawler class, so that it's a map of URL strings to page data objects – we have an ExtractedPageData type for this – rather than to numbers of visits. i.e.,

pages: Record<string, ExtractedPageData>;

This way, when we call our crawler, it will return a data structure that tells us both what URLs it found and some key details about each of them.

Assignment

Update the type of pages to Record<string, ExtractedPageData>.
Remove the logic that increments a counter when a page is visited; it's no longer relevant (or even correct, given the type change).
In the recursive crawling function, remove the direct call to getURLsFromHTML. Replace it with a call to extractPageData, and assign the return value to a variable, e.g.:
```
const data = extractPageData(html, currentURL);
```
After extracting the page data, save it to pages[normalizedURL]. This is the key part where we're putting actual data about each page in the pages map!
Since we called extractPageData instead of getURLsFromHTML, when we go to recurse, we need to iterate over the outgoing_links property of the page data we collected, e.g.:
```
for (const nextURL of data.outgoing_links)
```
Update any function signatures that still have the old type of pages (Record<string, number>). If you miss any, the TypeScript compiler may catch them for you...
Back in the main script file, when you call await crawlSiteAsync(), make sure the result is assigned to a variable, e.g. pages. (You may already have it this way.)

After the crawler runs, add the following console output, which reports that crawling is finished and verifies that appropriate page data exists for the first page that was visited:

console.log("Finished crawling.");
const firstPage = Object.values(pages)[0];
if (firstPage) {
  console.log(
    `First page record: ${firstPage["url"]} - ${firstPage["heading"]}`,
  );
}

Try running the crawler on https://learnwebscraping.dev/practice/ecommerce/. If everything looks right, then run and submit the CLI tests.