This lesson's interactive features are locked, please to keep using them

Extract Page Content

Our web crawler will need to know how to read a page of HTML. Now that we can normalize URLs, let's start extracting actual content from web pages. A web scraper that only normalizes URLs isn't very useful - we need to parse HTML and extract the meaningful information.

Let's start with just parsing the <h1> and first <p> tags.

For example, from this HTML page:

<html>
  <body>
    <h1>Welcome to Boot.dev</h1>
    <main>
      <p>Learn to code by building real projects.</p>
      <p>This is the second paragraph.</p>
    </main>
  </body>
</html>

We want to extract:

<h1>: "Welcome to Boot.dev"
The first <p>: "Learn to code by building real projects."

Assignment

We'll use a third-party HTML parsing library called JSDOM to find and extract links.

Stub out the new getHeadingFromHTML function and write some tests. Here's the signature:

function getHeadingFromHTML(html: string): string;

html is an HTML string
In your tests make sure that the function returns the text content of the <h1> tag if present, or the <h2> tag as a fallback.
Returns an empty string if neither an <h1> nor an <h2> tag is found.

Install jsdom and its types:

npm install jsdom
npm install -D @types/jsdom

This will install jsdom as a "dependency" (as opposed to vitest which is a "devDependency" and was installed with the -D flag). "Dev dependencies" are not required to run your application, they're only required for development (like testing). Regular dependencies are required to run the program itself.

I'll try not to give too many hints: you should go read the JSDOM docs! That said here are a few:

import { JSDOM } from 'jsdom'
new JSDOM(htmlBody) creates a new "document object model"
dom.window.document.querySelectorAll('a') returns an array of <a> tag "anchor" elements

Implement the getHeadingFromHTML function. We've already installed it as a dependency, you can import it using: import { JSDOM } from "jsdom";
Create a new function called getFirstParagraphFromHTML. Here's the signature:

function getFirstParagraphFromHTML(html: string): string;

In your tests make sure that the tests return the text content of the first <p> tag.
Returns an empty string if no <p> tag is found.

You may find that the first <p> tag doesn't always have the best or most useful results. I'd recommend searching for the <main> tag if it exists and find the first <p> tag within it, if it doesn't exist fallback to just
the first <p> tag.

Here are some example test cases to get you started:

test("getHeadingFromHTML basic", () => {
  const inputBody = `<html><body><h1>Test Title</h1></body></html>`;
  const actual = getHeadingFromHTML(inputBody);
  const expected = "Test Title";
  expect(actual).toEqual(expected);
});

test("getFirstParagraphFromHTML main priority", () => {
  const inputBody = `
    <html><body>
      <p>Outside paragraph.</p>
      <main>
        <p>Main paragraph.</p>
      </main>
    </body></html>
  `;
  const actual = getFirstParagraphFromHTML(inputBody);
  const expected = "Main paragraph.";
  expect(actual).toEqual(expected);
});

Make sure you have at least 3 test cases for each function you are testing!

Run and submit the CLI tests from the root of your module.