RSS

The whole point of the gator program is to fetch the RSS feed of a website and store its content in a structured format in our database. That way we can display it nicely in our CLI.

RSS stands for "Really Simple Syndication" and is a way to get the latest content from a website in a structured format. It's fairly ubiquitous on the web: most content sites have an RSS feed.

Structure of an RSS Feed

RSS is a specific structure of XML (I know, gross). We will keep it simple and only worry about a few fields. Here's an example of the documents we'll parse:

<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
<channel>
  <title>RSS Feed Example</title>
  <link>https://www.example.com</link>
  <description>This is an example RSS feed</description>
  <item>
    <title>First Article</title>
    <link>https://www.example.com/article1</link>
    <description>This is the content of the first article.</description>
    <pubDate>Mon, 06 Sep 2021 12:00:00 GMT</pubDate>
  </item>
  <item>
    <title>Second Article</title>
    <link>https://www.example.com/article2</link>
    <description>Here's the content of the second article.</description>
    <pubDate>Tue, 07 Sep 2021 14:30:00 GMT</pubDate>
  </item>
</channel>
</rss>

We'll then then parse this kind of document into a JavaScript objects like this:

type RSSFeed = {
  channel: {
    title: string;
    link: string;
    description: string;
    item: RSSItem[];
  };
};

type RSSItem = {
  title: string;
  link: string;
  description: string;
  pubDate: string;
};

If our program were running in a browser, we could use the built-in DOMParser API, but since we're in Node.js, we'll use fast-xml-parser instead.

Assignment

Install the fast-xml-parser package and its types:

npm i fast-xml-parser

Write a fetchFeed(feedURL: string) function. It should fetch a feed from the given URL, and, assuming that nothing goes wrong, return a filled-out RSSFeed object.
1. Fetch the feed data:
  - I set the User-Agent header to gator This is a common practice to identify your program to the server.
  - I resolved the response to a string using .text()
2. Parse the XML:
  - Use the XMLParser constructor from the fast-xml-parser package to create a new parser object.
  - Use the parse() method on the parser object to convert the XML string into a JavaScript object.
3. Extract the channel field:
  - Verify that the channel field exists before proceeding. Handle errors if it does not.
4. Extract metadata:
  - Ensure these fields are present and valid before continuing.
  - Retrieve the title, link, and description fields from the channel field.
5. Extract feed items:
  - If the channel field has an item field, it should be an array of objects. Use the Array.isArray() function to verify this. If it isn't an array, set the item field to an empty array.
  - For each item:
    - Extract fields such as title, link, description, and pubDate.
    - Skip any item that has missing or invalid fields.
6. Assemble the result:
  - Create an object with the channel metadata and the list of items.
  - Return this object as the final output.
Add an agg command. Later this will be our long-running aggregator service. For now, we'll just use it to fetch a single feed and ensure our parsing works. It should fetch the feed found at https://www.wagslane.dev/index.xml and print the entire object to the console.

Run and submit the CLI tests.

The Boot.dev CLI requires you to be signed in to submit your solution!

Copy/paste one of the following commands into your terminal:

Run

bootdev run 29c20abd-16d5-4c8c-866a-b8c9b9842cca

Submit

bootdev run 29c20abd-16d5-4c8c-866a-b8c9b9842cca -s

Run the CLI commands to test your solution.

npm run start agg
- Expecting exit code: 0
- Expecting stdout to contain all of:
  - The Zen of Proverbs
  - Optimize for simplicity

Using the Bootdev CLI

The Bootdev CLI is the only way to submit your solution for this type of lesson. We need to be able to run commands in your environment to verify your solution.

You can install it here. It's a Go program hosted on GitHub, so you'll need Go installed as well. Instructions are on the GitHub page.