A case study: Abate

A look under the hood

To build the application itself, I chose Next.js for its baked-in routing, caching, and API endpoints. Most of Abate’s magic happens on the backend using Axios for data fetching and Cheerio for page data scraping.

To curate a list of articles, for example, the frontend calls Abate’s API to fetch a list of articles from a news source and then parses through that data to display relevant information like article titles. For individual articles, Cheerio’s parsing power also enabled proper citation using the story title, author name, publication date, and link to the original article.

Below is a peek behind the curtain:

const axios = require("axios");
const cheerio = require("cheerio");

const getCNNArticle = (cnnUrl) => {
  return new Promise((resolve) => {
    resolve(
      axios(cnnUrl).then((pageData) => {
        let responses = [];
        let responseObject = {};
        let contentArray = [];
        const html = pageData.data;
        const $ = cheerio.load(html);
        responseObject.title = $(".headline").text();
        responseObject.authors = $(".byline--lite").text();
        responseObject.source = "CNN";
        responseObject.link = cnnUrl;

        const dateUpdated = $(".timestamp--lite")
          .text()
          .replace("Updated: ", "")
          .trimEnd()
          .trimStart();
        responseObject.date = dateUpdated;

        $(".article--lite .paragraph--lite").each(function () {
          if ($(this).text() === "See Full Web Article") {
            return;
          }
          contentArray.push($(this).text());
        });
        responseObject.content = contentArray;

        responses.push(responseObject);
        return responses;
      }),
    );
  });
};

export default getCNNArticle;

A few final thoughts

Early in development, one of the largest hurdles I faced was figuring out the legal aspects of an app like Abate. For instance, does scraping articles from news sources amount to copyright infringement, even with robust citations?

Another challenge I faced was the procurement of the article data itself. Some news sites have stripped down versions of their story repositories (such as CNN Lite). Nevertheless, the majority of news companies have either no public API, or restrict their APIs for paid users only.

I ultimately decided not to commercialize Abate given the legal questions and API limitations. Regardless, as an app I now use for myself, I can attest that it is always refreshing to get a simplified news experience when I’m feeling overwhelmed by the status quo.