superscrape v0.1.0 SuperScrape

Read an Atom/RSS feed and concatenate it with deduplicated historical items from past versions saved in the archive.org’s Wayback Machine.

Summary

Get all URL moments from Archive’s Wayback Machine

Read a single URL and return all available data. Intended for article and webpage scraping

Process a single or group of RSS/Atom feed urls

Process a single or group of RSS/Atom feed urls

lookup(url)

lookup(String.t) :: [String.t]

Get all URL moments from Archive’s Wayback Machine.

open(url)

open(String.t) :: map

Read a single URL and return all available data. Intended for article and webpage scraping.

process(url)

process(String.t | List.t) :: [map]

Process a single or group of RSS/Atom feed urls.

Returns all feed data per unique entry.

process!(url)

process!(String.t | List.t) :: [String.t]

Process a single or group of RSS/Atom feed urls.

Returns a list of unique URLs.