RSS from HTML

Trejkaz — Mon, 23 May 2005 10:00:00 +1000

From time to time I have a need to watch a bunch of links on a web page, where that page doesn’t have an RSS feed.

I use RSS to automate [Azureus][1], so this becomes a major problem for sites with a list of .torrent files but no feed

Requirements

Perl (duh.)
LWP::UserAgent (gentoo ebuild: dev-perl/libwww-perl)
XML::RSS (gentoo ebuild: dev-perl/XML-RSS)
URI::Escape (gentoo ebuild: dev-perl/URI)

The Code

#!/usr/bin/perl

use LWP::UserAgent;  # dev-perl/libwww-perl
use XML::RSS;        # dev-perl/XML-RSS
use URI::Escape;

# This is where the data will be fetched from.
my $url = uri_unescape($ENV{'QUERY_STRING'});

# Create the feed...
my $rss = XML::RSS->new( version => '1.0' );

# Do a query on the URL...
my $ua = LWP::UserAgent->new;
$ua->timeout(10);
$ua->env_proxy;
my $response = $ua->get($url);
die $response->status_line unless $response->is_success;
my $content = $response->content;

# Get the basic metadata for the page on the whole.
my $title;
if ($content =~ m/<title>\s*(.*?)\s*<\/title>/) {
    $title = $1;
}
#XXX: Later, see if it's worth doing the same for the description.

# This is the channel information.  Modify this as you please.
$rss->channel(
    'title'        => $title,
    'link'         => $url,
#    'description'  => "...",
);


# Now, find every link in the page.  This is where it gets fun...
while ($content =~ m/<a\s[^>]*?href\s*=\s*"(.*?)"[^>]*?>(.*?)<\/a>/gi) {
    my $link = $1;
    my $title = $2;

    # The link needs to be non-encoded, and absolute.
    $link = URI->new_abs($link, $url)->as_string;

    next unless $link =~ m/.torrent$/;

    $rss->add_item(
        'title'        => $title,
        'link'         => $link,
    );
}

# Done, generate the RSS as a string.
print "Content-Type: application/rss+xml\n\n";
print $rss->as_string;

Usage

Put the script somewhere (e.g. /cgi-bin/rssify) and access it like this:

http://example.com/cgi-bin/rssify?http://someothersite.com/myfiles/

The script will then fetch http://someothersite.com/myfiles/ and turn it into an RSS 1.0 feed.

The output is fairly simple. It takes the feed title from the page title and the page URL from the URL you gave it. It takes the content of the links to be the title of each item, and the full URL for each link becomes the link for each item. I didn’t bother with dates, as dates don’t make sense. Caveats

The script is currently far from secure, so do not put it on a live web site. An attacker can pretty much enter any URL

Trypticon: Tag html

RSS from HTML

Requirements

The Code

Usage