Here's an idea to develop a centralized automotive tech news aggregator/reader, with RSS and simple spidering/scraping techniques.
--


This is a fairly simple programming project that could go a long way toward unifying a large group of separate automotive tech forums.

We can create a central page for people to view their choice of RSS feeds from a variety of automotive tech forums. Like this:

1. Compile a large list of automotive tech forums whose content we would like to centrally syndicate.
2. Pull RSS feeds from the forums that are already RSS-enabled. Scrape (with permission) the forums that aren't, and convert the scraped data into a simple RSS feed.
3. Categorize the feeds by automotive brand, and by automotive system. That way, a user can select, for example, all of the "Ford" feeds, or all of the "Engine" feeds.
4. Assemble a simple interface for users to select the feeds of their choice.

We can serve the whole thing ad-free from CrankshaftCoalition.com, using an existing RSS display tool, or something that we put together ourselves.

On some sites, this will be easy. The newer versions of most popular web forum software are either RSS-capable out-of-the-box, or have RSS functionality available via a common modification or hack. For example, here's an RSS feed for this exact forum: http://www.crankshaftcoalition.com/...ss2&forumids=10 .

If any automotive board can't produce an RSS feed, we can scrape it and do it for them. We could scrape the "newest threads" page for each forum, or the main pages of each separate category. Since many automotive forums run on the same handful of forum software -- vBulletin, phpBB, IPB, UBB.threads, Ikonboard, etc., we can establish re-usable spidering/scraping templates.

If you have comments, suggestions, or criticism for a web-based automotive tech RSS aggregator -- please post in this thread.

I think this sounds like an excellent idea. It would be invaluable to have a centralized outlet for the enormous amount of automotive information that is floating around on the web right now. As RSS becomes more popular and widely spread, it seems like a perfect match for a project of this sort.

If we could get a number of people excited in and working on this project, we could do a real service to the community.

So where do we begin?

I'll start putting together a list of essential features, and we can take it from there.

We need a tool for spidering pages and converting them to RSS. Then, we need a tool for converting RSS to HTML or PHP; Magpie RSS looks good for this.

That would be all for the initial design. A method for searching feeds for keywords would come next, as well as some sort of user authentication system that could be used to store feed preferences and a few other settings.