Cloudfeaster is a web crawling and business process automation infrastructure that leverages modern software engineering tools, services and trends to create:

• (this repo is) a spider authoring and maintenance environment
• a service infrastructure for discovering and running spiders via a RESTful API

This is the story behind Cloudfeaster.

• this describes how to author spiders using Cloudfeaster
• see these instructions describe how to setup your development environment and start contributing to Cloudfeaster
• take a look at cloudfeaster/samples for some spiders that illustrate the core concepts
• and simonsdave/gaming-spiders is an entire repo of spiders so you can get a sense of the end-to-end