Security Crawl Maze is a comprehensive testbed for web security crawlers. It contains pages representing many (hopefully all) ways in which one can link resources from a valid HTML document. List of all the cases covered by Security Crawl Maze can be found below.

Security crawlers are interested in different findings than regular web crawlers. They are not interested in maximizing content coverage but in maximizing code coverage. This application is supposed to provide a unified and extensive way of testing efficiency of web security crawlers. First release contains only static linking of resources from html documents but future development will focus on adding more complex cases such as: Single Page Applications (Angular, Polymer), dynamically generated content (Blogs, e-commerce systems) and many other.

NOTE: Test cases for JS frameworks have to be built and bundled in order to work. If you use Docker, everything is automated. However, if you don't, you will have to build the projects manually.

The primary goal was to be able to run and deploy the app easily in any environment. Therefore, we provide a Dockerfile which enables you to deploy it to any cloud that is run by a provider of your choice. For local development or testing you can also make the app up and running quickly either in a local container or as a Python Flask app. Please, find the instructions below.

1. pull the project and enter the project's directory
2. build the docker image docker build -t crawlmaze .
3. run the image and expose port 80 docker run -p 80:8080 --name crawlmaze crawlmaze
4. to remove the container docker rm -f crawlmaze

1. pull the project and enter the project's directory
2. install pip dependencies pip install -r requirements.txt
3. run app python app.py

or use the Dockerfile in the root directory to build a container image and deploy it to any other cloud of your choice.

There is a publicly available instance of the application running at http(s)://security-crawl-maze.app.

• HTML folder contains directories named after tag names. e.g. html/body/a will contain tests for an <a> tag which is located in the HTML's body element,
• HTML files are usually named after the attribute that links a resource. e.g. html/body/a/href.html will contain one test case for an href attribute inside an <a> tag,
• Nested tags are placed in nested folders. e.g. html/body/form/button will contain tests for a <button> tag placed inside a <form> tag,
• Resources that are expected to be found by crawlers end with '.found' suffix e.g. html/body/a/href.html will contain a link to http://<HOST>/html/body/a/href.found. This way it's easy to identify a test case that is not found by your crawler.
• Files without extensions under the test-cases/ directory are required so that links to API endpoints are generated.

The application exposes an API endpoint that you can use to fetch a set of URLs that are expected to be found by crawlers. It is located under:

http://<HOST>/fetch-expected-results?path=<PATH>

where is a starting url of the crawl e.g.

http://<HOST>/fetch-expected-results?path=/html/body/form

returns:

[
    "/test/html/body/form/action.found",
    "/test/html/body/form/button/formaction.found"
]

Implemented test cases (resources to be found) are available in the blueprints/utils/resources/expected-results.json file.

1. Create a file for your test case and place it in an appropriate directory.
   • If your test content is generated dynamically by an API endpoint, add a file without an extension (e.g. test-cases/headers/link). This is to make sure the link to the test case is generated and is discoverable by crawlers.
   • If you're NOT creating any new child folder in the test-cases/ directory go to point 2.
   • Otherwise you have to add a new blueprint directory with all the relevant components. You can reuse the structure of already existing blueprints.
2. Add record which is to be found to the blueprints/utils/resources/expected_results.json file.
3. Test your crawler with the new test case!
4. Before creating a PR, make sure your code follows the Google Python Language Rules

Many of the test cases were borrowed from a document by cure53 HTTPLeaks.

See the LICENSE file.