Differer project aims to help Bug Bounty Hunters to find differences between several languages and libraries URL parsers. Not all of them behave in the same way and it might lead to unexpected vulnerabilities.

URLs format is defined in RFC 3986, however there are small differences between languages, libraries and how they deal with incorrect URLs. Some of them report an error to the caller, other raise exceptions and other go with the best-effort approach and try to fix them for you. It is exactly there where unexpected security issues might arise.

         foo://example.com:8042/over/there?name=ferret#nose
         \_/   \______________/\_________/ \_________/ \__/
          |           |            |            |        |
       scheme     authority       path        query   fragment
          |   _____________________|__
         / \ /                        \
         urn:example:animal:ferret:nose

A lot of work has been done in this particular topic already. One of the most popular places where it has been discussed is on Orange's presentation A New Era of SSRF - Exploiting URL Parser in Trending Programming Languages!.

This project doesn't bring any new attack technique, rather than that it tries to make the process of finding parser differences easier.

To be able to run the parsers against the desired URLs on demand and without worrying about setting up the compilers, interpreters or the messaging broker. The main use case of this project is to be deployed under Google Cloud Platform using the smallest amount of resources and letting it tear down the services if they aren't being used.

The only thing I want to do is to submit a URL somewhere and get the different parsers results.

Please, remember that this project focuses on maintainability, not on performance.

Here are the numbers when running the project with 400 URLs against 3 parsers. Parser instances located in europe-west1, have 128 MiB of RAM, 1 vCPU, max 4 instances per parser and up to 60 concurrent requests for each instance. The parsers are:

$ curl -o /dev/null -X'POST' -d @data.json -s -w "%{time_total}\n" "https://REDACTED/differer"
0.888706

For cold loads (calls that happen after cloudrun shuts down the containers) time is higher, as expected:

$ curl -o /dev/null -X'POST' -d @data.json -s -w "%{time_total}\n" "https://REDACTED/differer"
2.760033

Because I don't want to run the tools locally each time I want to see how different languages parse an URL. I simply run a query to my service and get the output.

Setting the project up using App Engine and Cloud Run allows me to forget about infrastructure. GCP shuts my services down and up, plus allows me to restrict the access to them thanks to the firewall and IAM rules.

However, the project can be used locally too. See local setup docs or GCP docs.

The configuration file is a simple YAML file. Here is an example if you want to run the project locally with Go's, Python's and Node's parsers. See the config_example.yaml file for a raw example.

---
runners:
  golang: http://golang-parseurl:8082/
  python3_urllib_urlparse: http://python-parseurl:8083/
  node_url_parse: http://node-parseurl:8084/
timeout: 10s

As long as your new runner listens on HTTP for a POST request containing the jobs, the service is agnostic and doesn't care about where or how you run each runner.

The Job and Result structure can be found into the Protocol Buffer the project uses. Use the protoc compiler to generate your language's jobs and results parsers. See this document for a complete example using Go.

Once your runner is deployed somewhere, just edit your config.yaml adding it.

For simplicity, let's assume you run all the services using Docker containers. Follow the local setup guide and then just send a request to differer with the URLs you want it to parse.

$ curl -s --request POST 'http://127.0.0.1:8080/differer' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "addresses": [
        "https://google.com:443/foobar",
        "http://user:[email protected]/?pwnz=1"
    ]
}' | jq .
{
  "results": [
    {
      "runner": "python3_urllib_urlparse",
      "string": "https://google.com:443/foobar",
      "outputs": {
        "id": "python3:urllib:urlparse",
        "value": "Scheme=https; Host=google.com:443; Path=/foobar;"
      }
    },
    {
      "runner": "python3_urllib_urlparse",
      "string": "http://user:[email protected]/?pwnz=1",
      "outputs": {
        "id": "python3:urllib:urlparse",
        "value": "Scheme=http; Host=user:[email protected]; Path=/; User=user:legit.com;"
      }
    },
    {
      "runner": "node_url_parse",
      "string": "http://user:[email protected]/?pwnz=1",
      "outputs": {
        "id": "node14:url.parse",
        "value": "Scheme=http:; Host=attacker.com; Path=/; User=user:legit.com"
      }
    },
    {
      "runner": "node_url_parse",
      "string": "https://google.com:443/foobar",
      "outputs": {
        "id": "node14:url.parse",
        "value": "Scheme=https:; Host=google.com:443; Path=/foobar;"
      }
    },
    {
      "runner": "golang",
      "string": "https://google.com:443/foobar",
      "outputs": {
        "id": "golang",
        "value": "Scheme=https; Host=google.com:443; Path=/foobar;"
      }
    },
    {
      "runner": "golang",
      "string": "http://user:[email protected]/?pwnz=1",
      "outputs": {
        "id": "golang",
        "value": "Scheme=http; Host=attacker.com; Path=/; User=user:legit.com;"
      }
    }
  ]
}

Indeed the service would be faster if runners would accept multiple tasks at a time, and changing it to support them would be straight forward. However, I decided to keep it as simple as possible as it's performant enough for me.

The project aims for maintainability and ease of use over performance. Feel free to fork it if you disagree.

Please, check the contributing documentation.

I decided to build this project after a discussion with some friends (Karel, Karim, Corben and Amal).