Skip to content

Rules with uppercase keywords can not be found when those rules are defined in a TOML that is extended. #1634

@craigmarksmith

Description

@craigmarksmith

Describe the bug

Rules with uppercase keywords find no results when those rules are defined in a TOML that is extended.

When the TOML is consumed into the Config object, the keywords for each rule in the TOML are converted to lowercase and then added to a summary of keywords on the Config object.

keywords[strings.ToLower(k)] = struct{}{}

Then, during the detect phase, these lowercase keywords are used to detect secrets in the the raw fragment which has also been converted to lowercase.

This works as intended for simple configurations, but when one configuration extends from another, the keywords of the base configuration are not converted to lowercase.

Take for example the configuration

title = "Extend Config"

[extend]
path="base.toml"

[[rules]]
    id = "aws-secret-key-again-again"
    description = "A new description"
# base.toml
title = "Base Configuration"

[[rules]]
    id = "aws-secret-key-again-again"
    description = "AWS Secret Key"
    regex = '''(?i)aws_(.{0,20})?=?.[\'\"0-9a-zA-Z\/+]{40}'''
    tags = ["key", "AWS"]
    keywords = ["AWS"]
[[rules.allowlists]]
    stopwords = ["fake"]

Once parsed, the above configuration results in the keyword AWS, rather than aws which is required for detect to work as expected.

To Reproduce

Clone this test project.

Navigate to the cloned folder and scan for secrets using

docker run -it --rm -w /myapp -v ${PWD}:/myapp zricethezav/gitleaks:latest detect -v -l trace --report-path /myapp/gitleaks.json --report-format json --source /myapp/src --config src/rule_extentions.toml --no-git

The results are:

    ○
    │╲
    │ ○
    ○ ░
    ░    gitleaks

6:09AM DBG using gitleaks config src/rule_extentions.toml from `--config`
6:09AM DBG extending config with /myapp/base_rules.toml
6:09AM TRC Scanning path: {/myapp/src/rule_extentions.toml }
6:09AM TRC Scanning path: {/myapp/src/secrets.txt }
6:09AM TRC Skipping finding due to global allowlist regex finding="glpat-Ibbbbbbbbbbbbbbbbbbb " path=/myapp/src/secrets.txt rule-id=gitlab_personal_access_token
6:09AM TRC Skipping finding due to global allowlist regex finding=glpat-Ibbbbbbbbbbbbbbbbbbb' path=/myapp/src/rule_extentions.toml rule-id=gitlab_personal_access_token
Finding:     glpat-Dbbbbbbbbbbbbbbbbbbb  //should be detect...
Secret:      glpat-Dbbbbbbbbbbbbbbbbbbb
RuleID:      gitlab_personal_access_token
Entropy:     1.596185
Tags:        [gitlab revocation_type gitlab_blocking]
File:        /myapp/src/secrets.txt
Line:        1
Fingerprint: /myapp/src/secrets.txt:gitlab_personal_access_token:1

however, https://gitlab.com/craigmsmith/test_extend_494511/-/blob/f9ac067ccc2d4cc5c38c807f8cc20b4dd1a65191/src/secrets.txt#L3 should also have been detected.

Expected behavior

https://gitlab.com/craigmsmith/test_extend_494511/-/blob/f9ac067ccc2d4cc5c38c807f8cc20b4dd1a65191/src/secrets.txt#L3 should also have been detected.

Screenshots
If applicable, add screenshots to help explain your problem.

Basic Info (please complete the following information):

  • OS:
  • Gitleaks Version:

Additional context
Add any other context about the problem here.

cc @zricethezav

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions