Skip to content

fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827)#1134

Merged
mogery merged 2 commits intomainfrom
mog/dedupe-index-html-php-slash-bare
Feb 5, 2025
Merged

fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827)#1134
mogery merged 2 commits intomainfrom
mog/dedupe-index-html-php-slash-bare

Conversation

@mogery
Copy link
Member

@mogery mogery commented Feb 4, 2025

Doesn't scrape the same URL twice if it also discovers the same thing with /index.html or /index.php on the end.

@mogery mogery requested a review from nickscamara February 4, 2025 18:41
@mogery mogery changed the title fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) Feb 4, 2025
@mogery mogery merged commit 34fe360 into main Feb 5, 2025
1 check passed
@mogery mogery deleted the mog/dedupe-index-html-php-slash-bare branch February 17, 2025 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants