Selecting htt:// as the protocol is seen as a valid URL. h:// and ht are correctly classified as invalid.

Any three or four character string as the protocol appears to be valid.

HTTP and HTTPS should be valid, unsure if there's any others we should support.

After setting the max pages per crawl value in the org settings to 100, the URL list still accepts up to 1000 URLs. Anish mentioned checking the max pages per crawl org setting and adjusting this value accordingly. Our standard plan will support up to 2000 pages per crawl, while 1000 is under that either way we should probably at least check if the org limit is below 1000 and set accordingly if yes?

Personally I think this is kind of an edge case and can't really imagine a situation people would run into this one in the wild... But would be nice to not let people enter a value that will result in not crawling everything intended?

Other than this, looks good!

Selecting htt:// as the protocol is seen as a valid URL. h:// and ht are correctly classified as invalid.
HTTP and HTTPS should be valid, unsure if there's any others we should support.

Generally I favor permissive URLs (new URL('a://b.c') passes, for example) but since we're crawling webpages, it does make sense to limit to http/s. @ikreymer any instances where this wouldn't be the case?

After setting the max pages per crawl value in the org settings to 100, the URL list still accepts up to 1000 URLs.

Could you log a follow-up issue for this, since it needs to be implemented regardless of the URL list max limit?

Could you log a follow-up issue for this, since it needs to be implemented regardless of the URL list max limit?

@SuaYoo Spun out into #1245!

Selecting htt:// as the protocol is seen as a valid URL. h:// and ht are correctly classified as invalid.
HTTP and HTTPS should be valid, unsure if there's any others we should support.

Generally I favor permissive URLs (new URL('a://b.c') passes, for example) but since we're crawling webpages, it does make sense to limit to http/s. @ikreymer any instances where this wouldn't be the case?

The crawler won't work for a URL that doesn't start with http:// or https://, so probably good to be restrictive here!

Yep, should consider a non-http/https scheme as invalid, will fail in the crawler.
We could potentially automatically add a scheme if none is present, default to https, so if user enters
'example.com' convert it to 'https://example.com' but maybe that should be a separate issue...

The crawler won't work for a URL that doesn't start with http:// or https://, so probably good to be restrictive here!

Updated!