Skip to content

[Feature]: Watch Crawl: Graph Queued URLs & Pages Crawled #1792

@Shrinks99

Description

@Shrinks99

What change would you like to see?

I would like more insight into how the crawl queue has grown over time, and how close it might be to finishing.

Screenshot 2024-05-08 144717

Context

We came up with this graph idea while crawling Vice but it never made it into an issue.

@edsu Also came up with this in parallel and prototyped it!! (Above image is also his)

From Ed on Mastodon:

I think it ought to be possible to use the data to project when the crawl might complete, by fitting a line to each and then projecting where they (hopefully) intersect...

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementRequests a change to a featurefront endRequires front end dev workui/uxThis issue requires UI/UX work

    Type

    No type

    Projects

    Status

    Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions