INS-6353 [Web] Setup header for web scrape task

Hi @ShihChun-H can you please assign this issue to me?

Hi @someshfengde , sure. The issue has been assigned to you.

thank you will start

Hi @ShihChun-H can you please help me get started for working on this issue. I've been trying to set up my machine according to contributions.md but it's not working out (it's been 50 + mins since pulling images from docker)

Also can you explain in more detail what I've to do?

from description mentioned I think I have to add headers to schema/ai-tasks.json lmk if I'm on right path.

I've been thinking to add this

    "headers": {
      "title": "Request Headers",
      "description": "HTTP headers to include in the request.",
      "type": "object",
      "additionalProperties": {
        "type": "string"
      }
    }

Thanks :)

Hi @someshfengde ,
Thanks for taking time on this.

can you please help me get started for working on this issue. I've been trying to set up my machine according to contributions.md but it's not working out (it's been 50 + mins since pulling images from docker)

It could be the several reasons. From my experience, you may need to increase your docker resources.
In the Docker Desktop, you can find them here. Could you please try it out again?

Or, sometimes restarting your PC / cleaning your docker resources could help as well.

Also can you explain in more detail what I've to do?

You have to add more params in web operator's tasks.json.
To scrape some websites requiring more information, scraper needs to set up the header to access the website.
So, you can add more optional params in scrapers.
And, the users can set up some tokens or key when they scrape specific sites.

I hope I answer your all questions.
Please feel free to ask me anything if there is further question!
Thank you again!

Hi @someshfengde, I'm following up to check on any progress made or any question encountered regarding this issue. Could you please provide an update? Thanks 🙏

sorry I have been busy for last couple of days. Will continue to work on it after some hours

Hey @someshfengde how's it going? If you have any PR for this, don't forget to submit it!

sorry got into other tasks I think it'll require me lots of efforts can you please assign this to someone else?

@someshfengde No worries - thank you for letting me know! Good luck with your other tasks

hii @kuroxx , if you dont mind can i look into this issue?

Hey @Sourabh782, sounds great! I have assigned it to you 🤝

Hey @Sourabh782 how's it going? Any blockers or progress?

If you have questions or need help, we have Discord community here: https://discord.gg/sevxWsqpGh

Hey @Sourabh782, not sure if you're still working on this but since it's been 2 weeks now - I will unassign this task. Thanks