Enhanced Content Extraction with Gemini PDF/Image Analysis

Changes

• Added PDF and image detection for sub-links within scraped pages
• Integrated Gemini Vision API for image analysis and bounding box detection
• Added PDF text extraction using Gemini's document understanding capabilities
• Appends extracted content to main markdown for comprehensive analysis

Technical Details

• New helper functions:
  • is_pdf_url() & is_image_url() for file type detection
  • gemini_extract_pdf_content() for PDF text extraction
  • gemini_extract_image_data() for image analysis and bounding boxes
• Modified scrape_url to include 'links' format for sub-link discovery
• Extracted content is clearly labeled with source URLs in the markdown output

Impact

This enhancement allows the crawler to discover and analyze content within PDFs and images that would otherwise be missed, providing more comprehensive results for user `queries.```