cookbooks/01-website-crawler.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\" id=\"top\">\n",
    " <img src=\"https://github.com/user-attachments/assets/10ba11e4-4ced-400e-a400-ee0f72541780\" alt=\"julep\" width=\"640\" height=\"320\" />\n",
    "</div>\n",
    "\n",
    "<p align=\"center\">\n",
    "  <br />\n",
    "  <a href=\"https://docs.julep.ai\" rel=\"dofollow\">Explore Docs (wip)</a>\n",
    "  &middot;\n",
    "  <a href=\"https://discord.com/invite/JTSBGRZrzj\" rel=\"dofollow\">Discord</a>\n",
    "  &middot;\n",
    "  <a href=\"https://x.com/julep_ai\" rel=\"dofollow\">&#120143;</a>\n",
    "  &middot;\n",
    "  <a href=\"https://www.linkedin.com/company/julep-ai\" rel=\"dofollow\">LinkedIn</a>\n",
    "</p>\n",
    "\n",
    "<p align=\"center\">\n",
    "    <a href=\"https://www.npmjs.com/package/@julep/sdk\"><img src=\"https://img.shields.io/npm/v/%40julep%2Fsdk?style=social&amp;logo=npm&amp;link=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2F%40julep%2Fsdk\" alt=\"NPM Version\"></a>\n",
    "    <span>&nbsp;</span>\n",
    "    <a href=\"https://pypi.org/project/julep\"><img src=\"https://img.shields.io/pypi/v/julep?style=social&amp;logo=python&amp;label=PyPI&amp;link=https%3A%2F%2Fpypi.org%2Fproject%2Fjulep\" alt=\"PyPI - Version\"></a>\n",
    "    <span>&nbsp;</span>\n",
    "    <a href=\"https://hub.docker.com/u/julepai\"><img src=\"https://img.shields.io/docker/v/julepai/agents-api?sort=semver&amp;style=social&amp;logo=docker&amp;link=https%3A%2F%2Fhub.docker.com%2Fu%2Fjulepai\" alt=\"Docker Image Version\"></a>\n",
    "    <span>&nbsp;</span>\n",
    "    <a href=\"https://choosealicense.com/licenses/apache/\"><img src=\"https://img.shields.io/github/license/julep-ai/julep\" alt=\"GitHub License\"></a>\n",
    "</p>\n",
    "\n",
    "## Task Definition: Spider Crawler Integration\n",
    "\n",
    "### Overview\n",
    "\n",
    "This task is a simple task that leverages the spider `integration` tool, and combines it with a prompt step to crawl a website for a given URL, and then create a summary of the results.\n",
    "\n",
    "### Task Tools:\n",
    "\n",
    "**Spider Crawler**: An `integration` type tool that can crawl the web and extract data from a given URL.\n",
    "\n",
    "### Task Input:\n",
    "\n",
    "**url**: The URL of the website to crawl.\n",
    "\n",
    "### Task Output:\n",
    "\n",
    "**output**: A dictionary that contains a `documents` key which contains the extracted data from the given URL. Check the output below for a detailed output schema.\n",
    "\n",
    "### Task Flow\n",
    "\n",
    "1. **Input**: The user provides a URL to crawl.\n",
    "\n",
    "2. **Spider Tool Integration**: The `spider_crawler` tool is called to crawl the web and extract data from the given URL.\n",
    "\n",
    "3. **Prompt Step**: The prompt step is used to create a summary of the results from the spider tool.\n",
    "\n",
    "4. **Output**: The final output is the summary of the results from the spider tool.\n",
    "\n",
    "```plaintext\n",
    "+----------+     +-------------+     +------------+     +-----------+\n",
    "|  User    |     |   Spider    |     |   Prompt   |     |  Output   |\n",
    "|  Input   | --> |   Crawler   | --> |   Step     | --> |   Step    |\n",
    "| (URL)    |     |             |     |            |     | Output    |\n",
    "+----------+     +-------------+     +------------+     +-----------+\n",
    "      |                |                  |                  |\n",
    "      |                |                  |                  |\n",
    "      v                v                  v                  v\n",
    "   \"https://spider.cloud\"   Extract data   Create summary   \"Here are the\n",
    "                            from URL       of results      results from the\n",
    "                                                        spider tool\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "\n",
    "To recreate the notebook and see the code implementation for this task, you can access the Google Colab notebook using the link below:\n",
    "\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/julep-ai/julep/blob/dev/cookbooks/01-website-crawler.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "### Additional Information\n",
    "\n",
    "For more details about the task or if you have any questions, please don't hesitate to contact the author:\n",
    "\n",
    "**Author:** Julep AI  \n",
    "**Contact:** [hey@julep.ai](mailto:hey@julep.ai) or  <a href=\"https://discord.com/invite/JTSBGRZrzj\" rel=\"dofollow\">Discord</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Installing the Julep Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade julep --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "\n",
    "# NOTE: these UUIDs are used in order not to use the `create_or_update` methods instead of\n",
    "# the `create` methods for the sake of not creating new resources every time a cell is run.\n",
    "AGENT_UUID = uuid.uuid4()\n",
    "TASK_UUID = uuid.uuid4()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Julep Client with the API Key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from julep import Client\n",
    "import os\n",
    "\n",
    "api_key = os.getenv(\"JULEP_API_KEY\")\n",
    "\n",
    "# Create a Julep client\n",
    "client = Client(api_key=api_key, environment=\"dev\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating an \"agent\"\n",
    "\n",
    "Agent is the object to which LLM settings, like model, temperature along with tools are scoped to.\n",
    "\n",
    "To learn more about the agent, please refer to the [documentation](https://github.com/julep-ai/julep/blob/dev/docs/julep-concepts.md#agent)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create agent\n",
    "agent = client.agents.create_or_update(\n",
    "    agent_id=AGENT_UUID,\n",
    "    name=\"Spiderman\",\n",
    "    about=\"AI that can crawl the web and extract data\",\n",
    "    model=\"gpt-4o\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining a Task\n",
    "\n",
    "Tasks in Julep are Github-Actions-style workflows that define long-running, multi-step actions.\n",
    "\n",
    "You can use them to conduct complex actions by defining them step-by-step.\n",
    "\n",
    "To learn more about tasks, please refer to the `Tasks` section in [Julep Concepts](https://github.com/julep-ai/julep/blob/dev/docs/julep-concepts.md#tasks)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "spider_api_key = os.getenv(\"SPIDER_API_KEY\")\n",
    "\n",
    "# Define the task\n",
    "task_def = yaml.safe_load(f\"\"\"\n",
    "name: Crawling Task\n",
    "\n",
    "# Define the tools that the agent will use in this workflow\n",
    "tools:\n",
    "- name: spider_crawler\n",
    "  type: integration\n",
    "  integration:\n",
    "    provider: spider\n",
    "    setup:\n",
    "      spider_api_key: \"{spider_api_key}\"\n",
    "\n",
    "# Define the steps of the workflow\n",
    "main:\n",
    "# Define a tool call step that calls the spider_crawler tool with the url input\n",
    "- tool: spider_crawler\n",
    "  arguments:\n",
    "    url: \"_['url']\" # You can also use 'inputs[0]['url']'\n",
    "  \n",
    "    \n",
    "- prompt: |\n",
    "    You are {{{{agent.about}}}}\n",
    "    I have given you this url: {{{{inputs[0]['url']}}}}\n",
    "    And you have crawled that website. Here are the results you found:\n",
    "    {{{{_['documents']}}}}\n",
    "    I want you to create a short summary (no longer than 100 words) of the results you found while crawling that website.\n",
    "\n",
    "  unwrap: True\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:olive;\">Notes:</span>\n",
    "- The reason for using the quadruple curly braces `{{{{}}}}` for the jinja template is to avoid conflicts with the curly braces when using the `f` formatted strings in python. [More information here](https://stackoverflow.com/questions/64493332/jinja-templating-in-airflow-along-with-formatted-text)\n",
    "- The `unwrap: True` in the prompt step is used to unwrap the output of the prompt step (to unwrap the `choices[0].message.content` from the output of the model).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating/Updating a task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating the task object\n",
    "task = client.tasks.create_or_update(\n",
    "    task_id=TASK_UUID,\n",
    "    agent_id=AGENT_UUID,\n",
    "    **task_def\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating an Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An execution is a single run of a task. It is a way to run a task with a specific set of inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating an execution object\n",
    "execution = client.executions.create(\n",
    "    task_id=TASK_UUID,\n",
    "    input={\n",
    "        \"url\": \"https://spider.cloud\"\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking execution details and output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to get the execution details and the output:\n",
    "\n",
    "1. **Get Execution Details**: This method retrieves the details of the execution, including the output of the last transition that took place.\n",
    "\n",
    "2. **List Transitions**: This method lists all the task steps that have been executed up to this point in time, so the output of a successful execution will be the output of the last transition (first in the transition list as it is in reverse chronological order), which should have a type of `finish`.\n",
    "\n",
    "\n",
    "<span style=\"color:olive;\">Note: You need to wait for a few seconds for the execution to complete before you can get the final output, so feel free to run the following cells multiple times until you get the final output.</span>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spider.cloud is a leading web crawling tool designed for AI applications, offering high-speed, scalable, and cost-effective data collection solutions. Built in Rust, it can crawl over 20,000 pages in seconds, making it significantly faster and cheaper than traditional scrapers. Spider supports various data formats, including LLM-ready markdown, and integrates seamlessly with major AI tools. It offers features like auto proxy rotations, custom browser scripting, and caching to enhance performance. Users can start with $200 in credits and explore features through a free trial. Spider is trusted by tech businesses worldwide for insightful data solutions.\n"
     ]
    }
   ],
   "source": [
    "# Get execution details\n",
    "execution = client.executions.get(execution.id)\n",
    "# Print the output\n",
    "print(execution.output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transition type:  init\n",
      "Transition output:  {'url': 'https://spider.cloud'}\n",
      "--------------------------------------------------\n",
      "Transition type:  step\n",
      "Transition output:  {'documents': [{'id': None, 'metadata': {'description': 'Experience cutting-edge web crawling with unparalleled speeds, perfect for LLMs, Machine Learning, and Artificial Intelligence. The fastest and most efficient web scraper tailored for AI applications.', 'domain': 'spider.cloud', 'extracted_data': None, 'file_size': 10031, 'keywords': ['AI agent stack', 'AWS infrastructure reduced', 'Auto Proxy rotations', 'Comprehensive Data Curation', 'Concurrent Streaming Save time', 'Data Collecting Projects Today Jumpstart web crawling', 'FAQ Frequently asked questions', 'Fastest Web Crawler', 'Latest sports news', 'Multiple response formats', 'Open Source Spider engine', 'Performance Tuned Spider', 'Seamless Integrations Seamlessly integrate Spider', 'Smart Mode Spider dynamically switches', 'Spider accurately crawls', 'Spider convert web data', 'Spider outputs HTML', 'Transform Convert raw HTML', 'WilliamEspegren Web crawler built', 'achieve crawling thousands', 'affordable web scraping', 'and`XML`for API responses', 'caching repeated web page crawls', 'cost step caching', 'crazy resource management Aaaaaaand', 'custom browser scripting', 'data = json', 'data formats including LLM-ready markdown', 'ensure continuous maintenance', 'ensuring data curation perfectly aligned', 'finest data collecting solution', 'full elastic scaling concurrency', 'handle extreme workloads', 'iammerrick Rust based crawler Spider', 'insightful data solutions', 'json headers =', 'large scraping projects', 'large-scale data collection', 'leading tech businesses worldwide', 'leading web crawling tool designed', 'real-time web data', 'request Python JSONL Copy ``` `import requests', 'requires JavaScript rendering', 'response = requests', 'robust Rust engine scales effortlessly', 'scrapes significantly faster', 'search engine results', 'traditional scraping services Spider API Request Modes', 'training AI models', 'user interface segment showing'], 'pathname': '/', 'resource_type': '.md', 'title': 'Spider: The Web Crawler for AI', 'url': '8475428e-4e0c-44de-967f-c14fb73cf490/spider.cloud/_cloud/12638123428881205758.md', 'user_id': '8475428e-4e0c-44de-967f-c14fb73cf490'}, 'page_content': \"To help you get started with Spider, we&rsquo;ll give you $200 in credits when you spend $100.[Terms apply](https://spider.cloud/promotion-spider-credits)\\n# The Web Crawler for AI Agents and LLMs\\nSpider offers the finest data collecting solution. Engineered for speed and scalability, it\\nallows you to elevate your AI projects.\\n[Get Started](https://spider.cloud/credits/new)View Preview\\n* Basic\\n* Streaming\\nExample request\\nPython\\nJSONL\\nCopy\\n```\\n`import requests, os, json\\nheaders = {\\n&#x27;&#x27;Authorization&#x27;&#x27;: f&#x27;&#x27;Bearer {os.getenv(&quot;&quot;SPIDER\\\\_API\\\\_KEY&quot;&quot;)}&#x27;&#x27;,\\n&#x27;&#x27;Content-Type&#x27;&#x27;: &#x27;&#x27;application/jsonl&#x27;&#x27;,\\n}\\njson\\\\_data = {&quot;&quot;limit&quot;&quot;:50,&quot;&quot;metadata&quot;&quot;:True,&quot;&quot;url&quot;&quot;:&quot;&quot;https://spider.cloud&quot;&quot;}\\nresponse = requests.post(&#x27;&#x27;https://api.spider.cloud/crawl&#x27;&#x27;, headers=headers, json=json\\\\_data, stream=True)\\nwith response as r:\\nr.raise\\\\_for\\\\_status()\\nfor chunk in r.iter\\\\_lines(\\nchunk\\\\_size=None, decode\\\\_unicode=True\\n):\\ndata = json.loads(chunk)\\nprint(data)`\\n```\\n[Free Trial](https://spider.cloud/credits/new?free-trial=1)\\nExample Response\\n## Built with the need for**Speed**\\nExperience the power of**Spider**, built fully in**Rust**for\\nnext-generation scalability.\\n### 2secs\\nCapable of crawling over 20k pages in batch mode\\n### 500-1000x\\nFaster than alternatives\\n### 500x\\nCheaper than traditional scraping services\\nSpider API Request Modes &middot; Benchmarked tailwindcss.com &middot;06/16/2024\\n[See framework benchmarks](https://github.com/spider-rs/spider/blob/main/benches/BENCHMARKS.md)\\n### Seamless Integrations\\nSeamlessly integrate Spider with a wide range of platforms, ensuring data curation\\nperfectly aligned with your requirements. Compatible with all major AI tools.\\n[LangChain integration](https://python.langchain.com/docs/integrations/document_loaders/spider)[LlamaIndex integration](https://docs.llamaindex.ai/en/stable/examples/data_connectors/WebPageDemo/#using-spider-reader)[CrewAI integration](https://docs.crewai.com/tools/SpiderTool/)[FlowWiseAI integration](https://docs.flowiseai.com/integrations/langchain/document-loaders/spider-web-scraper-crawler)[Composio integration](https://docs.composio.dev/introduction/foundations/components/list_local_tools#spider-crawler)[PhiData integration](https://docs.phidata.com/tools/spider)\\n### Concurrent Streaming\\nSave time and money without having to worry about bandwidth concerns by effectively\\nstreaming all the results concurrently. The latency cost that is saved becomes drastic as\\nyou crawl more websites.\\n### Warp Speed\\nPowered by the cutting-edge[Spider](https://github.com/spider-rs/spider)open-source project, our robust Rust engine scales effortlessly to handle extreme\\nworkloads. We ensure continuous maintenance and improvement for top-tier performance.\\n## Kickstart Your Data Collecting Projects Today\\nJumpstart web crawling with full elastic scaling concurrency, optimal formats, and AI scraping.\\n### Performance Tuned\\nSpider is written in Rust and runs in full concurrency to achieve crawling thousands of\\npages in secs.\\n### Multiple response formats\\nGet clean and formatted markdown, HTML, or text content for fine-tuning or training AI\\nmodels.\\n### Caching\\nFurther boost speed by caching repeated web page crawls to minimize expenses while\\nbuilding.\\n### Smart Mode\\nSpider dynamically switches to Headless Chrome when it needs to quick.\\nBeta\\n### Scrape with AI\\nDo custom browser scripting and data extraction using the latest AI models with no cost\\nstep caching.\\n### The crawler for LLMs\\nDon't let crawling and scraping be the highest latency in your LLM & AI agent stack.\\n### Scrape with no headaches\\n* Auto Proxy rotations\\n* Agent headers\\n* Anti-bot detections\\n* Headless chrome\\n* Markdown responses\\n### The Fastest Web Crawler\\n* Powered by[spider-rs](https://github.com/spider-rs/spider)\\n* 100,000 pages/seconds\\n* Unlimited concurrency\\n* Simple API\\n* 50,000 RPM\\n### Do more with AI\\n* Browser scripting\\n* Advanced extraction\\n* Data pipelines\\n* Ideal for LLMs and AI Agents\\n* Accurate labeling\\n## Achieve more with these new API features\\nOur API is set to stream so you can act in realtime.\\n![A user interface with a search bar containing the text &#34;Latest sports news,&#34; a green &#34;Submit&#34; button, and two icon buttons to display searching and extracting with the service.](https://spider.cloud/img/search_feature.webp)\\n### Search\\nGet access to search engine results from anywhere and easily crawl and transform pages to\\nLLM-ready markdown.\\n[Explore Search](https://spider.cloud/docs/api#search)\\n![A user interface segment showing three icons representing different stages of data transformation.](https://spider.cloud/img/transform_feature_example.webp)\\n### Transform\\nConvert raw HTML into markdown easily by using this API. Transform thousands of html pages\\nin seconds.\\n[Explore Transform](https://spider.cloud/docs/api#transform)\\n## Join the community\\nBacked by a network of early advocates, contributors, and supporters.\\n[GitHub discussions\\n](https://github.com/orgs/spider-rs/discussions)[Discord\\n](https://discord.spider.cloud)\\n[\\n![iammerrick's avatar](https://spider.cloud/img/external/iammerrick_twitter.webp)\\n@iammerrick\\nRust based crawler Spider is next level for crawling &amp; scraping sites. So fast.\\nTheir cloud offering is also so easy to use. Good stuff. https://github.com/spider-rs/spider\\n](https://twitter.com/iammerrick/status/1787873425446572462)\\n[\\n![WilliamEspegren's avatar](https://spider.cloud/img/external/william_twitter.webp)\\n@WilliamEspegren\\nWeb crawler built in rust, currently the nr1 performance in the world with crazy resource management Aaaaaaand they have a cloud offer, that&rsquo;s wayyyy cheaper than any competitor\\nName a reason for me to use anything else?\\ngithub.com/spider-rs/spid&hellip;\\n](https://twitter.com/WilliamEspegren/status/1789419820821184764)\\n[\\n![gasa's avatar](https://spider.cloud/img/external/gaza_twitter.webp)\\n@gasa\\n@gasathenaper\\nis the best crawling tool i have used. I had a complicated project where i needed to paste url and get the website whole website data. Spider does it in an instant\\n](https://x.com/gasathenaper/status/1810612492596383948)\\n[\\n![Ashpreet Bedi's avatar](https://spider.cloud/img/external/ashpreet_bedi.webp)\\n@Ashpreet Bedi\\n@ashpreetbedi\\nis THE best crawler out there, give it a try\\n](https://x.com/ashpreetbedi/status/1815512219003572315?s=46&t=37F5QP_8oKqOsNpHSo6VVw)\\n[\\n![Troyusrex's avatar](https://spider.cloud/img/external/troy_twitter.webp)\\n@Troyusrex\\nI found a new tool, Spider-rs, which scrapes significantly faster and handles more scenarios than the basic scraper I built did. Our use of Spider-rs and AWS infrastructure reduced the scraping time from four months to under a week.\\n](https://medium.com/@troyusrex/inside-my-virtual-college-advisor-a-deep-dive-into-rag-ai-and-agent-technology-84731b2928f7#1326)\\n[\\n![Dify.AI's avatar](https://spider.cloud/img/external/difyai.webp)\\n@Dify.AI\\n&#128375;&#65039;Spider @spider\\\\_rust\\ncan be used as a built-in tool in #Dify Workflow or as an LLM-callable tool in Agent. It allows fast and affordable web scraping and crawling when your AI applications need real-time web data for context.\\n](https://x.com/dify_ai/status/1818226971056243089)\\n## FAQ\\nFrequently asked questions about Spider.\\n### What is Spider?\\nSpider is a leading web crawling tool designed for speed and cost-effectiveness, supporting various data formats including LLM-ready markdown.\\n### Why is my website not crawling?\\nYour crawl may fail if it requires JavaScript rendering. Try setting your request to &#x27;chrome&#x27; to solve this issue.\\n### Can you crawl all pages?\\nYes, Spider accurately crawls all necessary content without needing a sitemap.\\n### What formats can Spider convert web data into?\\nSpider outputs HTML, raw, text, and various markdown formats. It supports`JSON`,`JSONL`,`CSV`, and`XML`for API responses.\\n### Is Spider suitable for large scraping projects?\\nAbsolutely, Spider is ideal for large-scale data collection and offers a cost-effective dashboard for data management.\\n### How can I try Spider?\\nPurchase credits for our cloud system or test the Open Source Spider engine to explore its capabilities.\\n### Does it respect robots.txt?\\nYes, compliance with robots.txt is default, but you can disable this if necessary.\\n### Unable to get dynamic content?\\nIf you are having trouble getting dynamic pages, try setting the request parameter to &quot;&quot;chrome&quot;&quot; or &quot;&quot;smart.&quot;&quot; You may also need to set `disable\\\\_intercept` to allow third-party or external scripts to run.\\n### Why is my crawl going slow?\\nIf you are experiencing a slow crawl, it is most likely due to the robots.txt file for the website. The robots.txt file may have a crawl delay set, and we respect the delay up to 60 seconds.\\n### Do you offer a Free Trial?\\nYes, you can try out the service before being charged for free at[checkout](https://spider.cloud/credits/new?free-trial=1).\\n## Comprehensive Data Curation for Everyone\\nTrusted by leading tech businesses worldwide to deliver accurate and insightful data solutions.\\n[Zapier](https://zapier.com/apps/spider/integrations)\\n### Next generation data for AI, scale to millions\\n[Start now](https://spider.cloud/credits/new)\\n### Company\\n* [About](https://spider.cloud/about)\\n* [Privacy](https://spider.cloud/privacy)\\n* [Terms](https://spider.cloud/eula)\\n* [FAQ](https://spider.cloud/faq)\\n### Resources\\n* [API](https://spider.cloud/docs/api)\\n* [Docs](https://spider.cloud/docs/overview)\\n* [Guides](https://spider.cloud/guides)\\n* [Spider.rs Docs](https://docs.rs/spider/latest/spider/)\\n### Services\\n* [Pricing](https://spider.cloud/credits/new)\\n* [Web Crawling and Scraping](https://spider.cloud/web-crawling-and-scraping)\\n[All systems normal.](https://spidercloud.statuspage.io/)\\n[\\nGitHub\\n](https://github.com/spider-rs/spider)[\\nDiscord\\n](https://discord.spider.cloud)[\\nTwitter\\n](https://twitter.com/spider_rust)\", 'type': 'Document'}]}\n",
      "--------------------------------------------------\n",
      "Transition type:  finish\n",
      "Transition output:  Spider.cloud is a leading web crawling tool designed for AI applications, offering high-speed, scalable, and cost-effective data collection solutions. Built in Rust, it can crawl over 20,000 pages in seconds, making it significantly faster and cheaper than traditional scrapers. Spider supports various data formats, including LLM-ready markdown, and integrates seamlessly with major AI tools. It offers features like auto proxy rotations, custom browser scripting, and caching to enhance performance. Users can start with $200 in credits and explore features through a free trial. Spider is trusted by tech businesses worldwide for insightful data solutions.\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Lists all the task steps that have been executed up to this point in time\n",
    "transitions = client.executions.transitions.list(execution_id=execution.id).items\n",
    "\n",
    "# Transitions are retreived in reverse chronological order\n",
    "for transition in reversed(transitions):\n",
    "    print(\"Transition type: \", transition.type)\n",
    "    print(\"Transition output: \", transition.output)\n",
    "    print(\"-\"*50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the same task with a different URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the same code to run the same task, but with a different URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "execution = client.executions.create(\n",
    "    task_id=TASK_UUID,\n",
    "    input={\n",
    "        \"url\": \"https://www.harvard.edu/\"\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Harvard University's website emphasizes its commitment to excellence in teaching, learning, and research\n",
      "It highlights initiatives related to food, including nutrition, sustainability, and healthful eating\n",
      "The site features experts like Christina Warinner and Leah Penniman, and initiatives like the Harvard Food Systems Initiative and Food Literacy Project\n",
      "It explores topics such as junk food cravings, vegan diets, and the impact of avocados on heart disease\n",
      "The site also showcases Harvard's efforts in sustainable food practices, food donation programs, and educational resources like free online cooking courses\n",
      "Additionally, it highlights the contributions of chefs within the Harvard community.\n"
     ]
    }
   ],
   "source": [
    "execution = client.executions.get(execution.id)\n",
    "print(\"\\n\".join(execution.output.split(\". \")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:olive;\">Note: you can get the output of the crawling step by accessing the corresponding transition's output from the transitions list.</span>\n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'documents': [{'id': None,\n",
       "   'metadata': {'description': 'Harvard University is devoted to excellence in teaching, learning, and research, and to developing leaders who make a difference globally.',\n",
       "    'domain': 'www.harvard.edu',\n",
       "    'extracted_data': None,\n",
       "    'file_size': 15310,\n",
       "    'keywords': ['Business School podcast',\n",
       "     'Christina Warinner Christina co-authored',\n",
       "     'Dental Medicine shares advice',\n",
       "     'Dining Services Learnings Report Learn',\n",
       "     'Dining Services team',\n",
       "     'Education alum started Bite Sized Education',\n",
       "     'Expand Image Joanne Chang Stephanie Mitchell',\n",
       "     'Expand Image Julia Child Paul Child Julia Child',\n",
       "     'Expand Image Ludger Wessels Chef Wessels',\n",
       "     'Expand Image Nick DiGiovanni Kris Snibbe',\n",
       "     'Expand Image Nisha Vora Photo',\n",
       "     'Flour Bakery owner Joanne Chang',\n",
       "     'Food Donation Program',\n",
       "     'Food Food nourishes',\n",
       "     'Food Law',\n",
       "     'Food Literacy Project',\n",
       "     'Food Literacy Project hosts',\n",
       "     'Free cooking courses Learn',\n",
       "     'Graduate School',\n",
       "     'Harvard Chan School',\n",
       "     'Harvard Chan School found',\n",
       "     'Harvard College first-year student explains',\n",
       "     'Harvard College student',\n",
       "     'Harvard College student stories',\n",
       "     'Harvard College student story',\n",
       "     'Harvard Divinity School explores',\n",
       "     'Harvard Food Systems Initiative',\n",
       "     'Harvard Food Systems Initiative Led',\n",
       "     'Harvard Square food scene',\n",
       "     'Harvard Staff Photographer',\n",
       "     'Harvard University Dining Services',\n",
       "     'Harvard University Dining Services launched',\n",
       "     'Harvard alum',\n",
       "     'Harvard alum Marcos Barrozo worked',\n",
       "     'Healthful Food Standards',\n",
       "     'Healthy Eating Plate',\n",
       "     'Historian Joyce Chaplin explored',\n",
       "     'Ingestible insights Harvard researchers',\n",
       "     'James Beard Leadership Award winner',\n",
       "     'Joanne Chang wears',\n",
       "     'Julia Child pours hot syrup',\n",
       "     'Leah Gose worked',\n",
       "     'Leah Penniman Leah',\n",
       "     'Luke MacQueen spent years creating',\n",
       "     'Michael Marquand Law School grad Nisha Vora left',\n",
       "     'Prevention Research Center',\n",
       "     'Sustainable Food Harvard',\n",
       "     'Sustainable Food Systems Graduate Certificate',\n",
       "     'Teaching science concepts',\n",
       "     'accelerating food shortages',\n",
       "     'ancient humans adapted',\n",
       "     'biggest problems facing farmers',\n",
       "     'chef Ludger Wessels',\n",
       "     'co-founded Soul Fire Farm',\n",
       "     'converting food waste',\n",
       "     'crave junk food',\n",
       "     'crave junk food Harvard Medical School',\n",
       "     'cultivate healthier eating habits',\n",
       "     'engage secondary students',\n",
       "     'exploring medical technology',\n",
       "     'food Graduate School alum John Ahrens cultivated',\n",
       "     'food Research shows',\n",
       "     'food connoisseur shares',\n",
       "     'food insecurity solutions',\n",
       "     'free online courses',\n",
       "     'gas fermentation-derived chocolate',\n",
       "     'heart disease risk',\n",
       "     'inspire elevated thinking',\n",
       "     'lab-grown fish fillets',\n",
       "     'leading authority providing science-based guidance',\n",
       "     'local nonprofit Food',\n",
       "     'peach industry fight brown rot',\n",
       "     'postmenopausal weight loss',\n",
       "     'reduce food waste',\n",
       "     'reduce greenhouse gas emissions',\n",
       "     'regenerative farming practices',\n",
       "     'seasonal rainfall patterns',\n",
       "     'selling meat-free meals',\n",
       "     'shape future food systems leaders',\n",
       "     'six-time American Culinary Federation competition award winner',\n",
       "     'specialties include regional German',\n",
       "     'vegetarian restaurant Clover'],\n",
       "    'pathname': '/',\n",
       "    'resource_type': '.md',\n",
       "    'title': 'Harvard University',\n",
       "    'url': '8475428e-4e0c-44de-967f-c14fb73cf490/www.harvard.edu/www_edu/12638123428881205758.md',\n",
       "    'user_id': '8475428e-4e0c-44de-967f-c14fb73cf490'},\n",
       "   'page_content': '[Skip to main content](#main-content)\\n[Harvard University](https://www.harvard.edu/)\\nSearch\\nMenu\\n[Harvard University](https://www.harvard.edu/)\\nClose\\n# Harvard University\\n## Food\\nFood nourishes us, inspires us, and brings us together. The Harvard community is exploring nutrition, sustainability, and the science behind the things we eat.\\n## [Tips for your teeth](https://hsdm.harvard.edu/news/maintaining-your-oral-health-during-holiday-season)\\nThe Harvard School of Dental Medicine shares advice for keeping your teeth healthy between holiday meals.\\n![]()\\n## Experts in the edible\\n### Christina Warinner\\nChristina co-authored a study showing that ancient humans adapted to eating starch-rich foods as far back as 100,000 years ago. These foods likely helped pave the way for the expansion of the human brain.\\nLearn more about the study\\n[Learn more about the study](https://news.harvard.edu/gazette/story/2021/05/study-explains-early-humans-ate-starch-and-why-it-matters/)\\n### Ayr Muir\\nAs the founder of the vegetarian restaurant Clover, Ayr is selling meat-free meals and reducing our impact on climate change one sandwich at a time.\\nExplore the Business School podcast\\n[Explore the Business School podcast](https://www.alumni.hbs.edu/stories/Pages/story-bulletin.aspx?num=7302)\\n### Leah Penniman\\nLeah, a James Beard Leadership Award winner, co-founded Soul Fire Farm to train people in regenerative farming practices that are now the go-to methods of sustainable and organic agriculture.\\nLearn more about her farm\\n[Learn more about her farm](https://hds.harvard.edu/news/2019/09/18/leah-penniman-fight-food-justice)\\n![](https://www.harvard.edu/wp-content/themes/core/assets/img/theme/shims/16x9.png)\\nTeaching science concepts through cooking\\n[Click to Play Video](https://www.youtube.com/watch?v=cOJYELbyYGg)\\n### Kate Strangfeld\\nThe Graduate School of Education alum started Bite Sized Education to engage secondary students in science. Her goal is to empower students to &ldquo;think like a scientist&rdquo; through food and cooking.\\n## Ingestible insights\\nHarvard researchers are exploring the effects that certain foods and types of eating have on our health.\\n[Learn more about what you should eat](https://nutritionsource.hsph.harvard.edu/what-should-you-eat/)\\n![Two cartoon heads, one with a soda inside it and one with a broccoli](https://www.harvard.edu/wp-content/uploads/2024/11/this_or_that-junk-food-craving-wondering2.png?w=736&#038;h=491&#038;crop=1)### Why we crave junk food\\nHarvard Medical School&#8217;s Uma Naidoo, the author of the books &ldquo;This Is Your Brain on Food&rdquo; and &ldquo;Calm Your Mind with Food,&rdquo; explains why we crave junk food and how to cultivate healthier eating habits.\\n[Why we crave junk food](https://news.harvard.edu/gazette/story/2024/09/why-do-we-crave-junk-food-diet-psychology/)\\n### Chocolate\\nand its effect on postmenopausal weight loss\\n![A bar of chocolate](https://www.harvard.edu/wp-content/uploads/2024/11/chocolate-1277002_1280.jpg?w=375&#038;h=281&#038;crop=1)[Chocolate](https://news.harvard.edu/gazette/story/2021/06/starting-the-day-off-with-chocolate-may-have-unexpected-benefits/)\\n### Ultra-processed food\\nand its influence on obesity and disease\\n![Nuggets and fries](https://www.harvard.edu/wp-content/uploads/2024/11/chicken-nuggets-246180_1280.jpg?w=375&#038;h=281&#038;crop=1)[Ultra-processed food](https://news.harvard.edu/gazette/story/2023/12/why-are-americans-so-sick-researchers-point-to-middle-grocery-aisles/)\\n### Vegan diet\\nand Alzheimer&#8217;s improvements\\n![A colorful salad](https://www.harvard.edu/wp-content/uploads/2024/11/food-1075228_1280.jpg?w=375&#038;h=281&#038;crop=1)[Vegan diet](https://news.harvard.edu/gazette/story/2024/07/alzheimers-study-finds-diet-lifestyle-changes-yield-improvements/)\\n### Avocados\\nand their effect on heart disease risk\\n![avocados](https://www.harvard.edu/wp-content/uploads/2024/11/avocado-8498520_1280.jpg?w=375&#038;h=281&#038;crop=1)[Avocados](https://news.harvard.edu/gazette/story/2022/04/an-avocado-a-week-may-lower-heart-disease-risk/)\\n### Late-night eating\\nand its role in depression and mood\\n![A person looking into a fridge at night](https://www.harvard.edu/wp-content/uploads/2024/11/AdobeStock_254026165.jpeg?w=375&#038;h=281&#038;crop=1)[Late-night eating](https://hms.harvard.edu/news/daytime-eating-mental-health)\\nHome cooking\\n## Chefs in our community\\n[Explore some of the amazing dishes from chefs all over Harvard](https://www.harvard.edu/in-focus/food/recipes/)\\nMeet the chefs\\n![Joanne Chang wears an apron in a kitchen](https://www.harvard.edu/wp-content/uploads/2024/11/020320_Chang_Joanne_09b.jpg?w=731)\\nExpand Image\\nJoanne Chang\\nStephanie Mitchell/Harvard Staff Photographer\\n&quot;Pastry Love,&quot; a cookbook by Harvard alum and Flour Bakery owner Joanne Chang, honors some of her decadent and delicious favorites.\\n[Explore one of Joanne&#039;s recipes](https://www.harvard.edu/in-focus/food/recipes#joanne)\\n![A student wearing an apron stands in a kitchen with a cutting board of purple cabbage in front of him](https://www.harvard.edu/wp-content/uploads/2022/10/062629_Chef_006-e1675280860455.jpg?w=1200)\\nExpand Image\\nNick DiGiovanni\\nKris Snibbe/Harvard Staff Photographer\\nWhile at Harvard College, Nick DiGiovanni, who competed on MasterChef, created his own concentration in food and climate.\\n[Explore one of Nick&#039;s recipe](https://www.harvard.edu/in-focus/food/recipes#nick)\\n![Julia Child pours hot syrup over a Croquembouche in a kitchen](https://www.harvard.edu/wp-content/uploads/2022/10/Screen-Shot-2022-10-25-at-8.49.09-AM.png?w=634)\\nExpand Image\\nJulia Child\\nPaul Child\\nJulia Child&rsquo;s television series &ldquo;The French Chef,&rdquo; which first aired in 1963, launched a revolution in cooking and eating in the United States. Her papers are part of the collection at the Schlesinger Library.\\n[Explore one of Julia&#039;s recipes](https://www.harvard.edu/in-focus/food/recipes#julia)\\n![A woman stands in a kitchen](https://www.harvard.edu/wp-content/uploads/2022/10/Nisha_Portrait_2500-copy-e1675280779442.jpg?w=1500)\\nExpand Image\\nNisha Vora\\nPhoto by Michael Marquand\\nLaw School grad Nisha Vora left the legal life for vegan cooking.\\n[Explore one of Nisha&#039;s recipes](https://www.harvard.edu/in-focus/food/recipes#nisha)\\n![Headshot of chef Ludger Wessels against a brick wall](https://www.harvard.edu/wp-content/uploads/2022/10/LudgerWessels-e1675281023551.jpg?w=1600)\\nExpand Image\\nLudger Wessels\\nChef Wessels, a six-time American Culinary Federation competition award winner, is executive chef at Harvard University Dining Services. His specialties include regional German and French cuisine.\\n[Explore one of Ludger&#039;s recipes](https://www.harvard.edu/in-focus/food/recipes#ludger)\\n## Further into food\\n### The Nutrition Source\\nA leading authority providing science-based guidance on food and nutrition. Explore articles, recipes, and tools.\\nExplore the resources\\n[Explore the resources](https://nutritionsource.hsph.harvard.edu/)\\n### Free cooking courses\\nLearn about fermentation or explore the chemistry and physics of cooking in these free online courses.\\nSign up for a free cooking course\\n[Sign up for a free cooking course](https://pll.harvard.edu/catalog?keywords=cooking)\\n### The sacredness of food\\nThe Harvard Divinity School explores the many ways that food overlaps with religions around the world.\\nLearn more from the Divinity School\\n[Learn more from the Divinity School](https://news-archive.hds.harvard.edu/news/2016/11/22/sacredness-food)\\n## Sustenance solutions\\nThe Harvard community is exploring solutions for the biggest problems facing farmers, consumers, and the planet.\\n### The need for food\\nReserachers at Harvard Chan School found an[increase in food insufficiency](https://www.hsph.harvard.edu/news/press-releases/)following the decrease in SNAP benefits; Graduate School of Design[students looked at food insecurity solutions in Mississippi](https://news.harvard.edu/gazette/story/newsplus/designing-food-security-in-rural-mississippi/), ranked as America&rsquo;s hungriest state; Harvard alum[Leah Gose worked to improve how communities respond to hunger](https://gsas.harvard.edu/news/hungry-change), starting with her hometown of Atlanta; and Historian Joyce Chaplin explored[the long history of who gets to eat](https://hmsc.harvard.edu/2020/07/01/food-and-status-a-discussion-with-historian-joyce-chaplin/).\\n### The risks for food\\nResearch shows that changes to seasonal rainfall patterns are[accelerating food shortages](https://salatainstitute.harvard.edu/interview-unpredictable-rains-erode-food-security-gains/)across the globe; Harvard alum Marcos Barrozo worked to[lower emissions from Brazil&rsquo;s cattle industry](https://gsas.harvard.edu/news/beef-climate-change); the Wyss Institute is exploring medical technology to[help the peach industry fight brown rot](https://wyss.harvard.edu/media-post/arbor-armor-could-fcmbl-save-the-peach-industry/?q=arbor); and the Schools of Engineering and Public Health are teaming up to[reduce food waste and foodborne illness](https://seas.harvard.edu/news/2022/06/food-packaging-system-reduces-health-risks-and-saves-food)in a novel way.\\n### The future of food\\nGraduate School alum John Ahrens cultivated[lab-grown fish fillets](https://gsas.harvard.edu/news/reel-hope-ocean-conservation)to address the global demand for seafood; Wyss Institute&#8217;s Luke MacQueen spent years creating[an animal-free meat that tastes and feels like the real thing](https://wyss.harvard.edu/news/from-montreal-to-a-menu-near-you/); the founders of Circe created the[world&rsquo;s first gas fermentation-derived chocolate](https://otd.harvard.edu/news/circe-bioscience-licenses-technology-to-decarbonize-industry-with-microbes-developed-at-wyss-institute-at-harvard-university/)in response to the worldwide shortage of cocoa; and David Weitz&rsquo;s lab is exploring[converting food waste into sugar alternatives](https://salatainstitute.harvard.edu/salata-institute-funds-five-new-climate-research-projects/).\\n## Food at Harvard\\n![](https://www.harvard.edu/wp-content/uploads/2024/10/082922_Global_158-1-1168x934-1.jpg?w=500&#038;h=500&#038;crop=1)\\n### Healthful and Sustainable Food\\nHarvard&#039;s Office for Sustainability pledges to reduce greenhouse gas emissions from food by 25% by 2030 as part of the Coolfood Pledge.\\n[Learn more about Harvard&#039;s sustainable and healthful food](https://sustainable.harvard.edu/our-plan/how-we-operate/food/)\\n![](https://www.harvard.edu/wp-content/uploads/2024/10/Family-Meals-before-and-after-768x639b.jpg?w=500&#038;h=500&#038;crop=1)\\n### Food Donation Program\\nIn 2014, Harvard University Dining Services launched an effort to address chronic hunger among its neighbors in Cambridge and Boston by partnering with the local nonprofit Food for Free to donate nearly 2,000 nutritious meals each week to families in need.\\n[Learn more about the program](https://dining.harvard.edu/about-huds/sustainability/food-donation-program)\\n![](https://www.harvard.edu/wp-content/uploads/2024/10/Annenberg_0.jpeg?w=500&#038;h=500&#038;crop=1)\\n### Finding home through food\\nA Harvard College first-year student explains what the food is like at Harvard, and how to cure food-homesickness.\\n[Learn more from Harvard College student stories](https://college.harvard.edu/student-life/student-stories/finding-home-through-food)\\n![](https://www.harvard.edu/wp-content/uploads/2024/10/MG_5399-1-1024x683-1.jpg?w=500&#038;h=500&#038;crop=1)\\n### Harvard Food Systems Initiative\\nLed by Harvard University Dining Services in collaboration with Harvard Faculty and practitioners in the field, Harvard Food Systems Initiative is an educational program to inspire elevated thinking and change to shape future food systems leaders for a more sustainable future.\\n[Learn more about the initiative](https://hfsi.harvard.edu/)\\n![](https://www.harvard.edu/wp-content/uploads/2024/10/img-2670_1-1.jpg?w=500&#038;h=500&#038;crop=1)\\n### The Harvard Square food scene\\nA Harvard College student and food connoisseur shares the best places to grab a bite in the square.\\n[Learn more from this Harvard College student story](https://college.harvard.edu/student-life/student-stories/harvard-square-food-scene)\\n![](https://www.harvard.edu/wp-content/uploads/2024/10/061824_Farmers_Market_Opening_052.jpg?w=500&#038;h=500&#038;crop=1)\\n### Food Literacy Project\\nThe Food Literacy Project hosts a fellowship program for students to help cultivate an understanding of food from the ground up. Education focuses on sustainability, nutrition, food preparation and community.\\n[Learn more about the program](https://dining.harvard.edu/food-literacy-project)\\n![](https://www.harvard.edu/wp-content/uploads/2024/11/LEARNINGS-REPORT-PRINT-AUG30LR_24_25.jpg?w=624&#038;h=624&#038;crop=1)\\n### Dining Services Learnings Report\\nLearn how Harvard&#039;s Dining Services team has created places and experiences that are formative to how Harvard lives on in the memories of our students.\\n[Explore the report](https://dining.harvard.edu/about-huds/learnings-reports)\\nExplore the food and nutrition organizations, exhibits, and collections at Harvard.\\n[Food Law and Policy Clinic](https://hls.harvard.edu/clinics/in-house-clinics/food-law-and-policy-clinic/)\\n[The Nutrition Source](https://www.hsph.harvard.edu/nutritionsource/)\\n[Harvard University Dining Services](https://dining.harvard.edu/)\\n[Harvard Food Systems Initiative (HFSI)](https://hfsi.harvard.edu/)\\n[Healthy Eating Plate](https://www.hsph.harvard.edu/nutritionsource/healthy-eating-plate/)\\n[Historic Cookbooks at Schlesinger Library\\n](https://guides.library.harvard.edu/schlesinger/historic_cookbooks)\\n[Food Literacy Project](https://dining.harvard.edu/food-literacy-project/flp-programs-events)\\n[Harvard Farmers&#8217; Market](https://dining.harvard.edu/farmers-market)\\n[Sustainable Food Systems Graduate Certificate](https://extension.harvard.edu/academics/programs/sustainable-food-systems-graduate-certificate/)\\n[Prevention Research Center on Nutrition and Physical Activity at the Harvard Chan School (HPRC)](https://www.hsph.harvard.edu/prc/)\\n[Sustainable and Healthful Food Standards](https://green.harvard.edu/topics/food)\\nYOU MAY ALSO LIKE\\n## Related In Focus topics\\n* [Sleep](https://www.harvard.edu/in-focus/sleep/)\\n* [Healthy living](https://www.harvard.edu/in-focus/healthy-living/)\\n* [Mindfulness &#038; Meditation](https://www.harvard.edu/in-focus/mindfulness-meditation/)\\n## Security &amp; Brand\\n## Website\\n## Get In Touch\\nCopyright &copy;2024 The President and Fellows of Harvard College\\n[![Harvard University](https://www.harvard.edu/wp-content/themes/core/assets/img/theme/branding-assets/footer-logo.svg)](https://www.harvard.edu/)\\n* [Instagram![Instagram](https://www.harvard.edu/wp-content/uploads/2023/11/Instagram-1.png)](https://www.instagram.com/harvard/)\\n* [TikTok![TikTok](https://www.harvard.edu/wp-content/uploads/2023/11/tiktok-1.png)](https://www.tiktok.com/@harvard)\\n* [LinkedIn![LinkedIn](https://www.harvard.edu/wp-content/uploads/2023/11/Linkedin-1.png)](https://www.linkedin.com/school/harvard-university)\\n* [Facebook![Facebook](https://www.harvard.edu/wp-content/uploads/2023/11/FB3.png)](https://www.facebook.com/Harvard/)\\n* [YouTube![YouTube](https://www.harvard.edu/wp-content/uploads/2023/11/youtube-1.png)](https://www.youtube.com/harvard)',\n",
       "   'type': 'Document'}]}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transitions = client.executions.transitions.list(execution_id=execution.id).items\n",
    "\n",
    "transitions[1].output"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}