I will just add the points of note from the issue here:

• Attempting to use vLLM as the deployment backend yields a generic 500 error, as well as ray workers dying due to OOM errors, however transformers appears to work.
• There appears to be extremely low vRAM usage, even during inference, I am not sure if this is as intended.

Thanks for sharing this example!

I’ve reviewed your PR, and it looks great overall. Regarding your comments:

1. To use vllm as the deployment backend, you’ll need to apply a patch file. You can do this by running:

   conda activate sllm-worker
   ./serverless_llm/store/vllm_patch/patch.sh

2. The low vRAM usage during inference could be due to several factors, including model size, batch size, and the total available vRAM. Since this example uses a relatively small model (opt-1.3b), the low vRAM usage is expected.

Let me know if you have further questions!

Thanks for the quick reply, regarding your replies:

1. The main issue here is more that vLLM seems to have too much overhead for the colab's RAM resources, and seems to be patched in by default (running the .sh file to check returns true), but if you are fine with transformers as the backend for the demo that is what has been used.
2. That's fine then, again not a massive issue for this, just was a bit concerned as the model only seemed to use ~0.1GB VRAM so I was unsure whether the layers were being correctly offloaded to the GPU.

We’ve just released our first PyPI package, which significantly reduces installation time. Could you please update the notebook according to the latest installation guide?

2. That's fine then, again not a massive issue for this, just was a bit concerned as the model only seemed to use ~0.1GB VRAM so I was unsure whether the layers were being correctly offloaded to the GPU.

While using a small portion of VRAM is expected, 0.1GB is unusually low. For example, opt-1.3 loaded with fp16 should consume at least 2.6GB. This might be caused by a bug. Have you checked the nohup output? You can check it with !cat nohup.out.