Question: #174
Description
What is your installation issue?
localcolabfold fails to run even tiny peptide sequences with cuda out of memory errors. I was encouraged by the README indicating that GPUs were optional. My environment has a small GPU built into the laptop (2Gb) of RAM, and I am testing with something small, but even that fails.
Computational environment
- OS: [WSL2]
- CUDA version if Linux (Show the output of
/usr/local/cuda/bin/nvcc --version.)
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
Output from colabfold_batch input.fa output with a < 25 aa peptide.
WARNING: You are welcome to use the default MSA server, however keep in mind that it's a
limited shared resource only capable of processing a few thousand MSAs per day. Please
submit jobs only from a single IP address. We reserve the right to limit access to the
server case-by-case when usage exceeds fair use. If you require more MSAs: You can
precompute all MSAs with `colabfold_search` or host your own API and pass it to `--host-url`
2023-09-01 15:31:25.996296: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-09-01 15:31:30.007547: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/gpu/asm_compiler.cc:114] *** WARNING *** You are using ptxas 10.1.243, which is older than 11.1. ptxas before 11.1 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.
2023-09-01 15:31:30.285635: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/gpu/asm_compiler.cc:114] *** WARNING *** You are using ptxas 10.1.243, which is older than 11.1. ptxas before 11.1 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.
2023-09-01 15:31:30.354863: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/gpu/asm_compiler.cc:114] *** WARNING *** You are using ptxas 10.1.243, which is older than 11.1. ptxas before 11.1 is known to miscompile XLA code, leading to incorrect results or invalid-address errors.
2023-09-01 15:31:39.206490: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-09-01 15:31:39.206559: E external/org_tensorflow/tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:695] could not allocate CUDA stream for context 0x3bb5e40: CUDA_ERROR_OUT_OF_MEMORY: out of memory
To Reproduce
Fresh installation in WSL2 with current install script and supporting nvidia libraries.
Set up environment variables as suggested
export TF_FORCE_UNIFIED_MEMORY="1"
export XLA_PYTHON_CLIENT_MEM_FRACTION="4.0"
export XLA_PYTHON_CLIENT_ALLOCATOR="platform"
export TF_FORCE_GPU_ALLOW_GROWTH="true"
Expected behavior
I was expecting it to attempt to fold my small input. I was hoping that should it have insufficient GPU ram on my modest GPU ( I think it has just 2Gb), that it would use regular RAM, or that there would be an option to ignore the GPU, but it instead seems to try to use the GPU and fail. This might be a) impossible b) something I'm missing in the docs, or c) an install problem?
thanks in advance
Darren
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02 Driver Version: 531.14 CUDA Version: 12.1 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro M620 On | 00000000:01:00.0 On | N/A |
| N/A 0C P0 N/A / N/A| 965MiB / 2048MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+