-
Notifications
You must be signed in to change notification settings - Fork 313
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cuda out of memory #84
Comments
You're having the same issue as here. I am not sure what gpu specifications you are running it on, but theres a chance yours cannot handle the backpropagation. The only way I found to run the reactive net was to comment/edit the lines they described in this link. |
I have an interesting observation. You only encounter this problem when using the argument "--method 'reactive'", which only changes the network model. So I think the root of this problem is not lack of GPU memory (I use 3080 with 12GB memory, equal to that of Titan X). There might be something wrong with the reactive net model. |
It's a memory leak problem. The GPU memory usage increases per iteration. So no better how large memory your device has, the program will Eat It Up. I think it has something to do with the computation graph of the reactive_net, which caused the memory leak issue. |
I have the problem that the terminal show that CUDA out of menmory when I run "python main.py --is_sim --method 'reactive' --experience_replay --save_visualizations". However everything is ok when I run "python main.py --is_sim --push_rewards --experience_replay --explore_rate_decay --save_visualizations". Do you know this is why?
Thank you!
The text was updated successfully, but these errors were encountered: