Skip to content

Isaaclab simulation crashed when training, only report warnings #1450

Open
@privateer0744

Description

@privateer0744

When I’m running the tutorial example Modifying an existing Direct RL Environment, the simulation crashed and reported these warnings.

./isaaclab.bat -p source/standalone/workflows/rl_games/train.py --task Isaac-H1-Direct-v0 --headless

[INFO][AppLauncher]: Loading experience file: D:\projects\IsaacLab\source\apps\isaaclab.python.headless.kit
[Warning] [omni.isaac.kit.simulation_app] Modules: [‘omni.kit_app’] were loaded before SimulationApp was started and might not be loaded correctly.
[Warning] [omni.isaac.kit.simulation_app] Please check to make sure no extra omniverse or pxr modules are imported before the call to SimulationApp(…)
Loading user config located at: ‘c:/users/17985/.conda/envs/isaaclab/lib/site-packages/omni/data/Kit/Isaac-Sim/4.2/user.config.json’
[Info] [carb] Logging to file: c:/users/17985/.conda/envs/isaaclab/lib/site-packages/omni/logs/Kit/Isaac-Sim/4.2/kit_20241121_201412.log
2024-11-21 12:14:12 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn’t available.
2024-11-21 12:14:15 [3,055ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: Intel(R) Iris(R) Xe Graphics
2024-11-21 12:14:15 [3,055ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: Intel(R) Iris(R) Xe Graphics

[INFO]: Parsing configuration from: <class ‘omni.isaac.lab_tasks.direct.humanoid.h1_env.H1EnvCfg’>
[INFO]: Parsing configuration from: D:\projects\IsaacLab\source\extensions\omni.isaac.lab_tasks\omni\isaac\lab_tasks\direct\humanoid\agents\rl_games_ppo_cfg.yaml
[INFO] Logging experiment in directory: D:\projects\IsaacLab\logs\rl_games\humanoid_direct
Setting seed: 42
[INFO]: Base environment:
Environment device : cuda:0
Environment seed : 42
Physics step-size : 0.008333333333333333
Rendering step-size : 0.016666666666666666
Environment step-size : 0.016666666666666666
[INFO]: Time taken for scene creation : 2.677555 seconds
[INFO]: Scene manager:
Number of environments: 2
Environment spacing : 4.0
Source prim name : /World/envs/env_0
Global prim paths : [‘/World/ground’]
Replicate physics : True
[INFO]: Starting the simulation. This may take a few seconds. Please wait…
[INFO]: Time taken for simulation start : 0.540319 seconds
[INFO]: Completed setting up the environment…
self.seed = 42
Started to train
Exact experiment name requested from command line: 2024-11-21_20-14-19
seq_length: 4
current training device: cuda:0
Error executing job with overrides:
Traceback (most recent call last):
File “D:\projects\IsaacLab\source\standalone\workflows\rl_games\train.py”, line 178, in
main()
File “D:\projects\IsaacLab\source\extensions\omni.isaac.lab_tasks\omni\isaac\lab_tasks\utils\hydra.py”, line 102, in wrapper
hydra_main()
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\hydra\main.py”, line 94, in decorated_main
_run_hydra(
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\hydra_internal\utils.py”, line 394, in _run_hydra
_run_app(
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\hydra_internal\utils.py”, line 457, in _run_app
run_and_report(
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\hydra_internal\utils.py”, line 223, in run_and_report
raise ex
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\hydra_internal\utils.py”, line 220, in run_and_report
return func()
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\hydra_internal\utils.py”, line 458, in
lambda: hydra.run(
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\hydra_internal\hydra.py”, line 132, in run
_ = ret.return_value
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\hydra\core\utils.py”, line 260, in return_value
raise self._return_value
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\hydra\core\utils.py”, line 186, in run_job
ret.return_value = task_function(task_cfg)
File “D:\projects\IsaacLab\source\extensions\omni.isaac.lab_tasks\omni\isaac\lab_tasks\utils\hydra.py”, line 99, in hydra_main
func(env_cfg, agent_cfg, *args, **kwargs)
File “D:\projects\IsaacLab\source\standalone\workflows\rl_games\train.py”, line 170, in main
runner.run({“train”: True, “play”: False, “sigma”: train_sigma})
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\rl_games\torch_runner.py”, line 133, in run
self.run_train(args)
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\rl_games\torch_runner.py”, line 113, in run_train
agent = self.algo_factory.create(self.algo_name, base_name=‘run’, params=self.params)
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\rl_games\common\object_factory.py”, line 15, in create
return builder(**kwargs)
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\rl_games\torch_runner.py”, line 37, in
self.algo_factory.register_builder(‘a2c_continuous’, lambda **kwargs : a2c_continuous.A2CAgent(**kwargs))
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\rl_games\algos_torch\a2c_continuous.py”, line 15, in init
a2c_common.ContinuousA2CBase.init(self, base_name, params)
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\rl_games\common\a2c_common.py”, line 1143, in init
A2CBase.init(self, base_name, params)
File “C:\Users\17985.conda\envs\isaaclab\lib\site-packages\rl_games\common\a2c_common.py”, line 249, in init
assert(self.batch_size % self.minibatch_size == 0)
AssertionError
2024-11-21 12:14:22 [10,280ms] [Warning] [carb] Recursive unloadAllPlugins() detected!

I have transfered the assets to local server and the Unitree h1.usd model can be successfully loaded in demo script.
But the simulation runs normally when switch to another task with the same train.py script:
./isaaclab.bat -p source/standalone/workflows/rl_games/train.py --task Isaac-Humanoid-Direct-v0 --headless

System Info

Describe the characteristic of your environment:

  • Isaac Sim Version: [4.2.0]
  • OS: [Win11]
  • GPU: [RTX 3060 Geforce (6g VRAM)]
  • CUDA: [12.6]
  • GPU Driver: [561.17]

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions