[Bug Report] No checkpoints in the directory error when resume training with --video set #1209
Description
Describe the bug
An error of No checkpoints in the directory will throw when resume from a previous training with --video set. This is because a new log folder will be created before the check.
Steps to reproduce
To resume a training from the last training with --video
python source/standalone/workflows/rsl_rl/train.py --task Isaac-Repose-Cube-Allegro-v0 --headless \
--num_env 12 \
--resume True --videoError message
Error executing job with overrides: []
Traceback (most recent call last):
File "/home/weiy/Projects/code/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/utils/hydra.py", line 91, in hydra_main
func(env_cfg, agent_cfg, *args, **kwargs)
File "/home/weiy/Projects/code/IsaacLab/source/standalone/workflows/rsl_rl/train.py", line 128, in main
resume_path = get_checkpoint_path(log_root_path, agent_cfg.load_run, agent_cfg.load_checkpoint)
File "/home/weiy/Projects/code/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/utils/parse_cfg.py", line 192, in get_checkpoint_path
raise ValueError(f"No checkpoints in the directory: '{run_path}' match '{checkpoint}'.")
ValueError: No checkpoints in the directory: '/home/weiy/Projects/code/IsaacLab/logs/rsl_rl/allegro_cube/2024-10-10_12-11-44' match 'model_.*.pt'.
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.System Info
Describe the characteristic of your environment:
- Commit: cb9fee6
- Isaac Sim Version: 4.2
- OS: Ubuntu 24.04.1
- GPU: RTX 3090
- CUDA: 12.4
- GPU Driver: 550.54.15
Additional context
NA
Checklist
- I have checked that there is no similar issue in the repo (required)
- I have checked that the issue is not in running Isaac Sim itself and is related to the repo
Acceptance Criteria
Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.
- Criteria 1
- Criteria 2