Since 2.23.0 main server processes does not prepare data chunks anymore. The process is done in dedicated workers.

Make sure you have at least one active pod cvat_worker_chunks.

It is not really intuitive, hovewer in this specific case the code means, that this worker was not able to prepare chunk in one minute. It may refer to different things (you do not have such workers or all these workers are busy).

That approach was chosen to keep API compatibility.

Related #8781

Hey @bsekachev, thank you for the response!

This is what cvat-backend-worker-chunks logs say:

Traceback (most recent call last):
  File "/home/django/manage.py", line 20, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute
    settings.INSTALLED_APPS
  File "/opt/venv/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__
    self._setup(name)
  File "/opt/venv/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup
    self._wrapped = Settings(settings_module)
  File "/opt/venv/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/django/cvat/settings/production.py", line 5, in <module>
    from .base import *
  File "/home/django/cvat/settings/base.py", line 417, in <module>
    os.makedirs(EVENTS_LOCAL_DB_ROOT, exist_ok=True)
  File "/usr/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/django/data/cache/events'

I saw this in my "broken" setup, reinstalled the chart (i.e. removed all volume claims and the namespace in k8s) and still see these errors.

I started from scratch (I believe), created a new task, and this is what I see in the annotation interface:

It is able to load first 8, but not more. The same 429 in the logs.

I tried what appears to be the solution from #8781 (comment) (i.e. set accessModes to ReadWriteMany for cvat-kvrocks-data and cvat-backend-data PVCs), but it made no difference.

May be there is something useful in pod logs?

Not really, but then again I would not know which pod is important for this (the chunks one only has this traceback repeated over and over again, the server only the 429 and some 200 responses, others also do not seem to have important clues).

I switched to v2.22.0 though, it seems to work, lol.

Hi team,

I’m encountering a similar issue when trying to annotate images in CVAT, and I wanted to share my experience and findings.

Actions Before Raising This Issue

I searched through existing issues and did not find anything similar.
I reviewed the documentation for any relevant information.

Steps to Reproduce

While this isn't a strict reproducer, here’s what I did that led to the issue:

Uploaded 19 4K .png files to a project.
Annotated approximately 7 of them.
Found myself unable to move to the remaining images.
Expected Behavior
I expected everything to work seamlessly by default.

Context

I am facing a uvicorn error when trying to annotate images. Here’s a relevant line from the cvat-backend logs:

Verify

Open In Editor
Run

2024-12-09 18:43:25,606 DEBG 'uvicorn-0' stdout output:
INFO:     10.15.0.40:0 - "GET /api/jobs/1/data?org=&quality=compressed&type=chunk&index=0 HTTP/1.0" 429 Too Many Requests

It’s puzzling because I’m just a single user trying to annotate images through the UI, and I’m not sure why there are too many requests.

Environment
I’m using the provided Helm chart from commit 12eec54.

Additional Information
I noticed that since version 2.23.0, the main server processes no longer prepare data chunks; this is now handled by dedicated workers. I made sure that at least one cvat_worker_chunks pod is active, but it seems that the worker is unable to prepare chunks in a timely manner, possibly due to being busy or not existing.

I also encountered a PermissionError in the cvat-backend-worker-chunks logs, indicating that it cannot create directories due to permission issues:

Verify

Open In Editor
Run

PermissionError: [Errno 13] Permission denied: '/home/django/data/cache/events'

After reinstalling the chart and removing all volume claims and the namespace in Kubernetes, I still see these errors. I even tried setting the access modes to ReadWriteMany for the cvat-kvrocks-data and cvat-backend-data PVCs, but that did not resolve the issue.

I switched back to version 2.22.0, and it seems to work without these problems.

If anyone has insights or suggestions on how to resolve this, I would greatly appreciate it. Thank you for your help!

Best,
Sowmithri Sriram.
sriramsowmithri9807 --> github