Hi @johnseekins! Thank you for so detailed issue!
Could you please provide some additional screenshots form our dashboards:

• storage. LSM parts
• storage. Disk writes/reads
• storage. Active merges
• resource usage. Open FDs
• overview. Disk space used

Thank you!

active merges and lsm parts:

disk r/w and fds:

Disk used:

And a few other interesting panels:

There is definitely a huge spike in indexdb and small merges during that time, but they seem to be symptoms (as they spike up suddenly during the event and then taper off...)

Hm, the LSM parts graph suggests VM has about 6-8k parts on disk. And "vm_hdd Objects" (if I'm reading it right) suggests there 3kk objects were deleted over the 2.5h. Which does not correlate with amount of parts either merged or deleted by VM.
The spike in ActiveMerges can be explained by RowsRerouted graph - every time vmstorage receives new time series it results into new parts for index on disk and consequent merges...

Is there any chance that some type of cronjob enabled in the OS which runs every Monday at midnight? I can recall a similar issue with cronjob for fstrim process causing lags for our SSDs every week.

We can't find anything...but I'll check again. The one suspicious thing was an MD scan of the hypervisor's OS drive...but that only happens once a month...and triggers two hours after the event starts.

But fstrim on the boxes is scheduled for Monday at midnight!

root@store-1:~# systemctl status fstrim.timer
● fstrim.timer - Discard unused blocks once a week
     Loaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Wed 2021-06-16 14:08:16 UTC; 1 months 19 days ago
    Trigger: Mon 2021-08-09 00:00:00 UTC; 3 days left
   Triggers: ● fstrim.service
       Docs: man:fstrim

Jun 16 14:08:16 store-1 systemd[1]: Started Discard unused blocks once a week.

So drive trimming is causing this? That's strange, given the data drive is a ceph block device and the OS drive is an Openstack block device. But I'm ready to believe it!

I've disabled fstrim on all the storage nodes ('cause whether or not that's the actual problem, there's no reason for it to be running on these hosts).

This was fstrim. It may be worth documenting that fstrim can have an adverse affect on the system somewhere?

Hm, I'm not sure this is exactly relevant to VM. The case I mentioned happened to me 5y ago and it was a Postgres cluster suffering every Sunday because of fstrim. I was just lucky to recall it now and ask you to check if there is something similar in your system.
However, here's commit to mention fstrim in Tuning section.

It wasn't directly related to VictoriaMetrics, no. But disabling fstrim on these virtual hosts meant the weekly I/O spikes didn't happen.