Add pulse checks for ensuring disk health on VmHosts #2202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

mohi-kalantari merged 4 commits into ubicloud:main from mohi-kalantari:vmh-disk-monitoring

Feb 25, 2025

Contributor

mohi-kalantari commented Nov 6, 2024

Multiple checks using different linux tools are added to determine the health of the disks. The code also supports both raided and not raided VmHosts

github-actions bot commented Nov 6, 2024 •

edited

Loading

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

mohi-kalantari force-pushed the vmh-disk-monitoring branch from c74acaf to eede09c Compare

November 6, 2024 22:06

mohi-kalantari changed the title ~~Add pulse checks for ensuring disk health on VmHosts~~ WIP: Add pulse checks for ensuring disk health on VmHosts

mohi-kalantari force-pushed the vmh-disk-monitoring branch 3 times, most recently from cd3bc90 to e2bd310 Compare

November 8, 2024 08:55

aamederen reviewed

View reviewed changes

lib/filesystem_parser.rb Outdated Show resolved Hide resolved

rhizome/host/bin/prep_host.rb Outdated Show resolved Hide resolved

prog/vm/host_nexus.rb Outdated Show resolved Hide resolved

prog/vm/host_nexus.rb Outdated Show resolved Hide resolved

prog/vm/host_nexus.rb Outdated Show resolved Hide resolved

model/vm_host.rb Outdated Show resolved Hide resolved

lib/filesystem_parser.rb Outdated Show resolved Hide resolved

Contributor

aamederen commented Nov 8, 2024

Since this is also a warm-up, it could be nice to split this change into multiple commits and explain each change in the commit messages. The migration could be one commit, population of the UNIX devices could be another and finally the change in the health check logic, each with their own specs.

mohi-kalantari force-pushed the vmh-disk-monitoring branch 2 times, most recently from 453f521 to e1c64f0 Compare

November 12, 2024 11:44

Contributor Author

mohi-kalantari commented Nov 12, 2024

I have read the CLA Document and I hereby sign the CLA

Contributor Author

mohi-kalantari commented Nov 12, 2024

recheck

mohi-kalantari force-pushed the vmh-disk-monitoring branch 2 times, most recently from 17deb53 to a928c4a Compare

November 12, 2024 12:27

aamederen reviewed

View reviewed changes

model/vm_host.rb Outdated Show resolved Hide resolved

aamederen reviewed

View reviewed changes

prog/vm/host_nexus.rb Outdated Show resolved Hide resolved

aamederen reviewed

View reviewed changes

model/vm_host.rb Outdated Show resolved Hide resolved

aamederen reviewed

View reviewed changes

lib/system_parser.rb Outdated Show resolved Hide resolved

mohi-kalantari force-pushed the vmh-disk-monitoring branch from a928c4a to cd4381d Compare

November 13, 2024 10:22

aamederen reviewed

View reviewed changes

model/vm_host.rb Outdated Show resolved Hide resolved

aamederen reviewed

View reviewed changes

model/vm_host.rb Outdated Show resolved Hide resolved

aamederen reviewed

View reviewed changes

model/vm_host.rb Outdated Show resolved Hide resolved

mohi-kalantari force-pushed the vmh-disk-monitoring branch from cd4381d to f7f62f5 Compare

November 13, 2024 11:37

aamederen approved these changes

View reviewed changes

Contributor

aamederen left a comment

LGTM, I would however like to have one more eye on this

aamederen requested a review from a team

November 13, 2024 11:53

fdr requested changes

View reviewed changes

Collaborator

fdr left a comment

at a minimum, two things:

If storing an array, why not use either a text array, or even more likely, a jsonb
The migration should be broken out from other application changes to not make rollbacks fussy.

fdr reviewed

View reviewed changes

model/storage_device.rb Outdated Show resolved Hide resolved

fdr requested review from jeremyevans and byucesoy

November 19, 2024 02:44

fdr reviewed

View reviewed changes

prog/vm/host_nexus.rb Outdated Show resolved Hide resolved

Contributor Author

mohi-kalantari commented Nov 20, 2024

I'll also fix the commits and separate the migration commit and others when review is done. thanks for pointing out. @fdr

jeremyevans reviewed

View reviewed changes

Contributor

jeremyevans left a comment

Looks good. I only have one remaining comment.

lib/system_parser.rb Outdated Show resolved Hide resolved

mohi-kalantari force-pushed the vmh-disk-monitoring branch from 2ceaa28 to b5b05fe Compare

November 22, 2024 12:03

mohi-kalantari requested a review from jeremyevans

November 22, 2024 12:04

mohi-kalantari force-pushed the vmh-disk-monitoring branch from b5b05fe to aaf6c55 Compare

November 22, 2024 12:09

jeremyevans reviewed

View reviewed changes

Contributor

jeremyevans left a comment

Previous comment was addressed. One additional comment.

cache/schema.cache Outdated Show resolved Hide resolved

mohi-kalantari force-pushed the vmh-disk-monitoring branch 2 times, most recently from f00ed76 to df2f8ce Compare

November 25, 2024 06:31

mohi-kalantari force-pushed the vmh-disk-monitoring branch from df2f8ce to 02a5609 Compare

December 6, 2024 14:45

byucesoy reviewed

View reviewed changes

Member

byucesoy left a comment

Overall, looks OK IMO.

@mohi-kalantari, do we have any blocker for this PR, other than Daniel's comments?

mohi-kalantari force-pushed the vmh-disk-monitoring branch from 02a5609 to 84a5b06 Compare

January 8, 2025 15:15

mohi-kalantari force-pushed the vmh-disk-monitoring branch 4 times, most recently from 69517bf to 76b482d Compare

January 20, 2025 11:21

mohi-kalantari changed the title ~~WIP: Add pulse checks for ensuring disk health on VmHosts~~ Add pulse checks for ensuring disk health on VmHosts

mohi-kalantari force-pushed the vmh-disk-monitoring branch from 76b482d to 3bbdcfb Compare

February 21, 2025 11:47

fdr approved these changes

View reviewed changes

mohi-kalantari force-pushed the vmh-disk-monitoring branch 4 times, most recently from 1501ef3 to 6bc194f Compare

February 24, 2025 13:06

mohi-kalantari added 2 commits

February 24, 2025 14:35


          Add new field to storage_device table

66bc099

In order to determine and save the disk devices of each VmHost
We store an array of strings representing the device names. Also
updated the schema caches.


          Update parsers to extract disk device names

0ddddd3

Added functions for determining the disk devices and also
extracting the underlying device names of raided disks.

mohi-kalantari force-pushed the vmh-disk-monitoring branch from 6bc194f to db540db Compare

February 24, 2025 13:36

mohi-kalantari added 2 commits

February 24, 2025 14:52


          Install required packages for monitoring disk health

6ef9c97

smartmontools is installed when bootstrapping a VmHost. It will be used
as a healthcheck for disk devices of VmHosts.


          Implement health check logics on VmHost Disks

f8e147a

Multiple healthchecks were added to check disks using different tools
like smartctl, nvme cli, read/write on disk and reading kernel logs.

mohi-kalantari force-pushed the vmh-disk-monitoring branch from db540db to f8e147a Compare

February 24, 2025 13:52

mohi-kalantari merged commit 88cd29d into ubicloud:main

3 checks passed

github-actions bot locked and limited conversation to collaborators

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet