So I think I figured it out, the backup job that also does the cleanup wasn't doing a full cleanup. It uses find to do the job and uses the following roots /var/lib/jenkins/*/builds/ and /var/lib/jenkins/*/configurations/axis-nodes/*/builds/ but the axis jobs have names other than 'axis-nodes' including 'axis-MACHINE', 'axis-label', 'axis-v8test' and a bunch of others. Basically any multi-axis job has a name corresponding to the axis selected. What's more, the build directories are pretty deep within some of these due to the way they are configured.

So, I've changed the find to effectively do this: find /var/lib/jenkins/ -depth -type d -regex '/var/lib/jenkins/.*/builds/[0-9]+' -mtime +7 -exec rm -rf '{}' \;. So it now matches any depth of build directory and finds the job subdirectories underneath that are older than 7 days.

Back to 7 days and we're now at 86% disk usage on the machine by including these extra build directories.

@jbergstroem and @joaocgreis you might want to check my work on that .. particularly the axis thing @joaocgreis since that's your wheelhouse.

Regarding block storage, it turns out the reason we haven't done this yet is that this machine is in DigitalOcean's SFO1 which doesn't do block storage, so we'd have to redeploy the machine in a new datacenter to get this functionality unfortunately.

Regarding access, some of these key machines are accessible only by the "infra" group which is myself, @jbergstroem, @mhdawson and @joaocgreis. ci-release, www, backup(s) and a few others are reserved for this group.

Regarding further flakiness since my last message, as per @refack's comment, that's my fault as I inadvertantly upgraded Jenkins when I shouldn't have and got us into the Java 8 requirement territory as @gibfahn outlined in #775 and a large chunk of the machines couldn't properly connect! I've downgraded it again and it seems to be back to normal but we may need a bit of time to flush out current work.

Uh oh!

CI flakiness #798

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions