People are starting to use spliceAI on our HPC infrastructure. While running we noticed that TEMP folders are created with multiple millions (~4M) files each only a few kb.
In a HPC system with a massive parallel filesystem this is creating massive overhead.
Is there a way to change this behaviour? So that these TEMP files are combined in any way or switch to another method?
For example storing this data in a local database would improve performance massively.