My initial thought is that you may have been running parallel Jobs/Dataflows that both needed to write to cache. Run independently they would have worked fine. But together it was too much.
It could also have been other processes writing to D:.
I have jobs that write temp files as part of the processing. I had the admin create a separate area (drive) for the files to go to so that if I filled it up I wouldn’t cause problems with other jobs that were using normal caching, logs, etc.
True, multiple sys configs of this job are running.
I’ve dived in to the job and found several things that could be improved. Varying from better pushdown, stricter joins, better functional design etc. This brought down the load and the processing time a lot and now suddenly everything worked. Coming night it runs again in production and then we’ll for real.
I suspect it was a combination of a lot of things and in the end writing to the disk too fast with too many processes.
Unless with 2 instances of a job the same name for a cache is used? Does not seem likely, it looks like the PID is part of the file name.