We started getting notices from our help desk that users were logging into some systems and getting an error message. We were starting to do our Windows 10 1607 to 1703 in place upgrades and thought that the MS installer must be doing something wrong. This time around, it wasn’t MS but something we had done to expose a “bug” in the EPM logging process.
Notice “bug” is in quotes. This is because I call it a bug, Ivanti has been notified of the issue and they can reproduce it. I’m not sure they will classify it as a “bug” but either way, hopefully this post helps you if you find the similar issue.
What we noticed is that in the users c:\windows\temp folder a single .tmp file was several GB in size. Sometimes the file showed 0KB and other times it actually showed the massive size. The difference seemed to be if the process was still running that was creating the file.
The short of it was that we were pushing a managed script to the clients that simply wrote a registry key. This key was used by an automated process to key off of the need to perform an in place upgrade. The code looked something like this:
REMEXEC01=<qt/>%SystemRoot%\Sysnative\reg.exe<qt/> ADD HKLM\Software\ITTechSmith\EPM\Windows10 /v OSUpgradeRequired /t REG_SZ /d PendingWhat we were able to identify is that the .tmp file was a log file that EPM was generating and it was logging the same output over and over again until the entire disk space was full. What was in the log?
“Value OSUpgradeRequired exists, overwrite(yes/no)?”
Over and over again. No idea why it kept logging the same output a trillion times to fill up on average 180GB of disk space, but it did.
The solution? Fix the code that writes the registry key and then go and fix all the devices currently affected. The new code was simple enough, just adding the /F at the end.
REMEXEC01=<qt/>%SystemRoot%\Sysnative\reg.exe<qt/> ADD HKLM\Software\ITTechSmith\EPM\Windows10 /v OSUpgradeRequired /t REG_SZ /d Pending /fMost of the systems we use require a remote fix. This was done over the network for each asset tag doing the following. RDP, Bomgar or other means to simply remote to the device failed since there was literally no disk space:
TAKSKILL /S SYSTEMNAME /IM reg.exeThe above command would kill off the process(es) writing the registry key and in turn EPM would hold a lock on the .tmp file. Once the process was killed off, the .tmp file lock was released. Then simply deleting all .tmp files in the c:\windows\temp directory freed up all of the disk space.
Once the disk space issue was resolved, we could proceed with the in place upgrade as normal.
Also note, several file scanning applications were not able to find and identify the offending file. To find the .tmp file, we were able to find a device that was not filling up the disk space any further because reg.exe had already been killed off with a reboot. We cleared off some space and WinDir Stat was able to find the temp file several gigs in size.
To find what was in the .tmp file, we had to move it to another system, then use MS DOS with the “type runABCD.tmp” command where runABCD.tmp was the file generated by EPM. 180GB was too big even for Notepad++.
