Windows PGO Tailcall build failed with error MSB4166: Child node "6" exited prematurely

[vstinner]

Example on AMD64 Windows PGO Tailcall 3.x: https://buildbot.python.org/#/builders/1870/builds/341

    43>Link:
         Finished generating code
         _zstd.vcxproj -> C:\bbarea\3.x.itamaro-win64-srv-22-aws.tailcall.pgo\build\PCbuild\amd64\instrumented\_zstd.pyd
       FinalizeBuildStatus:
         Deleting file "C:\bbarea\3.x.itamaro-win64-srv-22-aws.tailcall.pgo\build\PCbuild\obj\315amd64_PGInstrument\_zstd\_zstd.tlog\unsuccessfulbuild".
         Touching "C:\bbarea\3.x.itamaro-win64-srv-22-aws.tailcall.pgo\build\PCbuild\obj\315amd64_PGInstrument\_zstd\_zstd.tlog\_zstd.lastbuildstate".
    43>Done Building Project "C:\bbarea\3.x.itamaro-win64-srv-22-aws.tailcall.pgo\build\PCbuild\_zstd.vcxproj" (Build target(s)).
    42>Link:
         Finished generating code
     0>MSBUILD : error MSB4166: Child node "6" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "4" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "7" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "5" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "16" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     1>Done Building Project "C:\bbarea\3.x.itamaro-win64-srv-22-aws.tailcall.pgo\build\PCbuild\pcbuild.proj" (Build target(s)) -- FAILED.
     0>MSBUILD : error MSB4166: Child node "14" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "9" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "2" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "13" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "11" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "3" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "15" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "12" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "10" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.
     0>MSBUILD : error MSB4166: Child node "8" exited prematurely. Shutting down. Diagnostic information may be found in files in "C:\Users\Administrator\AppData\Local\Temp\MSBuildTempa26c2c9e364d4d1d8ee1e2d93eeda949\" and will be named MSBuild_*.failure.txt. This location can be changed by setting the MSBUILDDEBUGPATH environment variable to a different directory.

The previous build (340) used MSVC 14.50.35717:

C:\Program Files\Microsoft Visual Studio\18\Community\VC\Tools\MSVC\14.50.35717\bin\HostX86\x86\CL.exe

Another example with similar errors: https://buildbot.python.org/#/builders/1870/builds/329

cc @chris-eibl @Fidget-Spinner @zooba @itamaro

[chris-eibl]

Yeah, I've noticed these, too. The first one happened soon after the builder went online. Never happened for me locally. Seems to be a very rare hiccup according to google results.

[vstinner]

MSB4166 diagnostic code says:

This error usually indicates an MSBuild issue, and should be reported to Microsoft.

msbuild.exe is used with /m option which spawns multiple worker processes, one per CPU. Log:

C:\bbarea\3.x.itamaro-win64-srv-22-aws.tailcall.pgo\build>"C:\Program Files\Microsoft Visual Studio\18\Community\MSBuild\Current\Bin\msbuild.exe" "C:\bbarea\3.x.itamaro-win64-srv-22-aws.tailcall.pgo\build\PCbuild\pcbuild.proj" /t:Build /m /v:n /p:Configuration=PGInstrument /p:Platform=x64 /p:IncludeExternals=true /p:IncludeCTypes=true /p:IncludeSSL=true /p:IncludeTkinter=true /p:DisableGil= /p:UseTestMarker= /p:GIT="C:\Program Files\Git\bin\git.exe" /p:UseJIT= /p:UseTIER2= /p:PyStats= /p:UseTailCallInterp=true /p:DisableRemoteDebug= /p:StackRefDebug= /p:UsePymallocHugepages=

A workaround for MSB4166 error is to add /nodeReuse:false option (or short option: /nr:false) to the msbuild command, or use the MSBUILDDISABLENODEREUSE=1 environment variable. It does not always solve the issue.

[chris-eibl]

[...] should be reported to Microsoft.
It does not always solve the issue.

Yeah, that's what I've read back then as well.

Never happened for me locally

Please also note, that I've never seen this failure in CI. So I think we shouldn't do anything in build.bat or the github workflows but maybe only on that specific build bot?