Console.SetOut can deadlock with writes through a previously captured Console.Out on Unix

[rolfbjarne]

Description

On Unix, writing concurrently through the current Console.Out and a previously
captured Console.Out can deadlock after Console.SetOut redirects output to a
writer that forwards to the original writer.

The deadlock is caused by a lock-order inversion between two synchronized
TextWriter instances:

• O is the original Console.Out.
• R is the current Console.Out, installed by Console.SetOut.

The two write paths acquire these writers in opposite orders:

1. Writing through R holds R in SyncTextWriter.WriteLine, then the
   forwarding writer writes to O: R → O.
2. Writing directly through O holds O in SyncTextWriter.WriteLine,
   then UnixConsoleStream reaches ConsolePal.WriteFromConsoleStream, which
   locks the mutable, current Console.Out (R): O → R.

If these paths execute concurrently, each thread can wait forever for the
writer owned by the other thread.

This was originally encountered as an intermittent deadlock in
dotnet/macios#26537. NUnit redirected Console.Out
to a capture writer that forwards off-context output to the original writer.
An asynchronous callback wrote through the redirected writer while the test
runner's event-pump thread wrote through the saved original writer. Neither
NUnit's forwarding writer nor the test runner explicitly locked these writers;
the conflicting locks came from SyncTextWriter and ConsolePal.

Reproduction

Create a .NET 10 console application and replace Program.cs with:

using System.Text;

// Keep this true for a deterministic reproduction. Change it to false to
// reproduce the same deadlock naturally, without explicitly locking either
// writer.
bool useDeterministicReproduction = true;

TextWriter originalOut = Console.Out;

Console.WriteLine ($"Starting the {(useDeterministicReproduction ? "deterministic" : "racy")} reproduction.");
Console.SetOut (new ForwardingTextWriter (originalOut));

TextWriter redirectedOut = Console.Out;

if (useDeterministicReproduction)
	RunDeterministicReproduction ();
else
	RunRacyReproduction ();

void RunDeterministicReproduction ()
{
	using var redirectedWriterLocked = new ManualResetEventSlim ();
	using var originalWriterLocked = new ManualResetEventSlim ();

	var redirectedWriterThread = new Thread (() => {
		lock (redirectedOut) {
			redirectedWriterLocked.Set ();
			originalWriterLocked.Wait ();

			redirectedOut.WriteLine ("redirected writer");
		}
	});

	var originalWriterThread = new Thread (() => {
		redirectedWriterLocked.Wait ();

		lock (originalOut) {
			originalWriterLocked.Set ();

			originalOut.WriteLine ("original writer");
		}
	});

	redirectedWriterThread.Start ();
	originalWriterThread.Start ();

	redirectedWriterThread.Join ();
	originalWriterThread.Join ();
}

void RunRacyReproduction ()
{
	var redirectedWriterThread = new Thread (() => {
		while (true) {
			Console.WriteLine ("redirected writer");
			Thread.Yield ();
		}
	}) {
		IsBackground = true,
	};

	var originalWriterThread = new Thread (() => {
		while (true) {
			originalOut.WriteLine ("original writer");
			Thread.Yield ();
		}
	}) {
		IsBackground = true,
	};

	redirectedWriterThread.Start ();
	originalWriterThread.Start ();

	redirectedWriterThread.Join ();
	originalWriterThread.Join ();
}

sealed class ForwardingTextWriter : TextWriter {
	readonly TextWriter writer;

	public ForwardingTextWriter (TextWriter writer)
	{
		this.writer = writer;
	}

	public override Encoding Encoding => writer.Encoding;

	public override void Write (char value)
	{
		writer.Write (value);
	}

	public override void Write (string? value)
	{
		writer.Write (value);
	}

	public override void WriteLine (string? value)
	{
		writer.WriteLine (value);
	}
}

Run:

dotnet run -c Release

The default path prints its initial line and then deterministically deadlocks.

The explicit lock statements in this path only make the existing lock-order
inversion 100% reproducible. They acquire the same writer monitors that the
subsequent WriteLine calls acquire implicitly, and those acquisitions are
reentrant. The events ensure that each thread owns its first writer before
either requests the second.

To reproduce without explicitly locking either writer, change:

bool useDeterministicReproduction = false;

The two background threads then repeatedly exercise the two real write paths
until they race in the wrong order. On my machine this mode deadlocks quickly;
in one run it stopped making progress after producing 1,074 bytes of output.

Expected behavior

Concurrent writes through the current Console.Out and a previously obtained
Console.Out should not deadlock.

In particular, a console stream obtained before Console.SetOut should not
later synchronize on an unrelated writer installed as the current
Console.Out.

Actual behavior

Both threads remain blocked in monitor acquisition:

• The redirected-writer thread owns R and waits for O.
• The original-writer thread owns O and waits for R in
  ConsolePal.WriteFromConsoleStream.

Attaching LLDB shows both threads waiting below Monitor_Enter_Slowpath.

Runtime implementation

Console.SetOut synchronizes the supplied writer:

if (newOut != TextWriter.Null)
{
    newOut = TextWriter.Synchronized(newOut);
}

SyncTextWriter.Write* methods use MethodImplOptions.Synchronized, so a write
through either O or R holds that writer's monitor.

On Unix, the original writer eventually reaches:

internal static unsafe void WriteFromConsoleStream(
    SafeFileHandle fd,
    ReadOnlySpan<byte> buffer)
{
    EnsureConsoleInitialized();

    lock (Console.Out)
    {
        Write(fd, buffer);
    }
}

The problematic part is that an existing console stream synchronizes on the
mutable current Console.Out, rather than on a stable lock associated with the
stream or console state.

Git history indicates that WriteFromConsoleStream and this
lock (Console.Out) were introduced in commit
ec0251bf15d18aaffff01741e032bd197670f473, as part of
#94414.

Environment

Reproduced with:

.NET SDK:       10.0.302
SDK commit:     35b593bebf
Runtime:        Microsoft.NETCore.App 10.0.10
RID:            osx-arm64
Operating system: macOS 26.6.2 (25G83)
Architecture:   arm64

The affected implementation is Unix-specific; I have not reproduced or
investigated this on Windows.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-console
See info in area-owners.md if you want to be subscribed.

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• [MTP] Hanging on v1.6.2 microsoft/testfx#5158