asynManager: fail cleanly instead of hanging on an oversize traceIO truncate size

[physwkim]

Problem

setTraceIOTruncateSize() takes a size_t, but its two callers pass a
signed value:

• asynRecord's TSIZ field is epicsInt32 (asynRecord.c)
• the iocsh command asynSetTraceIOTruncateSize(..., int size)
  (asynShellCommands.c)

A negative value such as -1 converts to a size_t near SIZE_MAX.
With that size the function called

epicsMutexMustLock(pasynBase->lockTrace);
if(size>ptracePvt->traceBufferSize) {
    free(ptracePvt->traceBuffer);
    ptracePvt->traceBuffer = callocMustSucceed(size, sizeof(char), ...);

callocMustSucceed() loops forever (epicsThreadSuspendSelf) when the
allocation can't be satisfied — and it does so while holding the global
pasynBase->lockTrace mutex, so every other thread that traces then blocks
on that mutex too. A single

caput <record>.TSIZ -1

wedges tracing for the whole IOC.

The pre-existing free() before the allocation is a second problem: even a
recoverable allocator would leave traceBuffer dangling on failure.

Fix

Allocate the replacement with calloc() and, on NULL, unlock and return
asynError — an unsatisfiable request fails cleanly instead of hanging.
Free the existing buffer only once the new one is in hand, so a failure
leaves the current trace buffer intact. Both entry points funnel through
this single owner, so the record and iocsh paths are fixed together.

Test

Adds an asynPortDriverTest case: setTraceIOTruncateSize((size_t)-1)
returns asynError, and a normal size still returns asynSuccess. Without
the fix the test hangs on the oversize request instead of completing
(241/241 with the fix).