calc engine: fix four out-of-bounds accesses in the sCalc/aCalc perform paths

[physwkim]

calc engine: fix four out-of-bounds accesses in the sCalc/aCalc perform paths

Summary

Four memory-safety defects in the calc expression engines, each reachable
from an ordinary .db-authored expression (no debug build, no special
build flags). Three are out-of-bounds reads/writes past a fixed-size
buffer; one is an inert copy bound that lets a long string literal overrun
its stack cell. Each fix is a one-line, minimal guard that matches a guard
idiom already present in the same file. No refactoring, no behavior change
on the in-range paths.

One commit per defect:

Commit	File	Defect
sCalc: clamp the shift count against a negative operand	calcApp/src/sCalcPerform.c	OOB read/write on <</>> with a negative count
sCalc: bound the LITERAL_STRING copy by incrementing its counter	calcApp/src/sCalcPerform.c	OOB write copying a >39-char literal
aCalc: cap the SUBRANGE upper bound at arraySize-1	calcApp/src/aCalcPerform.c	OOB read one past the array on SUBRANGE
calcUtil: reject an nderiv fit window larger than the array	calcApp/src/calcUtil.c	OOB read in DERIV on arrays shorter than 5

Details

1. sCalcPerform.c — <</>> with a negative shift count (OOB read/write)

In the RIGHT_SHIFT/LEFT_SHIFT case the character-shift count is
computed and clamped only from above:

j = myNINT(ps1->d);
j = myMIN(j,SCALC_STRING_SIZE);   /* clamped at 40, never below 0 */

For a negative j (the shift operand evaluates negative at runtime)
the string branch runs off the 40-byte local_string:

• RIGHT_SHIFT: ps->s[i] = (i>=j)?ps->s[i-j]:' ' — with j<0 the
  condition i>=j is always true, so it reads ps->s[i-j], i.e.
  ps->s[i+|j|], past the end of the buffer.
• LEFT_SHIFT: the loop bound becomes i < SCALC_STRING_SIZE - j
  (= 40 + |j|), so ps->s[i] is written out to s[40+|j|-1] — an
  out-of-bounds write into the adjacent stack element.

Reachability: any scalcout expression of the form AA << B (or >>)
whose B operand goes negative at runtime — an ordinary evaluation, not a
malformed database.

Fix: clamp the count to [0, SCALC_STRING_SIZE] using the same
myMAX(myMIN(...),0) idiom this file already uses for the SUBRANGE
bounds (aCalcPerform.c:1533). This closes the OOB read/write on the
string branch. The pure-numeric (int)>>(int) branch shifts by the
untouched operand and is left as-is: a negative numeric shift count is
C-language UB but not a memory-safety issue, and clamping it would change
long-standing numeric results.

2. sCalcPerform.c — LITERAL_STRING copy bound never advances (OOB write)

case LITERAL_STRING:
    INC(ps);
    ps->s = &(ps->local_string[0]);
    s = ps->s;
    for (i=0; (i<SCALC_STRING_SIZE-1) && *post; )   /* i never incremented */
        *s++ = (char)*post++;
    *s = '\0';

The loop was written as a bounded copy, but i is never incremented, so
i < SCALC_STRING_SIZE-1 (i.e. i < 39) stays permanently true and the
copy runs to the end of the literal. A quoted literal longer than 39
characters therefore writes past local_string[40] inside the stack
element, on every evaluation.

Reachability: anyone who can load a database — a single long quoted
string literal in an sCalcout/scalcout expression.

Fix: increment i in the loop so the intended SCALC_STRING_SIZE-1
bound actually applies (the copy stops at 39 chars, *s='\0' terminates).

3. aCalcPerform.c — SUBRANGE upper bound admits arraySize (OOB read)

i = myMAX(myMIN(i,arraySize),0);
j = myMIN(j,arraySize);                 /* arraySize itself is admitted */
...
for (k=0; i<=j; k++, i++) ps->a[k] = ps->a[i];   /* reads ps->a[arraySize] */

ps->a holds arraySize doubles. j is an inclusive upper index but
is clamped to arraySize, a value the element count itself reaches (e.g.
AA[3,N] where N is the array length). The inclusive copy loop then
reads ps->a[arraySize], one element past the buffer. Under ASAN, or an
unlucky allocation, this crashes the IOC on a legal expression.

Fix: cap j at arraySize-1, the last valid index. This also
corrects the SUBRANGE_IP element count numEl = j+1, which
over-reported by one at the same boundary. The analogous string
SUBRANGE in sCalcPerform.c is not affected — its copy loop is
guarded by the string's NUL terminator (*s1), so reaching index
strlen reads the in-bounds terminator; only the array path lacks a
terminator, so only the array clamp is changed.

4. calcUtil.c — DERIV/nderiv fixed 5-point window over-reads short arrays (OOB read)

nderiv() fixes its fit window at m = 2*npts+1 (deriv() passes
npts=2, so m=5) and then unconditionally does:

• fitpoly(x, y, m, ...), whose accumulation reads x[0..m-1]/y[0..m-1]
  regardless of the caller's actual point count n;
• a tail loop lx[j] = x[(n-m)+j] - x[n-m] which, for n<m, indexes
  x[] with a negative offset.

nderiv never compares m with n, and fitpoly's only guard (n<3)
tests its own argument, which is the constant m. aCalc reaches this with
1+lastEl-firstEl points, which a 2-element array makes 2 — so
DERIV(AA) on an array (or window) shorter than 5 reads before/past the
operand buffer.

Fix: guard n < m at the top of nderiv and return the same -1
error the fit helpers already return (if (n<3) return(-1) in
fitpoly()/pfit(), if (n<2) return(-1) in lfit()). Both callers
(deriv() at aCalcPerform.c:985 and the nderiv-with-npts case at
:613) route through this one function, so the single guard covers both;
the :613 caller already constrains npts so that m <= n, so the guard
is a no-op on that path and only rejects the genuinely-too-short deriv()
case.

Testing

Not compiled locally — this checkout has no EPICS base support tree
configured, so make was not run here. Each change is a single-line guard
verified by reading the surrounding code and the macro expansions:

• The myMAX(myMIN(j,SCALC_STRING_SIZE),0) form is byte-for-byte the same
  macro nesting already used at aCalcPerform.c:1533, so it carries no new
  precedence risk.
• j = myMIN(j,arraySize-1) and if (n < m) return(-1); are trivial and
  mirror existing idioms in the same files.
• The LITERAL_STRING fix only adds the missing i++ to the for
  increment clause.

CI will compile all targets. These are guard additions on
out-of-range/negative inputs and do not change results on any in-range
path, so existing regression expectations are preserved.