Repository navigation
ENH: Add regression tests for the bug fixes of the past two days - #123
hjmjohnson wants to merge 24 commits into
Conversation
8a301b1 to
62a5f0e
Compare
gdevenyi
left a comment
There was a problem hiding this comment.
Review by Google Carbon
Verdict: Request Changes (Draft; conflicting + bundles 8 unmerged standalone PRs & regresses 2 master fixes)
The cifti unterminated-XML extension fix itself (axio_read_cifti_ext switching to axio_read_buf(ext->edata, ext->esize - 8) instead of reading past ext->edata with strlen) is sound and fixes a real heap buffer overread!
However, because this stack was sliced from an interleaved integration branch, #123 has three blockers before it can merge:
- Bundles 8 unmerged standalone PRs (
#89, #90, #97, #101, #106, #107, #108, #109) + conflicts withmaster: Please rebase/extract#123ontomaster(or directly on#109once#109merges) so it contains only theciftiunterminated-extension commit (axio_read_cifti_ext+cext_unterminated.niitest). - Regresses two commits already on
master:nifti2/nifti_regress_test/cmake_testscripts/c22_copy_image.shcarries an older draft of50a9216that is missingexit 1on the secondnii_cmpfailure and missing${prefix}.raw*cleanup!- Carries an older
dup_attr.niacopy conflicting withmaster's8b47943.
- Add Sanitizer
FAIL_REGULAR_EXPRESSIONoncifti_tool_unterminated_cext(cifti/CMakeLists.txt:51-52): Because CTest ignores non-zero exit codes wheneverPASS_REGULAR_EXPRESSIONmatches, an ASan heap-buffer-overflow or LSan leak after printingMapName : demowould still pass withoutFAIL_REGULAR_EXPRESSION "ERROR: (Address|Leak)Sanitizer|runtime error".
| -prefix ${prefix}.2.1.i16.nii.gz \ | ||
| -convert2dtype NIFTI_TYPE_INT16 -convert_fail_choice warn | ||
| if cmp ${prefix}.0.i16.nii.gz ${prefix}.2.1.i16.nii.gz | ||
| if nii_cmp ${prefix}.0.i16.nii.gz ${prefix}.2.1.i16.nii.gz |
There was a problem hiding this comment.
[Google Carbon] Regression vs master (50a9216): This branch carries an older copy of the c22_copy_image.sh commit that omits exit 1 in the else branch below line 63 and omits rm -f ${prefix}.raw* at cleanup. Rebasing onto master and dropping all already-merged / standalone PR commits will drop this file from #123 completely.
There was a problem hiding this comment.
Agreed that this copy is older than master's 50a9216. It comes from the stack's stale base, and it drops out when the stack is rebased onto current master, which is the next, separate push.
62a5f0e to
a8ea828
Compare
|
Pushed a8ea828 with this PR's own fix: a sanitizer FAIL_REGULAR_EXPRESSION on tool_unterminated_cext (dfb9f55). The copies of master's c22_copy_image.sh and dup_attr.nia come from the stack's stale base. They drop out when the stack is rebased onto current master, which will be a separate, rebase-only push. |
Run the Build and Test workflow on this repository's branch. Its
trigger named "main" while the branch is "master", so it had never
executed on master and did not execute on pull requests; valgrind,
AddressSanitizer, UndefinedBehaviorSanitizer, scan-build, gcov
coverage, and the only shared-library configuration were all
unexercised. Also move its macOS entry off the retired macos-11 image.
Restructure the per-PR matrix around the axes that change what is
compiled:
shared BUILD_SHARED_LIBS gates TEST_INSTALL, so the ON legs are
the first to run install_linking and cover the install
and export path.
fslstyle -DFSLSTYLE is not additive. It rewrites behaviour in
niftilib and nifti2 through global -DFSLSTYLE, -DPIGZ and
-DREJECT_COMPLEX, and one site has an #else that changes
the pixdim[0] value written to disk. Neither value
subsumes the other.
build_type One static Debug leg, which is what makes assert() live.
Drop both macos gcc legs: /usr/bin/gcc there is a clang shim reporting
AppleClang, so they duplicated the macos clang legs exactly. The
optional libraries are built everywhere rather than forming an axis,
since they are additive.
Add three jobs. The minimal configuration builds znzlib and niftilib
alone, which is what a downstream project vendoring the core reader
selects and which nothing else configures. The oldest supported CMake
job guards cmake_minimum_required, invisible to runners that all carry
a recent CMake. The exported symbol baseline diffs the dynamic symbol
set against a committed file, so a change to the ABI has to be updated
in the same commit rather than landing unnoticed.
Verified on Ubuntu 24.04 with GCC 13.3.0, the compiler the Linux
runners use:
shared, Release, FSLSTYLE=OFF 345/345, install_linking passed
static, Debug, FSLSTYLE=ON 344/344, no assertion aborts
minimal configures and builds, no tests
CMake 3.28.3 344/344
The baseline holds 448 symbols across six libraries, which is the
figure the linkage changes have been asserting without a way to check
it.
The workflow steps were exercised by running them directly in the
runner container image rather than through act, whose image omits
cmake.
(cherry picked from commit 77b7418)
The dashboard script restates the CTestConfig.cmake settings because CTest's delayed initialization does not pick them up, and the restated copy had drifted from the original. Submit over https rather than http. my.cdash.org serves plain http without redirecting, so submissions were going unencrypted. Use the 00:00:00 EST nightly start time that CTestConfig.cmake declares, rather than 01:00:00 UTC. The two differ by four hours, so nightly builds were filed under the wrong day on the dashboard. (cherry picked from commit 063e838)
The 2025 conversion to GitHub Actions replaced .travis.yml with two
workflows but left the dashboard script untouched, so it still
required a Travis environment. The workflow that drives it named the
wrong branch and never ran, which is why nothing surfaced the
mismatch.
Rename travis_dashboard.cmake to github_dashboard.cmake and take its
inputs from the runner that actually exists. CTEST_SITE now comes from
RUNNER_OS rather than the required TRAVIS_APP_HOST, whose absence
aborted the script before it configured anything, and the build name
carries RUNNER_OS in place of TRAVIS_OS_NAME.
Compare the branch name against its value rather than against the
literal string "ENV{BUILD_SOURCEBRANCHNAME}", so the Continuous and
Nightly models can be selected at all, and give the workflow a branch
name on push events as well as on pull requests.
Apply the same corrections to local_dashboard.cmake.
(cherry picked from commit f578e03)
FslGetHdrImgNames and FslSetIntensityScaling are defined here but declared nowhere, while FslGetIntensityScaling and FslInit are already published. axml_recur_find_xml sits beside axml_recur in afni_xml.h the same way. The upstream fslio was deleted in 2015 in favour of a C++ replacement, so this copy is the surviving one and its header is ours to correct. Additive; no symbol changes. (cherry picked from commit 4673c40)
Nine functions in fslio.c have external linkage and no declaration in any header. Nothing in the tree calls them across a translation unit, AFNI's vendored copy never calls them, and no public source outside a vendored copy of this file references them, so a caller would have had to declare them itself. Collect their prototypes in one block so the boundary between internal and published is visible in one place. FslSetVoxUnits and FslGetVoxUnits have no caller at all and are left under #if 0 rather than deleted; FslSetTimeUnits and FslGetTimeUnits are published, so the asymmetry is worth keeping visible. Removes nine symbols from libfslio, which the baseline records. (cherry picked from commit 8cdcf1a)
nifti1_io.c and nifti2_io.c each define nifti_fileexists with external linkage, so libniftiio and libnifti2 export the same name and ELF link order decides which one a caller linking both resolves to. Declaring it locally silences the warning without choosing between publishing it and making it static; that choice needs the duplication settled first, and it covers a hundred names, not this one. (cherry picked from commit 54e74e1)
Keeps FslGetAuxFile and FslSetAuxFile deriving the same length so the pair cannot drift if the field width changes. sizeof is 24, so the copy still writes at most 23 bytes. (cherry picked from commit 0c0e66c)
Both header converters multiply the dimensions into nim->nvox without
checking the product:
for( ii=1, nim->nvox=1; ii <= nhdr.dim[0]; ii++ )
nim->nvox *= nhdr.dim[ii];
Seven NIFTI-1 dimensions of 32767 are enough to overflow int64_t, and a
NIFTI-2 header needs only two dimensions to do it:
nifti2_io.c:4838: runtime error: signed integer overflow:
1152780773560811521 * 32767 cannot be represented in type 'int64_t'
Signed overflow is undefined, and what the compiler does produce is a
voxel count that no longer describes the file. nvox then goes on to
nifti_get_volsize(), which multiplies it by nbyper for another
unchecked product, and that result is used as an allocation size and a
read length.
Both products are now checked before they are made, in the way the
surrounding code already reports a bad header. A header that overflows
is rejected with a message instead of producing a silently wrong image.
No valid image is affected: an int64_t voxel count is more than any
file can hold.
Found by fuzzing the header converters with clang's libFuzzer under
UndefinedBehaviorSanitizer.
(cherry picked from commit 66c76c2)
The first commit guards both converters in nifti2/nifti2_io.c.
niftilib/nifti1_io.c has the same loop in nifti_convert_nhdr2nim(), on
the same untrusted path, and was left unguarded.
$ nifti1_tool -disp_nim -infiles huge.nii # dim[0]=7, dim[1..7]=32767
dim 32 8 7 32767 32767 32767 32767 32767 32767 32767
nvox 64 1 -1073512449
The wrapped count then becomes the data allocation size and the result
of nifti_get_volsize(). With the guard the header is refused:
** ERROR: nifti_convert_nhdr2nim: dim[] overflows the voxel count
nifti_image.nvox is a size_t in this library and an int64_t in the
NIFTI-2 one, so the bound here is SIZE_MAX rather than INT64_MAX.
nifti_get_volsize() is size_t * size_t to match. dim[0] is already
bounded by need_nhdr_swap(), and the loop that raises every dim to at
least 1 still runs first, so the dim[ii] > 0 test never has to reject.
Two more nvox loops remain in each library, in
nifti_update_dims_from_array() and update_nifti_image_for_brick_list().
Both take dims that a caller has already set rather than dims read from
a file, and reach them from nifti_tool's command line, so they are left
for a separate change.
(cherry picked from commit 147a07a)
…agic claims
swap_nifti_header() takes a void pointer and a version number, and picks
the struct to swap from the version:
if ( ni_ver == 0 ) nifti_swap_as_analyze((nifti_analyze75 *)hdr);
else if( ni_ver == 1 ) nifti_swap_as_nifti1((nifti_1_header *)hdr);
else if( ni_ver == 2 ) nifti_swap_as_nifti2((nifti_2_header *)hdr);
Two callers pass it a nifti_1_header, 348 bytes, together with the
version taken from that header's own magic string by NIFTI_VERSION().
A file whose magic is "n+2" therefore has 540 bytes swapped in place:
192 bytes of read and write past the end of the caller's stack object.
nifti_read_n1_hdr() reads the header straight from a file, so the bytes
that decide this come from the file being read. A 348 byte file with
sizeof_hdr = 348, dim[0] byte-swapped, magic = "n+2"
is enough; need_nhdr_swap() then reports that swapping is needed and the
overflow happens before any validity check. AddressSanitizer:
ERROR: AddressSanitizer: stack-buffer-overflow
READ of size 1 ...
#0 nifti_swap_4bytes nifti2_io.c:3051
#1 nifti_swap_as_nifti2 nifti2_io.c:3194
#2 nifti_read_n1_hdr nifti2_io.c:5394
Address ... is located in stack of thread T0 at offset 412 in frame
[64, 412) 'nhdr' (line 5339) <== Memory access ... overflows
nifti_convert_n1hdr2nim() has the same line and takes its header from
the caller, so any application that fills a nifti_1_header itself can
reach it too.
Both call sites know the struct they hold, so both now ask for the swap
that struct supports: analyze when the magic is absent, NIFTI-1
otherwise. Headers claiming versions 3 to 9, which swap_nifti_header()
previously refused with a message and left unswapped, are now swapped as
NIFTI-1 as well, which is the only interpretation 348 bytes allow.
nifti1_io.c is not affected: its swap_nifti_header() takes a
nifti_1_header * and a flag, so it cannot choose a wider struct.
Found by fuzzing nifti_convert_n1hdr2nim() with clang's libFuzzer under
AddressSanitizer.
(cherry picked from commit 16a6b67)
nifti_tool.c has the same defect this branch fixes in the library: it
passes NIFTI_VERSION(*nhdr), read from the magic string, to
swap_nifti_header(), which selects the struct to swap by that number.
act_mod_hdrs() and act_swap_hdrs() hold a nifti_1_header of 348 bytes.
A magic of "n+2" makes NIFTI_VERSION() return 2, so swap_nifti_header()
treats the allocation as a 540 byte nifti_2_header and reads and writes
past its end. Both functions refuse a header that is valid NIFTI-2, but
a header that is valid as neither version reaches the call.
$ printf ... > evil.nii # 348 bytes, sizeof_hdr byte swapped,
# magic "n+2"
$ nifti_tool -mod_hdr -mod_field descrip hello -overwrite \
-infiles evil.nii
==2747364==ERROR: AddressSanitizer: heap-buffer-overflow
#0 nifti_swap_4bytes nifti2_io.c:3029
#1 nifti_swap_as_nifti2 nifti2_io.c:3172
#2 swap_nifti_header nifti2_io.c:3131
#3 act_mod_hdrs nifti_tool.c:3389
0 bytes after 348-byte region allocated in nifti_read_n1_hdr()
Both now pass 1, or 0 for ANALYZE, which are the two 348 byte layouts.
act_mod_hdr2s() holds a nifti_2_header and gets the explicit 2, as
act_swap_hdrs() already does for its own NIFTI-2 display path.
old_swap_nifti_header() takes a nifti_1_header and a boolean, so the
calls beside these need no change.
(cherry picked from commit 522e624)
Under FSLSTYLE the block that reports "Multiple possible filenames" has
three defects, and they cannot be separated.
free(basename); <- freed here
char *gzname = calloc(...); <- unchecked
strcpy(gzname, hdrname); <- used at once
...
fprintf(stderr,"... %s\n", basename); <- read after free
exit(134); <- kills the caller
basename is read by the message, so it cannot be freed at the top.
gzname goes straight into strcpy(), so it has to be checked. And once
the function returns NULL instead of calling exit(), every path out has
to release what it holds, which is what ties the three together: moving
the free of basename decides what the error paths must free, and there
are no error paths to speak of until exit() is gone.
FSLSTYLE is off by default but is a supported configuration:
-DFSLSTYLE:BOOL=ON turns it on, and nifti2/Makefile defines it always.
Built and tested with -DFSLSTYLE=ON.
The nifti_image_read() half of the original change is a separate commit.
(cherry picked from commit 0ddf192)
mind = get_map_index(xt->xchild[kid]);
if( kid >= 0 ) MIM_disp_funcs[mind](ofp, xt->xchild[kid], verb);
The guard tests kid, the loop counter, which is never negative, so it is
always true. The index actually used is mind, and get_map_index()
returns -1 for any element name not in MIM_kids[]. A CIFTI file
containing an unrecognised element under MatrixIndicesMap therefore reads
a function pointer from before the start of MIM_disp_funcs and calls it.
Reported by the clang static analyzer as security.ArrayBound, "Out of
bound access to memory preceding 'MIM_disp_funcs'".
(cherry picked from commit b8d5246)
Drives cifti_tool with -eval_type show_summary over two XML fixtures. The first holds a MatrixIndicesMap child whose name get_map_index() does not resolve, and fails without the preceding commit: the sanitizer legs report a global-buffer-overflow, an 8-byte read 8 bytes before MIM_disp_funcs. The second holds a BrainModel and asserts it is still displayed, so that skipping the unresolved name cannot pass by skipping every name. These are the first tests for the cifti library. The fixtures are small enough to keep in tree, so neither needs the external testing data. (cherry picked from commit 06cb5a6)
process() returned without releasing either the afni_xml_t it parsed or the nifti_image the non-cext path fills in, so every run leaked the whole tree. Both free routines already accept NULL, so neither path needs a guard. LeakSanitizer is on by default under AddressSanitizer on Linux but not on Apple, so the leak ended the process with a non-zero status on the sanitizer job alone. That also discarded the buffered standard output, which is why the summary text went missing there rather than merely being followed by a leak report. (cherry picked from commit e96d1e9)
A dim[0] outside 1..7 is used to index dim[] in nifti_convert_n2hdr2nim, so a header carrying a large negative value reads far outside the struct. The fixture is a 604-byte NIFTI-2 header holding such a value; without the check the tool segfaults in an ordinary Release build, so this test guards on every CI job rather than only the sanitizer one. (cherry picked from commit 2f6742c)
nifti_read_n1_hdr reads 348 bytes, so swapping the buffer as whatever width its magic names reaches past the end when the magic says n+2. The fixture is a big-endian NIFTI-1 header wearing an n+2 magic, which the sanitizer legs report as a stack-buffer-overflow without the fix. -check_hdr is the seam that reaches it; -disp_hdr does not. (cherry picked from commit e3990de)
act_mod_hdrs() and act_swap_hdrs() hold a 348-byte nifti_1_header and swapped it at the width its magic named, reading past the end for an n+2 magic. Both reuse the fixture added for the library-side check, and both rewrite their input, so each test runs against its own copy. (cherry picked from commit 3deb735)
Three fixtures, because the two guards are reached separately: a product that wraps nvox, the NIFTI-2 form of the same, and one whose nvox fits while nvox * nbyper does not. Without the guards the tool accepts all three and reports a negative nvox. The regex pins which refusal fired; an exit-code test would not tell this rejection apart from any other error. (cherry picked from commit 01ce2fa)
nifti_convert_nhdr2nim carries its own copy of the guards, so it needs its own tests; reverting the NIFTI-1 half alone leaves the nifti2 tests green and turns exactly these two red. (cherry picked from commit 329e869)
loc_strnlen read one byte past the buffer it was given. The reachable caller is axml_read_buf, where the buffer is an extension's edata, allocated esize-8 and filled exactly; axml_read_file allocates one spare byte and absorbs the overread, so it is not the seam. The fixture is a NIFTI-1 image carrying a CIFTI extension whose payload is space-padded to hold no NUL. The overread is reported by the sanitizer legs; the build stays green without them. (cherry picked from commit 457f8bf)
The tool writes both members of the pair itself, so no data is needed. The driver asserts exit 1 and the diagnostic, because a library reports the clash to its caller rather than ending the process, and a plain add_test cannot tell that exit from the abort it replaced. It then removes the .gz and repeats, so an unconditional NULL return could not pass. Guarded on FSLSTYLE_NAME_CONFLICTS, which is off by default, so the test is registered only where the code is compiled. (cherry picked from commit 7a0b98b)
Two paths returned without releasing the header, and one without releasing the duplicated name: a write into a directory that refuses it, and a duplication that fails because the input is the other NIfTI version. Both are reachable from the shipped tools with no data beyond what -make_im writes. The script asserts only that the paths are reached and reported; the leak is what the memcheck and sanitizer legs see. It skips itself for root, whom a read-only directory does not stop. (cherry picked from commit c051fd8)
c22_copy_image compares by content because gzip output is not reproducible across zlib implementations, but that property was only exercised where the system zlib is zlib-ng. Every CI leg ships stock zlib, so the comparison could revert to comparing compressed bytes and the suite would stay green. Extracting the helper lets it be tested directly. gzip -1 against -9 manufactures the byte difference on any zlib, and the two negative assertions pin that differing content is still reported as different and that the uncompressed path still works. (cherry picked from commit b5023c5)
a8ea828 to
e5aa40c
Compare
|
Rebase-only push onto master (e5aa40c). Master already carries reviewed copies of six of this PR's commits, so those are dropped; the two test-registration conflicts keep master's dup_attr test and this PR's own additions. No review changes in this push. |
Re-submission of #80, reverted from
masteron 2026-09-24. Content isunchanged from the original.
Position 1 of 11 in the deep stack. Base:
master.Stack order
stack/test-regression-coverage<- this PRmasterstack/pr-fix-alloc-null-checksstack/test-regression-coveragestack/fix-axml-skip-depthstack/pr-fix-alloc-null-checksstack/pr-fix-analyzer-leaksstack/fix-axml-skip-depthstack/pr-fix-sign-conversionstack/pr-fix-analyzer-leaksstack/fix-fslio-64bit-arithmeticstack/pr-fix-sign-conversionstack/fix-cifti-null-streamstack/fix-fslio-64bit-arithmeticstack/pr-fix-calloc-transposed-argsstack/fix-cifti-null-streamstack/pr-fix-shorten-64-to-32stack/pr-fix-calloc-transposed-argsstack/fix-image-read-complex-checkstack/pr-fix-shorten-64-to-32stack/pr-fix-xml-read-errorsstack/fix-image-read-complex-checkThe order is the order these changes sat on
masterbefore the revert, soit builds and tests at every step.
Commits introduced by this PR
Ordering for all the re-submitted work is tracked in #84.