Skip to content

resdata: Add version 6.3.3 - #3237

Open
luhenry wants to merge 10 commits into
mainfrom
resdata
Open

luhenry wants to merge 10 commits into
mainfrom
resdata

Conversation

@luhenry

@luhenry luhenry commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Compiles the libresdata C/C++ core and its pybind11 extensions, with backward-cpp, fmt and elfutils built from source by Conan. Upstream publishes no riscv64 wheel.

Mirrors upstream's testing.yml.

Differs from upstream

  • build-frontend is build - build[uv] needs a host uv the runner lacks.
  • before-build also stages the Conan dependencies' licence texts.
  • gpl_sources job archives the elfutils tarball and Conan recipe for the release.
  • CIBW_ENVIRONMENT pins SETUPTOOLS_SCM_PRETEND_VERSION - the Apply patches step's git apply dirties the checkout, so git describe reports a post-6.3.3 dev version instead of the clean tag (gotcha 31). The bare env var is needed, not the _FOR_RESDATA suffix: setup.py calls setuptools_scm.get_version() directly without a dist_name, so the per-package variant is never checked.

Matrix: cp312/cp313/cp314 - upstream ships no free-threaded wheel.

Testing

  • same as upstream, with numpy, scipy, pandas and pydantic-core taken as wheels only, since the registry trails PyPI for them.
  • 15-minute per-test timeout (thread method) - a hang in native code otherwise holds the runner for the whole job.
  • test_performance_with_many_keys gets 60 s instead of upstream's 10 s (patch 0002).

License: Wheel statically links elfutils (GPL-2.0-or-later or LGPL-3.0-or-later), backward-cpp, fmt, pybind11, zlib, bzip2, xz and zstd; upstream ships no licence text for them, so the build adds it.

Patches

  • 0001-Fix-infinite-loops-at-EOF-where-char-is-unsigned.patch - Backport [https://github.com/equinor/resdata/commit/ce341a3666855c57839466d9daa876252b54c8c0]. Without it test_broken_file loops forever, since char is unsigned on riscv64. Reproduces on x86_64 with -funsigned-char.
  • 0002-Raise-test_performance_with_many_keys-timeout-for-sl.patch - Inappropriate [the 10s ceiling is sized for upstream's x86 CI]. The benchmark always does 7 summary.x calls over 5000 keys, about 1.4 s each on riscv64 against 0.1 s on x86_64. The 10 s marker ran out on the seventh call, and since its timeout takes precedence over --timeout=900, it ended the whole session. The work is linear in the key count; nothing loops.

All three build legs (cp312/cp313/cp314) pass on real riscv64 hardware.

Build resdata's riscv64 manylinux wheels from the upstream tag with
cibuildwheel, mirroring upstream's testing.yml. Conan builds the
statically linked dependencies from source; their licence texts are
staged into the wheel and the elfutils source and recipe are archived
for the release.

luhenry commented Oct 14, 2026 •

Copy link
Copy Markdown
Member Author

Same mechanic as the other recent campaign PRs: check_commit_messages (job 116023092412) failed on check_no_ai_attribution against the pull_request "opened" webhook's frozen body snapshot, captured before an auto-injected attribution footer was stripped from the PR body. The live PR body has been clean since right after creation.

pr-checks.yml already triggers on edited for exactly this case, and that run has already passed. No fix to push.

luhenry added a commit that referenced this pull request Oct 15, 2026
…ated

topk-sdk (#3230), zope-index (#3228), zccache (#3229), kfnetlist (#3231),
ligo-skymap (#3233), quantile-forest (#3232), fbuild (#3235),
py-ballisticcalc-exts (#3238), resdata (#3237), oxapy (#3239),
pyramids-gis (#3240), biscuit-python (#3223), ansys-mapdl-reader (#3227)
all had open PRs that queue.yml never reflected (status still queued,
pr still null). Verified each PR independently before updating.
…urce

pip install "cmake<4" can never resolve a riscv64 wheel -- the cmake
PyPI project has published one only from 4.0 on -- so it fell back to
building CMake's own C++ source, which failed on a missing OpenSSL dev
package this image doesn't carry for an unrelated build dependency.
Patches pyproject.toml's build-system requirement to drop the <4 upper
bound (CMakeLists.txt's own cmake_minimum_required is 3.16, well under
4.0's removed-compatibility floor) and applies it the same way every
other patched port in this repo does (checkout python-wheels alongside
the external repo, git apply before the build step); updates
CIBW_BEFORE_BUILD's matching manual pip install to match.
@luhenry
luhenry marked this pull request as draft October 15, 2026 12:14
The cp314-manylinux_riscv64 leg died during CIBW_BEFORE_BUILD while Conan
built ninja from source: cmake's compiler check link-tests with
-static-libstdc++ -static-libgcc, and the manylinux_2_39_riscv64 (Rocky 10)
image ships only the shared libstdc++.so, not libstdc++.a, so the link fails
with "/usr/bin/ld: cannot find -lstdc++". This happens before any
Python-version-specific code runs, so it is not cp314-specific - it would
equally hit cp312/cp313 once their legs reach the same step.

Same fix already proven in this repo for the identical image/error in
build-ddtrace.yml, build-libclang.yml and build-coacd.yml: install
libstdc++-static via CIBW_BEFORE_ALL_LINUX (CRB is already enabled on this
image, so no --enablerepo flag is needed).

luhenry commented Oct 16, 2026 •

Copy link
Copy Markdown
Member Author

Failure: Build resdata 6.3.3 cp314-manylinux_riscv64 (run 38711493152, head 192bfe1) died in CIBW_BEFORE_BUILD, not in resdata's own build: while conan install was building ninja (a build tool it pulls in), cmake's compiler sanity check links with -static-libstdc++ -static-libgcc and failed with /usr/bin/ld: cannot find -lstdc++.

Root cause: this is riscv64-image-specific, not a cp314/pybind11/ABI issue, and not an upstream resdata bug. quay.io/pypa/manylinux_2_39_riscv64 is Rocky 10-based and only ships the shared libstdc++.so, not the static libstdc++.a that -static-libstdc++ needs. It happens before any Python-version-specific code runs (during dependency staging, same for every cpXYZ leg), so cp312/cp313 would hit the identical failure once their legs reached this step — they just hadn't run yet (this runner only had capacity to start cp314 first). This is exactly gotcha 77 (skills/python-project-porting/references/gotchas/native-deps-and-linking.md), already hit and fixed the same way for this same image in build-ddtrace.yml, build-libclang.yml, and build-coacd.yml.

Fix: added CIBW_BEFORE_ALL_LINUX: dnf -y install libstdc++-static to build-resdata.yml (CRB is already enabled on this image, so no --enablerepo flag needed, matching build-coacd.yml). Pushed as e5aecc5.

I couldn't run a full riscv64 rehearsal locally (this sandbox's Docker can't reach quay.io through its network policy), so validation here is the YAML parsing cleanly plus this being the identical, already-proven fix for the identical error/image elsewhere in the repo. Re-running CI on this PR should confirm it for all three Python legs.

Fixes PR #3237's riscv64 build failure: scikit-build's
check_for_bad_installs() trips on the stale CMake cache left behind
by python -m build's own metadata-then-wheel double invocation of
setup.py (scikit-build/scikit-build#521, closed not-planned).
The check_for_bad_installs() failure ("CMake-installed files must be
within the project root", with .bin/.libs/.include under cmake-build/)
is caused by CMake 4, not by a stale _skbuild cache: it reproduces on a
fresh x86_64 checkout with cmake 4.4.4 and goes away with cmake 3.31.

setup.py passes -DCMAKE_INSTALL_{BIN,LIB,INCLUDE}DIR=python/resdata/...
untyped, so they enter the cache as UNINITIALIZED. CMake 4's
GNUInstallDirs no longer converts them to PATH, and CMakeLists.txt's
if(SKBUILD) block then does set(... CACHE PATH ... FORCE) on them,
which turns the relative value into an absolute path under the build
directory. The generated cmake_install.cmake then installs to
cmake-build/python/resdata/... instead of ${CMAKE_INSTALL_PREFIX}/...,
which skbuild rejects. CMake 3.x types the entries in GNUInstallDirs
first, so the forced set keeps them relative.

Upstream pins cmake<4 for this. The earlier premise that no riscv64
cmake<4 wheel exists was wrong: pypi.riseproject.dev carries
cmake-3.31.6-py3-none-manylinux_2_39_riscv64, but pip preferred PyPI's
newer, sdist-only 3.31.10 and tried to compile it (gotcha 30).
PIP_ONLY_BINARY=cmake makes both the build-isolation install and
CIBW_BEFORE_BUILD resolve to our wheel, so restore upstream's cmake<4
and drop both patches along with the steps that applied them.
All three legs of run 39022584611 stopped printing right after
tests/rd_tests/test_rd_cmp.py and sat silent until the job's
timeout-minutes: 300 cancelled them. The next test is
test_rd_file.py::ResdataFileTest::test_broken_file, which opens a text
file named CASE.FINIT. The F prefix makes it a formatted file, so
rd_kw_fread_header() calls rd_kw_qskip() to find the opening quote.
That loop stores fgetc() in a plain char and stops on c == EOF. char is
unsigned on riscv64, so EOF reads back as 255, the test is always false,
and the loop spins forever at the end of a file with no quote in it.

Reproduced on x86_64 by building resdata 6.3.3 with -funsigned-char:
test_broken_file hangs in ResdataFile("CASE.FINIT") (pytest-timeout's
thread method shows the stack; the default signal method never fires,
since control never returns to Python). With the patch, the whole suite
passes on that build.

util_fskip_lines() and fscanf_grdecl_data() have the same bug, so the
patch changes all three variables to int. Upstream fixed them after the
6.3.x branch (ce341a3, d9eacc1, c052640); no 6.3.x release carries the
fixes yet.
…ethod

Only the first 27% of the suite has run on riscv64 so far, and a hang
there holds a runner for the rest of the job's 300 minutes before it
shows up as a bare "cancelled". pytest-timeout is already in upstream's
test group, so this adds no dependency.

The thread method is the one that works here: test_broken_file's hang
is a busy loop in libresdata, so control never returns to Python and
the default SIGALRM method never fires (checked on an x86_64
-funsigned-char build). The thread method dumps every thread's stack
and exits, which fails the job and names the test.

The slowest test seen on the riscv64 runner, test_grid_large_case,
takes about 108 s (18.8 s on x86_64), so 900 s leaves wide margin.
The cp313 job did not hang a second time: it stopped at the last test
of test_view_summary.py, test_performance_with_many_keys, whose own
@pytest.mark.timeout(10) takes precedence over --timeout=900. With the
thread method that marker inherits, the timeout ends the whole pytest
session instead of failing one test, which made it read as a hang.

The work is bounded: a call slower than max_time (1 s) makes
pytest-benchmark run exactly 7 calls (2 calibration + 5 rounds). The
captured stdout holds 6 complete summary.x tables, so the 10 s ran out
during the seventh and last call, at about 1.4 s a call against 0.1 s
on x86_64 (1.2 s for the whole test). A cProfile of one call is linear
in the 5000 keys, with no quadratic lookup. 60 s keeps the test a
regression guard while leaving the riscv64 runners about 5x headroom
(gotcha 38).
… poison it

The Apply patches step's git apply leaves the setuptools_scm checkout dirty,
so git describe reports 6.3.4.dev0+g<hash>.d<date> instead of the clean
6.3.3 tag (gotcha 31) - that's why the publish job failed: the wheel's
version has no matching entry in docs/packages/resdata.yaml, and PyPI's
6.3.3 build has no dev suffix to compare against anyway.
…SDATA suffix

The previous commit's SETUPTOOLS_SCM_PRETEND_VERSION_FOR_RESDATA had no effect:
setup.py calls setuptools_scm.get_version() directly without a dist_name
argument, so setuptools_scm never learns the package name and can't match
the per-package env var. Only the generic SETUPTOOLS_SCM_PRETEND_VERSION,
which applies regardless of dist_name, actually reaches get_version() here.
Confirmed by the publish job: the wheel version was still the dirty
6.3.4.dev0+g6bed49710.d20261018 string after the first fix.
@luhenry
luhenry marked this pull request as ready for review October 18, 2026 03:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant