Repository navigation
[Feature request/improvement] Split nofib test suite from main package #578
Description
Currently, the nofib test suite is part of the main accelerate package. This means users always get additional dependencies on testing-specific packages.
Currently, building this test-suite is guarded behind a cabal flag, which can be clunky to use. Additionally, it requires extra checks for conditional compilation.
Instead, it would probably be cleaner to have the nofib tests in a separate package, completely decoupling it from the main accelerate package (e.g. accelerate-nofib). The accelerate test suite for the reference interpreter, as well as the test suites of other backends, could depend on this new package.
Besides being easier to use for accelerate users, an added benefit is that the test suite can be expanded independently from the main package.
This would however mean turning this repository into a monorepo (which may change the experience for users pulling the dependency directly from git), or add a new repository (which also needs maintenance).
Activity
dpvanbalen commentedon Sep 16, 2026
What is the downside of putting it in a separate component, like doctest? Is there no way of making that importable into accelerate-llvm's test suites? Conceptually, that would solve all of the above issues.
ivogabe commentedon Sep 16, 2026
Thanks for posting this after our discussion this week, Tjalle!
@dpvanbalen Good suggestion. I think that could work so we should try that first, but I'm not that familiar with all the constraints in play here.
@TravisCardwell @tomsmeding I'm also curious what your opinions are
Tjalle-S commentedon Sep 16, 2026
Clearly, I am not up to date with my knowledge of Cabal :) A separate component should indeed work, using a sublibrary with
visibilityset to public. It does mean we lose a small bit of backward compatibility (because cabal version 3.0 is required), but that is 7 years old by now, so probably not a big deal.However, I don't think that allows updating the test suite without a new release of the main package (if we want that)
tomsmeding commentedon Sep 16, 2026
Sublibraries with public visibility haven't been visible on Hackage for a long time, but they indeed are now. More, it seems stack even supports this since a few years: commercialhaskell/stack#6343
Indeed, using a sublibrary doesn't allow updating the test suite without a release of the main package. Since the test suite currently consists essentially only of unit tests (in the form of a property test, but in terms of software engineering it's still a unit test!) and issue reproducers, there's generally no need to add anything to the test suite when accelerate itself doesn't change.
This might change if you want to include more integration tests into the test suite, such as some of the programs in https://github.com/tomsmeding/accelerate-tests and possibly more (CFAL?). If that is the goal, it does make sense to split out the test suite to a separate package, because programs are, in fact, sometimes written for accelerate without updating the compiler itself. ;)
Tjalle-S commentedon Sep 16, 2026
I suppose a second use case is for bugs in a backend (e.g. native) that weren't previously caught by the existing test suite. Then we can update the backend and test suite and leave the main package alone. I'm not sure how common that is (or even desirable)
dpvanbalen commentedon Sep 16, 2026
In terms of comparing to the current status, not being able to publish an update to nofib without updating accelerate isn't a regression, and I don't think we've ever wanted to?
Considering that accelerate-llvm is the main user of nofib, I think it's fine for accelerate-llvm to just point to the git version of nofib rather than fetch it from Hackage (which is also what it already currently does for accelerate)
It just seems like a lightweight alternative to completely extracting it to a separate package or repository, both of which would have more impact and could still be done later if we run into any new issues
tomsmeding commentedon Sep 16, 2026
Given the empirical development process of Accelerate so far, I think that falls under the case of people writing programs for accelerate without changing the compiler itself. :P But yes, bugs have been found, and in fact my linked accelerate-tests contains reproducers for some of them. Unfortunately some of those bugs were very probabilistic and as such their reproducers weren't obviously suitable for the main test suite. The tests in nofib/misc originate from backend bugs that we did consider suitable for the main test suite.
Independent of whether it should be a separate package or not, I do think there's a strong argument for at least splitting the tests into a separate component, as indeed the current flags-based setup is rather clunky in practice. The same holds for the -fdebug and -ftracy flags, in fact; at least tracy is likely hard to eliminate (since Setup.hs does stuff based on it) but perhaps in the future -fdebug can also be a Haskell-level switch instead of a switch in the build configuration.
TravisCardwell commentedon Sep 17, 2026
TLDR:
nofibflag would be a significant improvement! 👍Custombuilt-type, however, changing how we manage Tracy.nofibusing Git is viable.Premises
I will explicitly state the premises to make sure that my understanding is accurate. If I have anything wrong, please let me know!
In
accelerate, thenofibmodules must be exposed via a library (not a test suite likedoctest) becauseaccelerate-llvm-nativeandaccelerate-llvm-ptxtest suites must import from them.The current implementation uses a Cabal flag to enable building of all
nofibmodules. When the flag is not set, CPP macroACCELERATE_DISABLE_NOFIBis used in the top-levelnofibmodule to raise an error if used. The Cabal flag just affects thenofibmodules; it does not change anything else.Separation
I think that separating
nofibfrom theacceleratelibrary would be a significant improvement. Currently, the test suites are not able to require thataccelerateis built using the flag, so checks are left to runtime. With the separation, the dependencies can be correctly managed by Cabal, with no need to remember to build with flags (perhaps by changingcabal.project.localconfiguration).Public Sublibrary
Implementing the separation using a public sublibrary sounds good to me. Doing so is currently not possible, however, because
accelerateuses aCustombuild-type. With aCustombuild-type, Cabal uses the "whole package" code path and is unable to do per-component builds, required for sublibraries.We discussed this a bit recently, however, and it sounds like changing how we integrate with Tracy is on the table, removing the need to use a
Custombuild-type.Migrating to the newer
Hooksbuild-type would allow us to integrate with Tracy in a similar way, butHooksis a new feature that requires Cabal 3.14 (cabal-version: 3.14andcabal-install3.14 or later). That would significantly raise our lower bound.Another option is to manage Tracy source acquisition and building separately from Cabal. Looking into this option, it looks like the Tracy server (GUI) must indeed use the exact version that the client (Accelerate application) was built with.
Perhaps we could vendor the headers of a specific Tracy version so that
acceleratecan be built with Tracy support. Note that this would also make thesdistcomplete, with no need to download additional sources. (Currently,Setup.hsmay usegitorcurlto download sources from GitHub, which is problematic for companies that isolate builds for security and/or reproducibility reasons. Perhaps this is also an issue for Hackage builds.) Users would need to build and use the exact same version of the server (GUI), separately. Perhaps we could provide scripts in the repository to do this.I imagine that the current design was selected so that users can run
cabal install accelerate -f+tracyto download and build the configured version of Tracy with Accelerate. With this change, the command would no longer build the client; users would need to do that separately. Perhaps this is not such a significant change, however, as users were already responsible for installing Tracy requirements, which is system-specific.Regarding Cabal versions, sublibraries are supported since
cabal-version3.0, and we already plan to raise our lower bounds to (at least) that version. Note that public sublibraries were not stable untilcabal-install3.8, but that should not be a problem given thatcabal-install3.10.2.0 is the oldest version currently supported due to HSEC-2023-0015.The
accelerateandaccelerate:nofiblibraries would have the same version. There would be no need to test and maintain compatibility constraints between the two because they would always be in sync. Any changes toaccelerate:nofibwould require bumping the version of theacceleratepackage.Separate Package
The
accelerateandaccelerate-nofibpackages would have different versions. We would need to maintain compatibility constraints between the two.The
accelerate-llvm*packages use same-major constraints (such asaccelerate == 1.4.*, which can be written asaccelerate ^>=1.4since Cabal 2.0). Perhaps we would use similar constraints foraccelerate-nofib, requiring that the package must maintain compatibility with the same major versions of theaccelerateandaccelerate-llvm*packages. In this case, there is little benefit over using a public sublibrary.Alternatively,
accelerate-nofibcould be versioned completely separately. In this case, starting at0.1.0.0would make this obvious. Perhaps this is a good choice if development ofnofibis not very tightly coupled with development ofaccelerate, and a single version ofaccelerate-nofibwould support a wide range ofaccelerateversions.Adding another package increases maintenance burden.
No Hackage
Configuring the
accelerate-llvm-nativeandaccelerate-llvm-ptxtest suites to pullnofibsource directly from GitHub instead of using Hackage would work well if those packages were not on Hackage. They are on Hackage, however, so the dependencies should be on Hackage as well. I do not think that this is a viable option.Probabilistic Tests
Thank you for linking to accelerate-tests; I had not seen that repo yet!
One idea is to integrate such tests into the upstream repos, configuring them in separate test suites that use a Cabal flag so that they are only run when explicitly requested.
tomsmeding commentedon Sep 17, 2026
Agreed with much of what you say; it'd slipped my mind that
Customdoes not mesh with sublibraries.Very personally, I'd say that any change that reduces the amount of
Customis an improvement. However, achieving that is probably a bunch of work (refactoring to Hooks is not a smooth ride), and given my current involvement in Accelerate, my opinion is not the most important one.Historically it has been desirable for Accelerate to have very generous lower bounds on system dependencies (see here and the fact that until here we supported Cuda compute cababilities < 3.0). Depending on Cabal 3.14 would indeed be a significant departure from that. I personally don't care very much about that since ghcup exists and doesn't need root. Thus together with my previous personal opinion, if someone would find the time and energy to migrate
CustomtoHooks, I'd think that's awesome. :)... apart from not requiring us to eliminate
Custom, as desirable as that might be. In that case, however, I would suggest the monorepo approach (mirroring AccelerateHS/accelerate-llvm) to avoid the maintenance overhead, which is mostly nonexistent if there's just one maintainer but does exist with Accelerate's development model where PRs (e.g. by students) are a thing.ivogabe commentedon Sep 17, 2026
I think we should then first try to migrate to cabal hooks. The integration of Tracy in the Custom build is already fragile, so we definitely need to do something there, and the Custom build type also blocks other improvements as we saw in this thread. I created #579 to track that. @TravisCardwell perhaps you or your colleague could have a look at making this change.
This issue will then need to wait until we changed our build-type. If it is really not possible and we need to stick to the Custom build-type, then we can reconsider and choose between a mono-repo or new repo.
In my opinion it won't be a problem that we cannot release a new version of the test suite separate from Accelerate itself.
TravisCardwell commentedon Sep 18, 2026
I created a quick
Hooksprototype and discovered that it does not work with sublibraries (yet) either! There are also complications for Stack users. Please see #579 for details.ivogabe commentedon Oct 21, 2026
@Tjalle-S We are no longer using the Custom build-type and are already using components for the C ffi. This issue is thus unblocked; it should now be possible to move nofib to a separate component. Contributions are welcome if you still have time!
Tjalle-S commentedon Oct 22, 2026
I do have time, so I will start working on it soon.
ivogabe commentedon Oct 28, 2026
Thanks! Implemented in ivogabe#30