Added an integtest to verify correct running of the system when just a couple of TPG flags are turned off

[bieryAtFnal]

Description

At the 23-Mar-2026 NP02 DAQ Operations meeting, Michal mentioned a bug in appmodel that caused problems when testing candidate software for fddaq-v5.6.0. He asked why the bug had not been caught in regression testing. (slides 12 and 13 in this presentation)

This PR is for a new regression test that would have caught the problem. I propose that we add this integtest to our test suite in the hope of catching similar problems in the future.

Here are suggestions for reviewing this new test. They include a demonstration that the test would have caught the appmodel bug from March.

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260501_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b kbiery/disabled_tpg_test
git clone https://github.com/DUNE-DAQ/appmodel -b develop
cd ..

dbt-workarea-env
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh -k disabled_tpg

echo ""
echo -e "\U1F535 \U2705 Note that the disabled_tpg_test worked fine with the latest appmodel code. \U2705 \U1F535"
echo ""
echo ""
sleep 3

cd sourcecode/appmodel
git checkout ba6d4d4
cd ../../

dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh -k disabled_tpg

echo ""
echo -e "\U1F535 \U2705 Note that disabled_tpg_test reports lots of errors when run with the faulty appmodel code. \U2705 \U1F535"
echo ""
echo ""

Type of change

• New feature or enhancement (non-breaking change which adds functionality)

Testing checklist

• Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas

[MRiganSUSX]

Thanks @bieryAtFnal for this.

I can confirm that following your instructions, I seen:

• successful run with new appmodel

===============================
Local 1x1 Conf-StandAloneSSH_PM
===============================
..
✅ Sanity-check passed for file test_raw_run000101_0000_df-01_dw_0_20260521T133652.hdf5
✅ All Attribute tests passed for file test_raw_run000101_0000_df-01_dw_0_20260521T133652.hdf5
✅ Record count 83 is within a tolerance of 8.0 from an expected value of 80.0
✅ Trigger Candidate fragment count of 1 confirmed in all 83 records
✅ All Trigger Candidate fragments in 83 records have sizes between 128 and 128
✅ HSI fragment count of 1 confirmed in all 83 records
✅ All HSI fragments in 83 records have sizes between [72, 100] and 100
✅ WIBEth fragment count of 4 confirmed in all 83 records
✅ All WIBEth fragments in 83 records have sizes between 7272 and 28872
✅ Trigger Primitive fragment count of 0 confirmed in all 83 records
.

========================= 3 passed in 95.86s (0:01:35) =========================

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Thu 21 May 15:38:00 CEST 2026
Log file is: /tmp/pytest-of-mrigan/dunedaq_integtest_bundle_20260521153621.log

⮕ Running daqsystemtest/disabled_tpg_test.py ⬅
========================= 3 passed ✅ in 95.86s (0:01:35) =========================

• seen it fail with the old appmodel commit:

===============================
Local 1x1 Conf-StandAloneSSH_PM
===============================
.----------
🚨 Problem(s) found in logfile /tmp/pytest-of-mrigan/pytest-3/run0/log_mrigan_local-1x1-config_ru-01.txt:
2026-May-21 15:48:39,769 ERROR [void dunedaq::dfmodules::FragmentAggregatorModule::process_data_request(dunedaq::dfmessages::DataRequest&) at /tmp/root/spack-stage/spack-stage-dfmodules-NB_DEV_260501_A9-65ggpl5ujspszs3dx3zyiqzpvbtbq2tf/spack-src/plugins/FragmentAggregatorModule.cpp:177] Unable to determine the Data Request message sender for SourceID [subsystem: Trigger id: 1000]. No DataRequest will be sent to this data source for run/trigger/sequence number 101/1/0.
...
🚨 Problem(s) found in logfile /tmp/pytest-of-mrigan/pytest-3/run0/log_mrigan_local-1x1-config_dfo-01.txt:
2026-May-21 15:49:03,030 ERROR [void dunedaq::dfmodules::DFOModule::do_stop(const dunedaq::appfwk::DAQModule::CommandData_t&) at /tmp/root/spack-stage/spack-stage-dfmodules-NB_DEV_260501_A9-65ggpl5ujspszs3dx3zyiqzpvbtbq2tf/spack-src/plugins/DFOModule.cpp:194] TriggerDecision 1 didn't complete within timeout in run 101
...
🚨 Problem(s) found in logfile /tmp/pytest-of-mrigan/pytest-3/run0/log_mrigan_local-1x1-config_df-01.txt:
2026-May-21 15:49:06,200 ERROR [dunedaq::dfmodules::TRBModule::trigger_record_ptr_t dunedaq::dfmodules::TRBModule::extract_trigger_record(const dunedaq::dfmodules::TriggerId&) at /tmp/root/spack-stage/spack-stage-dfmodules-NB_DEV_260501_A9-65ggpl5ujspszs3dx3zyiqzpvbtbq2tf/spack-src/plugins/TRBModule.cpp:480] sending incomplete TriggerRecord downstream at Stop time  (trigger/run_number=1-0/101, 7 of 10 fragments included)
...
🚨 Problem(s) found in logfile /tmp/pytest-of-mrigan/pytest-3/run0/log_mrigan_local-1x1-config_mlt.txt:
2026-May-21 15:48:40,766 WARNING [void dunedaq::trigger::MLTModule::trigger_decisions_callback(dunedaq::dfmessages::TriggerDecision&) at /tmp/root/spack-stage/spack-stage-trigger-NB_DEV_260501_A9-pg3ympudnfuxeh4sjqzxlwcxh7qdlb4u/spack-src/plugins/MLTModule.cpp:396] Trigger is inhibited in run 101
...
=================================== FAILURES ===================================
_________ test_log_files[Local 1x1 Conf-StandAloneSSH_PM-run_dunerc0] __________
sourcecode/daqsystemtest/integtest/disabled_tpg_test.py:156: in test_log_files
    assert log_file_checks.logs_are_error_free(
E   AssertionError: assert False
E    +  where False = <function logs_are_error_free at 0x7fad2e6fbf60>([PosixPath('/tmp/pytest-of-mrigan/pytest-3/run0/log_mrigan_SSH_SHELL_process_manager.txt'), PosixPath('/tmp/pytest-of-...onfig_ru-01.txt'), PosixPath('/tmp/pytest-of-mrigan/pytest-3/run0/log_mrigan_local-1x1-config_df-controller.txt'), ...], True, True, {'-controller': [], '.*': ['WARNING: All log messages before absl::InitializeLog\\(\\) is called are written to STDERR'], 'local-connection-server': ['errorlog: -']}, verbosity_helper=<integrationtest.verbosity_helper.VerbosityHelper object at 0x7fad2dc366c0>)
E    +    where <function logs_are_error_free at 0x7fad2e6fbf60> = log_file_checks.logs_are_error_free
E    +    and   [PosixPath('/tmp/pytest-of-mrigan/pytest-3/run0/log_mrigan_SSH_SHELL_process_manager.txt'), PosixPath('/tmp/pytest-of-...onfig_ru-01.txt'), PosixPath('/tmp/pytest-of-mrigan/pytest-3/run0/log_mrigan_local-1x1-config_df-controller.txt'), ...] = <integrationtest.integrationtest_drunc.run_dunerc.<locals>.RunResult object at 0x7fad2db900e0>.log_files
E    +    and   <integrationtest.verbosity_helper.VerbosityHelper object at 0x7fad2dc366c0> = <integrationtest.integrationtest_drunc.run_dunerc.<locals>.RunResult object at 0x7fad2db900e0>.verbosity_helper
_________ test_data_files[Local 1x1 Conf-StandAloneSSH_PM-run_dunerc0] _________
sourcecode/daqsystemtest/integtest/disabled_tpg_test.py:166: in test_data_files
    assert len(run_dunerc.data_files) == expected_file_count, f"Unexpected file count: Actual: {len(run_dunerc.data_files)}, Expected: {expected_file_count}"
E   AssertionError: Unexpected file count: Actual: 0, Expected: 1
E   assert 0 == 1
E    +  where 0 = len([])
E    +    where [] = <integrationtest.integrationtest_drunc.run_dunerc.<locals>.RunResult object at 0x7fad2db900e0>.data_files
=========================== short test summary info ============================
FAILED sourcecode/daqsystemtest/integtest/disabled_tpg_test.py::test_log_files[Local 1x1 Conf-StandAloneSSH_PM-run_dunerc0]
FAILED sourcecode/daqsystemtest/integtest/disabled_tpg_test.py::test_data_files[Local 1x1 Conf-StandAloneSSH_PM-run_dunerc0]
==================== 2 failed, 1 passed in 81.10s (0:01:21) ====================

🔵 Copying the files from failed test /tmp/pytest-of-mrigan/pytest-3 to /tmp/pytest-of-mrigan/failed-pytest-3. 🔵
🔵 Please note that copied directories from failed tests typically get cleaned up after 26 hours, 🔵
🔵 or when 10 newer failures happen, whichever comes first. 🔵

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Thu 21 May 15:49:46 CEST 2026
Log file is: /tmp/pytest-of-mrigan/dunedaq_integtest_bundle_20260521154822.log

⮕ Running daqsystemtest/disabled_tpg_test.py ⬅
==================== 2 failed ❌, 1 passed ✅ in 81.10s (0:01:21) ====================

Additional comment, I initially tried testing this with the most recent nightly (NFD_DEV_260521_A9) and there was another problem running the first test (new appmodel, your branch of daqsystemtest):

((dbt) ) [mrigan@np04-srv-016 NFD_DEV_260521_A9]$ dunedaq_integtest_bundle.sh -k disabled_tpg

Integtests from the _daqsystemtest_ repo will be run...

🔵 Starting test 1 of 1... 🔵
⮕ Running daqsystemtest/disabled_tpg_test.py ⬅
============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /nfs/home/mrigan/dune-daq/NFD_DEV_260521_A9/sourcecode/daqsystemtest
configfile: pytest.ini
plugins: anyio-4.12.1, integrationtest-4.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
_______________ ERROR collecting integtest/disabled_tpg_test.py ________________
sourcecode/daqsystemtest/integtest/disabled_tpg_test.py:93: in <module>
    common_config_obj = data_classes.drunc_config()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^
E   AttributeError: module 'integrationtest.data_classes' has no attribute 'drunc_config'
=========================== short test summary info ============================
ERROR sourcecode/daqsystemtest/integtest/disabled_tpg_test.py - AttributeErro...
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 1.23s ===============================

🟡 WARNING: Unable to copy the pytest directory for this failed test (). 🟡

+++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ SUMMARY ++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++

Thu 21 May 15:23:58 CEST 2026
Log file is: /tmp/pytest-of-mrigan/dunedaq_integtest_bundle_20260521152354.log

If you understand this problem and it is not related, then I am very happy for this to be merged.

[bieryAtFnal]

Yes, the problem with using the latest nightly build was that the integrationtest infrastructure has changed since this PR was opened.
I've re-merged the develop branch into this branch and modified the new test to use the new confgen data_class.