Skip to content

Add MultiScale-Hybrid-mpnet submission - #182

Open
Vigora wants to merge 3 commits into
liamdugan:mainfrom
Vigora:main
Open

Add MultiScale-Hybrid-mpnet submission#182
Vigora wants to merge 3 commits into
liamdugan:mainfrom
Vigora:main

Conversation

@Vigora

@Vigora Vigora commented Aug 9, 2026

Copy link
Copy Markdown

This PR adds the MultiScale-Hybrid-mpnet detector submission to the RAID leaderboard.

The detector was trained on the RAID training split and should be categorized in the "trained on RAID" section.

Submission files:

  • metadata.json
  • predictions.json

@Vigora

Vigora commented Aug 11, 2026

Copy link
Copy Markdown
Author

Hi, just a gentle follow-up on this leaderboard submission. The evaluation workflow is still awaiting maintainer approval. Please let me know if anything is needed or should be changed on my side. Thank you!

@github-actions

Copy link
Copy Markdown

It looks like this eval run failed. Please check the workflow logs to see what went wrong, then push a new commit to your PR to rerun the eval.

@liamdugan

Copy link
Copy Markdown
Owner

Hi @Vigora it looks like there was an error evaluating your submission. Here's the full traceback

Traceback (most recent call last):
  File "/home/runner/work/raid/raid/leaderboard/hydrate.py", line 74, in hydrate_all
    result_fp = eval_submission(submission)
  File "/home/runner/work/raid/raid/leaderboard/hydrate.py", line 163, in eval_submission
    eval_results = run_evaluation(d, df, target_fpr=(0.05, 0.01))
  File "/home/runner/work/raid/raid/raid/evaluate.py", line 233, in run_evaluation
    df = load_detection_result(df, results)
  File "/home/runner/work/raid/raid/raid/evaluate.py", line 11, in load_detection_result
    scores_df = pd.DataFrame.from_records(results)
  File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/pandas/core/frame.py", line 2542, in from_records
    mgr = arrays_to_mgr(arrays, columns, result_index, typ=manager)
  File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/pandas/core/internals/construction.py", line 114, in arrays_to_mgr
    index = _extract_index(arrays)
  File "/opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/pandas/core/internals/construction.py", line 667, in _extract_index
    raise ValueError("If using all scalar values, you must pass an index")
ValueError: If using all scalar values, you must pass an index
Error: If using all scalar values, you must pass an index
Done. Wrote 0 results files.
Error: Process completed with exit code 1.

Can you make sure that you have a single floating point value in score for every single id present in the test data? Please check the other submissions for examples of the formatting needed for predictions.json

@Vigora
Vigora deployed to raid-main August 12, 2026 14:13 — with GitHub Actions Active
@Vigora

Vigora commented Aug 12, 2026

Copy link
Copy Markdown
Author

Hi Liam, thank you. I fixed predictions.json to the expected list-of-records format with one {"id": ..., "score": ...} record per prediction and pushed the correction to this PR. The new evaluation workflow is now awaiting approval.

@github-actions

Copy link
Copy Markdown

Eval run succeeded! Link to run: link

Here are the results of the submission(s):

MultiScale-Hybrid-mpnet

Release date: 2026-08-07

I've committed detailed results of this detector's performance on the test set to this PR.

On the RAID dataset as a whole (aggregated across all generation models, domains, decoding strategies, repetition penalties, and adversarial attacks), it achieved an AUROC of 90.66 and a TPR of 77.48% at FPR=5% and 59.90% at FPR=1%.
Without adversarial attacks, it achieved AUROC of 95.26 and a TPR of 88.46% at FPR=5% and 78.16% at FPR=1%.

If all looks well, a maintainer will come by soon to merge this PR and your entry/entries will appear on the leaderboard. If you need to make any changes, feel free to push new commits to this PR. Thanks for submitting to RAID!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants