Skip to content

Temporary: open a PR for a pre-upgrade benchmark baseline - #696

Closed
de-code wants to merge 1 commit into
mainfrom
benchmark-baseline-biorxiv-elife-tf
Closed

Temporary: open a PR for a pre-upgrade benchmark baseline#696
de-code wants to merge 1 commit into
mainfrom
benchmark-baseline-biorxiv-elife-tf

Conversation

@de-code

@de-code de-code commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Carries no change to the package: a PR needs a diff against main to exist, and the benchmark runs on a labelled PR.

@de-code
Temporary: open a PR for a pre-upgrade benchmark baseline
380625c 
Carries no change to the package: a PR needs a diff against main to exist,
and the benchmark runs on a labelled PR.
@de-code de-code self-assigned this Aug 18, 2026
@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown

ScienceBeam Parser Evaluation

Overall (60 docs across 6 corpora)

grobid 0.9.0-crf (default): 60 docs | sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf): 60 docs | sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife): 60 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf) sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife) Δ grobid 0.9.0-crf (default) Δ sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf)
title (exact) string 0.582 0.572 0.598 +0.016 +0.025
title (levenshtein) string 0.657 0.657 0.681 +0.025 +0.025
title (edit_sim) string 0.701 0.701 0.701 -0.000 -0.000
abstract (levenshtein) string 0.709 0.709 0.748 +0.039 +0.039
abstract (edit_sim) string 0.721 0.716 0.768 +0.047 +0.052
author_full_names (levenshtein) partial_ulist 0.720 0.706 0.762 +0.042 +0.056
author_full_names (edit_sim) partial_ulist 0.746 0.735 0.780 +0.033 +0.045
affiliation_text (levenshtein) partial_ulist 0.471 0.446 0.524 +0.053 +0.078
affiliation_text (edit_sim) partial_ulist 0.582 0.544 0.609 +0.026 +0.065
keywords (levenshtein) partial_ulist 0.766 0.000 0.000 -0.766 +0.000
keywords (edit_sim) partial_ulist 0.695 0.000 0.000 -0.695 +0.000
body_section_titles (levenshtein) partial_list 0.346 0.328 0.429 +0.083 +0.101
body_section_titles (edit_sim) partial_list 0.353 0.345 0.404 +0.051 +0.059
acknowledgement (levenshtein) string 0.474 0.361 0.509 +0.036 +0.148
acknowledgement (edit_sim) string 0.473 0.356 0.502 +0.029 +0.146
first_reference_text (levenshtein) string 0.552 0.552 0.600 +0.048 +0.048
first_reference_text (edit_sim) string 0.667 0.666 0.712 +0.045 +0.046
reference_title (levenshtein) partial_list 0.455 0.466 0.626 +0.170 +0.159
reference_title (edit_sim) partial_list 0.462 0.480 0.623 +0.162 +0.143
reference_doi (levenshtein) partial_ulist 0.711 0.567 0.604 -0.107 +0.037
reference_doi (edit_sim) partial_ulist 0.668 0.532 0.586 -0.082 +0.054
biorxiv (10 docs)

grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf): 10 docs | sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife): 10 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf) sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife) Δ grobid 0.9.0-crf (default) Δ sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf)
title (exact) string 0.947 0.889 0.889 -0.058 +0.000
title (levenshtein) string 1.000 1.000 1.000 +0.000 +0.000
title (edit_sim) string 1.000 1.000 0.999 -0.001 -0.001
abstract (levenshtein) string 0.750 0.750 0.750 -0.000 -0.000
abstract (edit_sim) string 0.754 0.755 0.823 +0.069 +0.068
author_full_names (levenshtein) partial_ulist 0.994 0.994 0.983 -0.011 -0.011
author_full_names (edit_sim) partial_ulist 0.972 0.972 0.960 -0.011 -0.011
affiliation_text (levenshtein) partial_ulist 0.873 0.842 0.942 +0.070 +0.100
affiliation_text (edit_sim) partial_ulist 0.846 0.817 0.897 +0.050 +0.079
keywords (levenshtein) partial_ulist 0.966 0.000 0.000 -0.966 +0.000
keywords (edit_sim) partial_ulist 0.951 0.000 0.000 -0.951 +0.000
body_section_titles (levenshtein) partial_list 0.275 0.277 0.695 +0.420 +0.418
body_section_titles (edit_sim) partial_list 0.294 0.302 0.638 +0.344 +0.337
acknowledgement (levenshtein) string 0.875 0.500 0.889 +0.014 +0.389
acknowledgement (edit_sim) string 0.836 0.502 0.880 +0.045 +0.378
first_reference_text (levenshtein) string 0.947 0.947 0.947 +0.000 +0.000
first_reference_text (edit_sim) string 0.913 0.913 0.913 +0.000 +0.000
reference_title (levenshtein) partial_list 0.702 0.804 0.868 +0.166 +0.063
reference_title (edit_sim) partial_list 0.660 0.764 0.827 +0.167 +0.063
reference_doi (levenshtein) partial_ulist 0.964 0.942 0.970 +0.007 +0.028
reference_doi (edit_sim) partial_ulist 0.932 0.931 0.968 +0.035 +0.037
ore (10 docs)

grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf): 10 docs | sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife): 10 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf) sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife) Δ grobid 0.9.0-crf (default) Δ sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf)
title (exact) string 0.667 0.667 0.947 +0.281 +0.281
title (levenshtein) string 0.750 0.750 1.000 +0.250 +0.250
title (edit_sim) string 0.840 0.840 1.000 +0.160 +0.160
abstract (levenshtein) string 0.824 0.824 0.824 +0.000 +0.000
abstract (edit_sim) string 0.805 0.807 0.840 +0.035 +0.033
author_full_names (levenshtein) partial_ulist 0.783 0.777 0.938 +0.155 +0.161
author_full_names (edit_sim) partial_ulist 0.775 0.770 0.929 +0.154 +0.160
affiliation_text (levenshtein) partial_ulist 0.565 0.517 0.804 +0.239 +0.287
affiliation_text (edit_sim) partial_ulist 0.574 0.522 0.803 +0.229 +0.281
keywords (levenshtein) partial_ulist 0.500 0.000 0.000 -0.500 +0.000
keywords (edit_sim) partial_ulist 0.446 0.000 0.000 -0.446 +0.000
body_section_titles (levenshtein) partial_list 0.279 0.323 0.172 -0.107 -0.151
body_section_titles (edit_sim) partial_list 0.298 0.343 0.230 -0.068 -0.113
acknowledgement (levenshtein) string 0.833 0.833 0.833 +0.000 +0.000
acknowledgement (edit_sim) string 0.870 0.802 0.799 -0.072 -0.003
first_reference_text (levenshtein) string 0.333 0.333 0.333 +0.000 +0.000
first_reference_text (edit_sim) string 0.528 0.528 0.514 -0.014 -0.014
reference_title (levenshtein) partial_list 0.228 0.284 0.315 +0.087 +0.031
reference_title (edit_sim) partial_list 0.265 0.315 0.340 +0.074 +0.024
reference_doi (levenshtein) partial_ulist 0.704 0.012 0.012 -0.692 +0.000
reference_doi (edit_sim) partial_ulist 0.558 0.003 0.003 -0.555 +0.000
pkp (10 docs)

grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf): 10 docs | sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife): 10 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf) sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife) Δ grobid 0.9.0-crf (default) Δ sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf)
title (exact) string 0.750 0.750 0.667 -0.083 -0.083
title (levenshtein) string 0.824 0.824 0.824 -0.000 -0.000
title (edit_sim) string 0.805 0.805 0.770 -0.035 -0.035
abstract (levenshtein) string 0.933 0.933 0.750 -0.183 -0.183
abstract (edit_sim) string 0.914 0.915 0.779 -0.135 -0.136
author_full_names (levenshtein) partial_ulist 0.905 0.905 0.930 +0.025 +0.025
author_full_names (edit_sim) partial_ulist 0.917 0.917 0.941 +0.025 +0.025
affiliation_text (levenshtein) partial_ulist 0.667 0.600 0.714 +0.048 +0.114
affiliation_text (edit_sim) partial_ulist 0.743 0.666 0.756 +0.013 +0.090
keywords (levenshtein) partial_ulist 1.000 0.000 0.000 -1.000 +0.000
keywords (edit_sim) partial_ulist 1.000 0.000 0.000 -1.000 +0.000
body_section_titles (levenshtein) partial_list 0.678 0.714 0.702 +0.024 -0.013
body_section_titles (edit_sim) partial_list 0.650 0.685 0.644 -0.006 -0.042
acknowledgement (levenshtein) string 0.800 0.500 1.000 +0.200 +0.500
acknowledgement (edit_sim) string 0.800 0.500 1.000 +0.200 +0.500
first_reference_text (levenshtein) string 1.000 1.000 1.000 +0.000 +0.000
first_reference_text (edit_sim) string 0.995 0.995 0.986 -0.009 -0.009
reference_title (levenshtein) partial_list 0.799 0.770 0.853 +0.054 +0.084
reference_title (edit_sim) partial_list 0.770 0.740 0.838 +0.067 +0.097
reference_doi (levenshtein) partial_ulist 0.889 0.889 0.889 +0.000 +0.000
reference_doi (edit_sim) partial_ulist 0.889 0.889 0.889 +0.000 +0.000
scielo_br (10 docs)

grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf): 10 docs | sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife): 10 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf) sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife) Δ grobid 0.9.0-crf (default) Δ sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf)
title (exact) string 0.462 0.462 0.750 +0.288 +0.288
title (levenshtein) string 0.571 0.571 0.750 +0.179 +0.179
title (edit_sim) string 0.628 0.628 0.707 +0.079 +0.079
abstract (levenshtein) string 0.750 0.750 0.750 +0.000 +0.000
abstract (edit_sim) string 0.697 0.692 0.753 +0.057 +0.062
author_full_names (levenshtein) partial_ulist 0.582 0.500 0.533 -0.049 +0.033
author_full_names (edit_sim) partial_ulist 0.635 0.566 0.574 -0.061 +0.008
affiliation_text (levenshtein) partial_ulist 0.000 0.000 0.062 +0.062 +0.062
affiliation_text (edit_sim) partial_ulist 0.357 0.299 0.326 -0.031 +0.028
keywords (levenshtein) partial_ulist 0.614 0.000 0.000 -0.614 +0.000
keywords (edit_sim) partial_ulist 0.544 0.000 0.000 -0.544 +0.000
body_section_titles (levenshtein) partial_list 0.449 0.271 0.396 -0.053 +0.125
body_section_titles (edit_sim) partial_list 0.485 0.354 0.376 -0.109 +0.022
acknowledgement (levenshtein) string 0.000 0.000 0.000 +0.000 +0.000
acknowledgement (edit_sim) string 0.000 0.000 0.000 +0.000 +0.000
first_reference_text (levenshtein) string 0.000 0.000 0.000 +0.000 +0.000
first_reference_text (edit_sim) string 0.391 0.369 0.431 +0.040 +0.062
reference_title (levenshtein) partial_list 0.443 0.457 0.675 +0.232 +0.219
reference_title (edit_sim) partial_list 0.451 0.468 0.647 +0.196 +0.179
reference_doi (levenshtein) partial_ulist 0.965 0.824 1.000 +0.035 +0.176
reference_doi (edit_sim) partial_ulist 0.960 0.718 0.999 +0.040 +0.281
scielo_mx (10 docs)

grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf): 10 docs | sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife): 10 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf) sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife) Δ grobid 0.9.0-crf (default) Δ sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf)
title (exact) string 0.333 0.333 0.000 -0.333 -0.333
title (levenshtein) string 0.462 0.462 0.182 -0.280 -0.280
title (edit_sim) string 0.520 0.520 0.314 -0.207 -0.207
abstract (levenshtein) string 0.333 0.333 0.750 +0.417 +0.417
abstract (edit_sim) string 0.550 0.522 0.788 +0.239 +0.266
author_full_names (levenshtein) partial_ulist 0.627 0.627 0.615 -0.011 -0.011
author_full_names (edit_sim) partial_ulist 0.659 0.659 0.607 -0.052 -0.052
affiliation_text (levenshtein) partial_ulist 0.100 0.095 0.100 +0.000 +0.005
affiliation_text (edit_sim) partial_ulist 0.287 0.296 0.305 +0.019 +0.009
keywords (levenshtein) partial_ulist 0.814 0.000 0.000 -0.814 +0.000
keywords (edit_sim) partial_ulist 0.692 0.000 0.000 -0.692 +0.000
body_section_titles (levenshtein) partial_list 0.000 0.000 0.000 +0.000 +0.000
body_section_titles (edit_sim) partial_list 0.000 0.000 0.000 +0.000 +0.000
acknowledgement (levenshtein) string 0.000 0.000 0.000 +0.000 +0.000
acknowledgement (edit_sim) string 0.000 0.000 0.000 +0.000 +0.000
first_reference_text (levenshtein) string 0.571 0.571 0.571 -0.000 -0.000
first_reference_text (edit_sim) string 0.547 0.557 0.604 +0.057 +0.047
reference_title (levenshtein) partial_list 0.264 0.249 0.249 -0.015 -0.000
reference_title (edit_sim) partial_list 0.304 0.302 0.294 -0.010 -0.008
reference_doi (levenshtein) partial_ulist 0.000 0.000 0.000 +0.000 +0.000
reference_doi (edit_sim) partial_ulist 0.000 0.000 0.000 +0.000 +0.000
scielo_preprints-jats (10 docs)

grobid 0.9.0-crf (default): 10 docs | sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf): 10 docs | sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife): 10 docs

Field (method) Type grobid 0.9.0-crf (default) sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf) sciencebeam-parser:pr-696-05c18a70-20260818.0814 (biorxiv_elife) Δ grobid 0.9.0-crf (default) Δ sciencebeam-parser:main-7fdb7b32-20260817.1136 (grobid_crf)
title (exact) string 0.333 0.333 0.333 +0.000 +0.000
title (levenshtein) string 0.333 0.333 0.333 +0.000 +0.000
title (edit_sim) string 0.411 0.411 0.415 +0.003 +0.003
abstract (levenshtein) string 0.667 0.667 0.667 +0.000 +0.000
abstract (edit_sim) string 0.606 0.605 0.627 +0.021 +0.021
author_full_names (levenshtein) partial_ulist 0.431 0.435 0.575 +0.144 +0.140
author_full_names (edit_sim) partial_ulist 0.521 0.525 0.668 +0.147 +0.143
affiliation_text (levenshtein) partial_ulist 0.620 0.621 0.519 -0.101 -0.102
affiliation_text (edit_sim) partial_ulist 0.688 0.662 0.565 -0.122 -0.096
keywords (levenshtein) partial_ulist 0.704 0.000 0.000 -0.704 +0.000
keywords (edit_sim) partial_ulist 0.535 0.000 0.000 -0.535 +0.000
body_section_titles (levenshtein) partial_list 0.393 0.383 0.608 +0.215 +0.225
body_section_titles (edit_sim) partial_list 0.393 0.385 0.537 +0.145 +0.152
acknowledgement (levenshtein) string 0.333 0.333 0.333 +0.000 +0.000
acknowledgement (edit_sim) string 0.333 0.333 0.333 +0.000 +0.000
first_reference_text (levenshtein) string 0.462 0.462 0.750 +0.288 +0.288
first_reference_text (edit_sim) string 0.627 0.632 0.823 +0.196 +0.191
reference_title (levenshtein) partial_list 0.297 0.235 0.792 +0.496 +0.558
reference_title (edit_sim) partial_list 0.319 0.291 0.795 +0.476 +0.504
reference_doi (levenshtein) partial_ulist 0.744 0.735 0.753 +0.010 +0.018
reference_doi (edit_sim) partial_ulist 0.670 0.653 0.659 -0.010 +0.006

@de-code de-code closed this Aug 18, 2026
@de-code
de-code deleted the benchmark-baseline-biorxiv-elife-tf branch August 18, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@de-code de-code

Labels

benchmark:smoke profile:biorxiv_elife

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@de-code