Suggestion: a bar for accepting new frameworks

[nigrosimone]

Nobody asked for this, so take it as an outside opinion.

I think a framework should not be accepted below some threshold: a number of GitHub stars, or of downloads on the registry of its language where one exists. Without a bar the board fills up with experiments and weekend projects, and each of them costs you what a real entry costs: the whole profile matrix, on the 64-core rig, rerun every time that entry changes. That bill only grows, and it is what kills a project like this in the end.

This includes me. Fulmine is new and it is under any threshold you would pick.

If you would rather keep the door open, the price does not have to be popularity. It can be a contribution instead: a merged PR to HttpArena, or a donation. Something that costs the author half an hour or a few dollars.

An author who will spend neither is telling you what the entry is worth to them, and that is a fair signal to use, because you are the ones paying for the run.

[Kaliumhexacyanoferrat]

We discussed those issues a lot and changed the categorization system a few times, so I just wanted to put a few points that explain the reasoning of the current system:

• HTTP Arena is not as popular as we want it to be, so we are in need of any marketing / reach we can get, setting up obstacles will hurt that goal
• People submitting their own frameworks that yield good results are good messengers as they want their results to be seen
• Allowing people to fiddle around is very important for us (therefore the tuned and engine classifications). There can be no progress if we block it.
• Having entries is (currently) cheap, as we do not issue full runs, so the only costs are the PRs that integrate the data. We might do full calibration runs in the future though, on which set of frameworks has not been decided yet.
• The bar you have been mentioning would be the experimental classification. We require frameworks to be documented and provide a sane usage guide, or they will miss a lot of visibility.
• We are very aware of a lot of frameworks with no or very little adoption - we balance allowing them to be on the leadboard with having the leadboard being meaningless due to too many irrelevant entries. That is the main reason for the flagship filtering.
• We are also aware that there are a lot of unmaintained entries on the board. We consider re-running the validation to make the current state visible and probably take additional actions based on the results.

I feel those discussions are very valueable to the project overall, so maybe we can take that into Discord and continue there?

Also I have seen your blog post on dev.to - thank you for spreading the word 😉

[nigrosimone]

Fair enough, and the marketing point answers my issue better than my issue did.

On that, three ideas, take what is useful.

A badge. Something a framework puts in its own README: "HTTP Arena 4° JS", or the composite score, with your mark on it and a link to the board filtered on that entry. Every maintainer who adds it advertises you for free, and it is the kind of thing people copy from each other. The numbers are already in site/data/results/, so it is a shields.io endpoint JSON and little else. I can open the PR if you want it.

Posts. A dev.to or Medium account for the project, one post every two or three months: what moved on the board, who entered, who fell. And technical ones written by the maintainers themselves, each about the tricks of their own language. They bring their readers with them and it costs you the editing only.

1v1. Two entries, no fixed profile, raise the load until they break, and publish where each one gave up. A kind of HTTP Mortal Kombat. It is fun to read and it says something the tables do not.

[MDA2AV]

badges are now available, check project readme

[nigrosimone]

badges are now available, check project readme

Awesome!!!

My dreams is "HTTP Mortal Kombat" 😂

[MDA2AV]

badges are now available, check project readme

Awesome!!!

My dreams is "HTTP Mortal Kombat" 😂

raising the load would be increasing number of connections?

[nigrosimone]

Yes, my idea is to increase the connection count until one of the two servers dies. Whoever survives will continue the competition until we find the immortal.

[nigrosimone]

hoping the testing machine doesn't die first 😂