assign_to_non_existent_node_social_net-analysis-1: the agent can analyze the problem correctly, but the expected fault type "dependency" is unclear

[KaminariOS]

It was able to gather enough info to tell that it is a node issue.
The agent response "misconfig" actually makes sense.

The eval failure is mostly caused by unclear definition of "denpency" and "misconfig", not the agent itself.

[gaganso]

Hi @KaminariOS, This is a good point. What do you think would be a good general solution for this?

The fault type taxonomy could be subjective. For an infrastructure engineer, this is a misconfiguation. For an application engineer, this is an issue in one of the dependencies of the application.

[KaminariOS]

The general problem is the lack of a partial scoring system.

The current eval metric is binary based on the final submission only, so it is very sensitive to how we define the correct answer.

Ideally, the scoring should be continuous and based on the trajectory, like you mentioned in #108.

I think other LLM agent benchmarks also faced a similar problem and I suggest tp have a look at their approaches.

[Tanaz-R]

Hello,
I face this issue as well, for example my agent for the problem `"Scale_pod_to_zero", submits "system_level": "Application", "fault_type": "Misconfiguration"}', in my point of view this is application deffination issue by user with misconfig, is that ok that in the evaluation code I add the valid combinations based on the which layer you should consider the system_level?

[HacksonClark]

Hello, I face this issue as well, for example my agent for the problem `"Scale_pod_to_zero", submits "system_level": "Application", "fault_type": "Misconfiguration"}', in my point of view this is application deffination issue by user with misconfig, is that ok that in the evaluation code I add the valid combinations based on the which layer you should consider the system_level?

I find that the RCA categories are quite arbitrary. Score them however you feel is reasonable.