Skip to content

Couple of minor bugs #4

@mahesh973

Description

@mahesh973

Hello, while testing the models on the benchmark, I encountered a couple of minor bugs, so I thought of mentioning them here to get them fixed.

  1. In the evaluation/evaluate_ecgbench.py script, item dictionary does not contain key No; I think it should be id(according to the test split from ECGBench(Testing) benchmark on HuggingFace).
def eval_mmmu(dir):
    print("====mmmu====")
    with open("/path/to/PULSE/data/ECGBench/mmmu-ecg.json", "r", encoding='utf-8') as f:
        data = json.load(f)
        answer_dict = {item["No"]: item["conversations"][1]["value"] for item in data}
    
    score_dict = {}
    for root, dirs, files in os.walk(dir):
        for file in files:
            if file.endswith(".jsonl"):
                predict_list = []
                golden_list = []
                file_path = os.path.join(root, file)
                if "mmmu-ecg" not in file_path:
                    continue
  1. In the LLaVA/llava/eval/model_ecg_resume.py script, the line "text": line["conversations"][0]["value"].replace("<image>\n",""), causes AttributeError due to being a list; looks like it should be a string "text": "".join(line["conversations"][0]["value"]).replace("<image>\n",""), to fix the issue.
def eval_model(args):
    # Model
    disable_torch_init()
    model_path = args.model_path
    model_name = get_model_name_from_path(model_path)
    tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, args.model_base, model_name)
    
    questions = []
    with open(args.question_file, "r") as f:
        json_data = json.load(f)
        for line in json_data:
            print(line)
            questions.append({"question_id": line["id"], 
                              "image": line["image"], 
                              "text": line["conversations"][0]["value"].replace("<image>\n",""),
                              "ans": line["conversations"][1]["value"]})

Great work on the paper. Thank you!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions