Repository navigation
[suggest] please refactor data module #688
Description
Is your feature request related to a problem? Please describe.
After sending this PR #677, I think the data module should be refactor, which is hard to use for user-defined dataset.
Decouple data_type and data_path
In
Line 94 in 51d8006
dataset_name is actually data_type, then load data from HF.
Actually, user's data would not open-source. We have to manually change source code here.
Support multiple data path
In
,OpenAIFormatDataset.__init__ only supports single path, not list of data path. It does not match real scenario: training data comes from difference domains or teams.
llamafactory style (https://github.com/hiyouga/LLaMA-Factory/blob/main/data/dataset_info.json) may be better.
Dataset constructor definition
In
Line 116 in 51d8006
If you do not set system_key for OpenAIFormatDataset, it crashes. So I have to change the code:
data = hf_datasets.OpenAIFormatDataset(
data_config["train_data_path"],
data_config["val_data_path"],
**{k: data_config[k] for k in ("chat_key", "system_key", "system_prompt") if k in data_config}
)
Activity
tpoisonooo commentedon Jul 18, 2025
Take advantage of the fact that this repo is new — refactoring early is more beneficial.
snowmanwwg commentedon Jul 20, 2025
Also relating to #127
yuki-97 commentedon Aug 13, 2025
Hi @tpoisonooo , thanks for the suggestion!
Have a design for this at #909, please let me know if you have any suggestions or concerns.
snowmanwwg commentedon Aug 26, 2025
@tpoisonooo Hi there, thanks again. We have some preliminary designs and would like to go through with you to get some feedback. Are you open to meet virtually? We can also chat on wechat wenwengao2014. Looking forward to hearing from you!