Skip to content

agriculture, cs, legal这3个数据集在切块前的jsonl文件是不是缺少了? #13

Description

@GYY1o

大佬好,我看到file_chunk.py文件中:

if name == "main":
max_token_size=1024
overlap_token_size=128
original_text_file="datasets/mix/mix.jsonl"
chunk_text_file="datasets/mix/mix_chunk.json"
dataset='mix'

这部分切块要选择原始数据集,但是我只在mix这个数据集下找到了mix.jsonl文件和mix_chunk.json文件,但是缺少了query文件,其余3个数据集只有xxx_query.jsonl文件和大佬你做的消融实验结果,并没有发现切块前的数据或者切块后的数据,大佬可以分享一下吗?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions