Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataset v2.0 #461

Open
wants to merge 103 commits into
base: main
Choose a base branch
from
Open

Dataset v2.0 #461

wants to merge 103 commits into from

Commits on Oct 3, 2024

  1. WIP

    @aliberts
    aliberts committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    ad115b6 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Merge remote-tracking branch 'origin/main' into user/aliberts/2024_09… 

    …_25_reshape_dataset
    @aliberts
    aliberts committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    17a1214 View commit details
    Browse the repository at this point in the history

  2. Add upload folders

    @aliberts
    aliberts committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    1016a98 View commit details
    Browse the repository at this point in the history

  3. Add info.json link

    @aliberts
    aliberts committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    07e113c View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' into user/aliberts/2024_09… 

    …_25_reshape_dataset
    @aliberts
    aliberts committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    028c17f View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2024

  1. Add pixel channels

    @aliberts
    aliberts committed Oct 6, 2024
    Configuration menu
    Copy the full SHA
    21ba4b5 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. Update info.json format

    @aliberts
    aliberts committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    2d75b93 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. Rework LeRobotDataset.__init__

    @aliberts
    aliberts committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    096824b View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'origin/main' into user/aliberts/2024_09… 

    …_25_reshape_dataset
    @aliberts
    aliberts committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    3113038 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Update LeRobotDataset.__get_item__

    @aliberts
    aliberts committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    b417ceb View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2024

  1. Add doc, scrap video_frame_keys attribute

    @aliberts
    aliberts committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    6d2bc11 View commit details
    Browse the repository at this point in the history

  2. Add huggingface-hub patch for offline snapshot_download with local_dir

    @aliberts
    aliberts committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    7f68088 View commit details
    Browse the repository at this point in the history

  3. Add padding keys and download_data option

    @aliberts
    aliberts committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    3ea5312 View commit details
    Browse the repository at this point in the history

  4. Add suggestions from code review

    @aliberts
    aliberts committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    8bd406e View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2024

  1. Add multitask support, refactor conversion script

    @aliberts
    aliberts committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    cf63334 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Extend v1 compatibility

    @aliberts
    aliberts committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    cbc51e1 View commit details
    Browse the repository at this point in the history

  2. Fix safe_version

    @aliberts
    aliberts committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    f96773d View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Cleanup, fix load_tasks

    @aliberts
    aliberts committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    835ab5a View commit details
    Browse the repository at this point in the history

  2. Update load_tasks doc

    @aliberts
    aliberts committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    da78bbf View commit details
    Browse the repository at this point in the history

  3. WIP add batch convert

    @aliberts
    aliberts committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    9433ac5 View commit details
    Browse the repository at this point in the history

  4. Add fixes for batch convert

    @aliberts
    aliberts committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    1102640 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Add episode chunks logic, move_videos & lfs tracking fix

    @aliberts
    aliberts committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    c146ba9 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. Write episodes as jsonlines

    @aliberts
    aliberts committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    50a75ad View commit details
    Browse the repository at this point in the history

  2. Add fixes for lfs tracking

    @aliberts
    aliberts committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    ad3f112 View commit details
    Browse the repository at this point in the history

  3. Add batch conversion log

    @aliberts
    aliberts committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    3ee3739 View commit details
    Browse the repository at this point in the history

  4. Cleanup

    @aliberts
    aliberts committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    7242c57 View commit details
    Browse the repository at this point in the history

  5. Add unitreeh and umi

    @aliberts
    aliberts committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    d0d8193 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/main' into user/aliberts/2024_09… 

    …_25_reshape_dataset
    @aliberts
    aliberts committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    354f37a View commit details
    Browse the repository at this point in the history

  7. Update doc

    @aliberts
    aliberts committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    be64d54 View commit details
    Browse the repository at this point in the history

  8. Cleanup

    @aliberts
    aliberts committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    beacb7e View commit details
    Browse the repository at this point in the history

  9. Add copyrights

    @aliberts
    aliberts committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    3a9f964 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. Remove caret requirement

    @aliberts
    aliberts committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    91e8ce7 View commit details
    Browse the repository at this point in the history

  2. Fix episodes.jsonl

    @aliberts
    aliberts committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    e7355ba View commit details
    Browse the repository at this point in the history

  3. Add download_metadata, move default paths

    @aliberts
    aliberts committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    1a51505 View commit details
    Browse the repository at this point in the history

  4. Add load_metadata

    @aliberts
    aliberts committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    bce3dc3 View commit details
    Browse the repository at this point in the history

  5. Move default paths, use jsonlines for tasks

    @aliberts
    aliberts committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    ac3798b View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2024

  1. Add file paths

    @aliberts
    aliberts committed Oct 20, 2024
    Configuration menu
    Copy the full SHA
    9316cf4 View commit details
    Browse the repository at this point in the history

  2. Change card creation

    @aliberts
    aliberts committed Oct 20, 2024
    Configuration menu
    Copy the full SHA
    e46bdb9 View commit details
    Browse the repository at this point in the history

  3. Add ImageWriter

    @aliberts
    aliberts committed Oct 20, 2024
    Configuration menu
    Copy the full SHA
    3b925c3 View commit details
    Browse the repository at this point in the history

  4. Add add_frame, empty dataset creation

    @aliberts
    aliberts committed Oct 20, 2024
    Configuration menu
    Copy the full SHA
    c1232a0 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/main' into user/aliberts/2024_09… 

    …_25_reshape_dataset
    @aliberts
    aliberts committed Oct 20, 2024
    Configuration menu
    Copy the full SHA
    9ebf8b8 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Add add_episode & task logic

    @aliberts
    aliberts committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    299451a View commit details
    Browse the repository at this point in the history

  2. add delete_episode, WIP on consolidate

    @aliberts
    aliberts committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    c4c0a43 View commit details
    Browse the repository at this point in the history

  3. Improve consistency between __init__() and create(), WIP on consolidate

    @aliberts
    aliberts committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    e991a31 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Add local_files_only, encode_videos, fix bugs to pass tests (WIP)

    @aliberts
    aliberts committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    a805458 View commit details
    Browse the repository at this point in the history

  2. Add channels to intelrealsense

    @aliberts
    aliberts committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    ee52b8b View commit details
    Browse the repository at this point in the history

  3. Fix tests

    @aliberts
    aliberts committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    b46db7e View commit details
    Browse the repository at this point in the history

  4. Remove populate dataset

    @aliberts
    aliberts committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    6c2cb6e View commit details
    Browse the repository at this point in the history

  5. Fix paths & add add_frame doc

    @aliberts
    aliberts committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    237a484 View commit details
    Browse the repository at this point in the history

  6. Remove total_episodes from default parquet path

    @aliberts
    aliberts committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    c72dc23 View commit details
    Browse the repository at this point in the history

  7. Update & fix conversion script

    @aliberts
    aliberts committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    c3c0141 View commit details
    Browse the repository at this point in the history

  8. Fix episode chunk

    @aliberts
    aliberts committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    9dca233 View commit details
    Browse the repository at this point in the history

  9. Update dataset doc

    @aliberts
    aliberts committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    fb73cdb View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. add write_stats, changes names, add some typing

    @aliberts
    aliberts committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    a2a8538 View commit details
    Browse the repository at this point in the history

  2. Fix visualization

    @aliberts
    aliberts committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    7ae8d05 View commit details
    Browse the repository at this point in the history

  3. Fix check_delta_timestamps

    @aliberts
    aliberts committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    b8bdbc1 View commit details
    Browse the repository at this point in the history

  4. Fix _query_videos return shapes

    @aliberts
    aliberts committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    07570f8 View commit details
    Browse the repository at this point in the history

  5. Fix consolidate

    @aliberts
    aliberts committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    1aba80d View commit details
    Browse the repository at this point in the history

  6. Nits

    @aliberts
    aliberts committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    0098bd2 View commit details
    Browse the repository at this point in the history

  7. Move ImageWriter creation inside the dataset

    @aliberts
    aliberts committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    0d77be9 View commit details
    Browse the repository at this point in the history

  8. Allow dataset creation without robot

    @aliberts
    aliberts committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    60865e8 View commit details
    Browse the repository at this point in the history

  9. Add error msg

    @aliberts
    aliberts committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    450eae3 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Add test_same_attributes_defined

    @aliberts
    aliberts committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    615894d View commit details
    Browse the repository at this point in the history

  2. Add todo

    @aliberts
    aliberts committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    8bcf81f View commit details
    Browse the repository at this point in the history

  3. Add json/jsonl io functions

    @aliberts
    aliberts committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    18ffa42 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Add video_info, fix image_writer

    @aliberts
    aliberts committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    e210d79 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2024

  1. Speedup test

    @aliberts
    aliberts committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    df3d2ec View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. Fix image writer

    @aliberts
    aliberts committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    51e87f6 View commit details
    Browse the repository at this point in the history

  2. Add sanity check

    @aliberts
    aliberts committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    4c22de2 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. Merge remote-tracking branch 'origin/main' into user/aliberts/2024_09… 

    …_25_reshape_dataset
    @aliberts
    aliberts committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    8d57093 View commit details
    Browse the repository at this point in the history

  2. Remove image_writer arg

    @aliberts
    aliberts committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    fee5fa5 View commit details
    Browse the repository at this point in the history

  3. Remove dataset from image_transform tests

    @aliberts
    aliberts committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    ee51f54 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. Add dataset fixtures

    @aliberts
    aliberts committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    ff84024 View commit details
    Browse the repository at this point in the history

  2. Add test_delta_timestamps.py

    @aliberts
    aliberts committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    e69f0c5 View commit details
    Browse the repository at this point in the history

  3. Update doc

    @aliberts
    aliberts committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    e1845d4 View commit details
    Browse the repository at this point in the history

  4. Update doc

    @aliberts
    aliberts committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    c70b8d0 View commit details
    Browse the repository at this point in the history

  5. Split fixtures into factories and files

    @aliberts
    aliberts committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    1267c3e View commit details
    Browse the repository at this point in the history

  6. Add fixtures in test_datasets

    @aliberts
    aliberts committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    ab23a4f View commit details
    Browse the repository at this point in the history

  7. Remove/comment obsolete tests

    @aliberts
    aliberts committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    443a9ee View commit details
    Browse the repository at this point in the history

  8. Remove obsolete code

    @aliberts
    aliberts committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    5ea7c78 View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. Mock snapshot_download

    @aliberts
    aliberts committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    cd1509d View commit details
    Browse the repository at this point in the history

  2. Add tasks and episodes factories

    @aliberts
    aliberts committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    2650872 View commit details
    Browse the repository at this point in the history

  3. Rename num_samples -> num_frames for consistency

    @aliberts
    aliberts committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    79d114c View commit details
    Browse the repository at this point in the history

  4. Simplify, add test content, add todo

    @aliberts
    aliberts committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    293bdc7 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2024

  1. Add img and img_tensor factories

    @aliberts
    aliberts committed Nov 2, 2024
    Configuration menu
    Copy the full SHA
    375abd3 View commit details
    Browse the repository at this point in the history

  2. Add test_image_writer, accept PIL images, improve ImageWriter perf in… 

    … main process
    @aliberts
    aliberts committed Nov 2, 2024
    Configuration menu
    Copy the full SHA
    6b2ec1e View commit details
    Browse the repository at this point in the history

  3. Add more options to img factories

    @aliberts
    aliberts committed Nov 2, 2024
    Configuration menu
    Copy the full SHA
    7a342db View commit details
    Browse the repository at this point in the history

  4. Add todo in skipped test

    @aliberts
    aliberts committed Nov 2, 2024
    Configuration menu
    Copy the full SHA
    df2cb51 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2024

  1. Fix test_online_buffer.py

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    ac79e8c View commit details
    Browse the repository at this point in the history

  2. Add LeRobotDatasetMetadata

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    e4ba084 View commit details
    Browse the repository at this point in the history

  3. Fix hanging

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    16103cb View commit details
    Browse the repository at this point in the history

  4. Fix vizualize

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    95a4b59 View commit details
    Browse the repository at this point in the history

  5. Fix werkzeug alert

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    c2d6fb6 View commit details
    Browse the repository at this point in the history

  6. Remove end-to-end tests

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    f6c90ca View commit details
    Browse the repository at this point in the history

  7. Deactivate policies backward compatibility test

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    56e4603 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'origin/main' into user/aliberts/2024_09… 

    …_25_reshape_dataset
    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    fde29e0 View commit details
    Browse the repository at this point in the history

  9. Fix advanced example 2

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    a6762ec View commit details
    Browse the repository at this point in the history

  10. Remove reset_episode_index

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    74270c8 View commit details
    Browse the repository at this point in the history

  11. Move calculate_episode_data_index

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    7b159a6 View commit details
    Browse the repository at this point in the history

  12. Fix test_examples

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    b69a132 View commit details
    Browse the repository at this point in the history

  13. Fix test_examples

    @aliberts
    aliberts committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    757ea17 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2024

  1. Refactor dataset features

    @aliberts
    aliberts committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    aed9f40 View commit details
    Browse the repository at this point in the history

  2. Fix tests

    @aliberts
    aliberts committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    f3630ad View commit details
    Browse the repository at this point in the history