Skip to content

Support asynchronous human preference gathering in RLHP implementation #716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 165 commits into
base: master
Choose a base branch
from
Open

Support asynchronous human preference gathering in RLHP implementation #716

wants to merge 165 commits into from

Conversation

@mschweizer
Copy link

@mschweizer mschweizer commented May 15, 2023

Description

Testing

We added extensive unit tests in test_preference_comparisons.py
Tests for 2 utility methods are currently missing and some existing tests still need fixing.

rk1a and others added 30 commits December 15, 2022 17:27
@rk1a
Adds RenderImageInfoWrapper
63e9112
@rk1a
Add PrefCollectGatherer
ed5362c
@rk1a
Adds configuration for post wrappers
9359ad4
@rk1a
Adds named_config for human preferences
eec88e0
@rk1a
Add experiment script for human preferences
346efde
@rk1a
Fix post wrappers config
e6a4400
@rk1a
Fix post wrappers config
4fa0726
@rk1a
Fix PrefCollect address
ae0087f
@rk1a
Merge branch 'human-pref-gatherer' of https://github.com/rk1a/imitation…
5257b11 
… into human-pref-gatherer
@rk1a
Extract preference querying into querent class
4b3968b
@rk1a
Merge branch 'human-pref-gatherer' into human-prefs
07c3866
@rk1a @mschweizer
Add querent and gatherer for human preferences
8ccaa71 
Co-authored-by: Robert Klassert <[email protected]>
Co-authored-by: Marvin Schweizer <[email protected]>
Add PreferenceQuerent tests, one for PrefCollectQuerent
3883a9f
Correct method signature
05f1da7
Test PreferenceGatherer and partially test SyntheticGatherer
7ba60c3
Fix bug
65acb03
Add gather preferences tests
2185a1e
Add pref collect gatherer tests
f7b22f2
@rk1a
Add PrefCollectGatherer tests
c0e884d
Add todos
fa587e5
Merge remote-tracking branch 'origin/human-prefs' into human-prefs
b27b8eb
@timbauman
Add support for a simple preference UI
9208b89
@timbauman
style
97c2171
@timbauman
style
5182acc
@timbauman
comments
f176bdc
@timbauman
add notebook
2540ba4
@timbauman
tutorial
3ff9579
@timbauman
lint
ce5efa7
@timbauman
lint
2ba2994
@timbauman
more lint
b3fd565
marvin.schweizer and others added 30 commits March 27, 2024 18:04
Extract handling of asynchronous preference collection to new parent …
389b719 
…class
Adapt SynchronousHumanGatherer to use new VideoBasedQuerent
9e57c81
Rename pref collect gatherer/querent to REST gatherer/querent
b7e5382
Merge branch 'human-prefs-base' into zoo-prefs-base
58a5e34 
# Conflicts:
#	src/imitation/algorithms/preference_comparisons.py
Adjust ZooniverseGatherer/Querent to new base classes
a951dc5
Remove Zooniverse classes
fb94706
Add empty _query method to VideoBasedQuerent
ee40083
Split video writing into smaller methods
cb1be27
Remove unused imports
0646dc7
@rk1a
Remove support for observations in video writing
09ddc0e
@rk1a
Merge pull request #3 from rk1a/refactor-human-prefs
e64dc00 
Refactor human prefs
@rk1a
Transfer video data via REST request and simplify VideoBasedQuerent
5d9a160
@rk1a
Fix tests as far as possible
a3175ac
Refactor Gatherer classes
7851d0f
@rk1a
Add test for video loading method
df49129
Fix bug in SyntheticGatherer
8a6e210
Adapt ConcretePreferenceGatherer
f934886
Improve variable naming
040af8b
Rename test for CommandLineGatherer
891e7f7
Fix test
4b9d1f9
Add documentation
2c632d8
Add documentation
eebb31d
Make method static
f2f722a
Remove whitespace
4c16550
@rk1a
Merge branch 'human-prefs-base' of https://github.com/rk1a/imitation …
7ba6e5f 
…into human-prefs-base
Merge remote-tracking branch 'origin/human-prefs-base' into human-pre…
f3fde8a 
…fs-base

# Conflicts:
#	src/imitation/algorithms/preference_comparisons.py
Refine rest interface and add documentation
7b2a260
Fix bug and integration test
38c7df9
Delete videos after test
fcfa92a
@rk1a
Merge pull request #4 from rk1a/human-prefs-base
d263216
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

3 more reviewers

@timokau timokau timokau left review comments

@Rocamonde Rocamonde Rocamonde left review comments

@rk1a rk1a rk1a left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

Release v1.x

Development

Successfully merging this pull request may close these issues.

7 participants

@mschweizer @rk1a @ernestum @timokau @Rocamonde @timbauman @dr-darryl-wright