Skip to content

Major Merge of Linfeng's project on "Seeing is Believing" #303

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lf-zhao wants to merge 499 commits into
base: master
from
Open

Major Merge of Linfeng's project on "Seeing is Believing" #303

Changes from all commits
Commits
Show all changes
499 commits
Select commit Hold shift + click to select a range
8ded816
add unposed image dataclass
lf-zhao Jan 21, 2025
ca205e9
add UnposedImageWithContext
lf-zhao Jan 21, 2025
51dee2a
major update, now collector can complete collection and running: upda…
lf-zhao Jan 21, 2025
aa8a1c6
major update to mock env: update Mock Obs classes for saving and for …
lf-zhao Jan 21, 2025
8c8382c
move mock obs into separate file to avoid cyclic import
lf-zhao Jan 21, 2025
d695b8b
the manual creator class is basically dummy now
lf-zhao Jan 21, 2025
7a1683d
minor
lf-zhao Jan 21, 2025
3812630
new docs: guide of testing planning using mock robot env
lf-zhao Jan 21, 2025
cc4e0c2
update to perceiver
lf-zhao Jan 21, 2025
915bf81
test on loading
lf-zhao Jan 21, 2025
f5981f6
major update to test manual collected env: fully working
lf-zhao Jan 21, 2025
4c910cf
update another test - this may be outdated
lf-zhao Jan 21, 2025
c845572
update docs on perceiver
lf-zhao Jan 21, 2025
6a169f3
[in progress] major update to mock perceiver
lf-zhao Jan 21, 2025
40bf20c
major update to mock spot perceiver: correctly handle predicate value…
lf-zhao Jan 22, 2025
b683ec6
minor
lf-zhao Jan 22, 2025
a5424af
minor
lf-zhao Jan 22, 2025
ea93041
major update: add consistency check for unknown/known, make sure upda…
lf-zhao Jan 22, 2025
43fbd44
working on updating the ground atoms saving logic
lf-zhao Jan 22, 2025
e4af8e4
update
lf-zhao Jan 22, 2025
aa55d28
add new field for saving atom and new GroundTruthPredicate class for …
lf-zhao Jan 22, 2025
a5980c8
major update: clean up state storage logic, separate state graph buil…
lf-zhao Jan 22, 2025
b0a396a
update the test for saving env with images
lf-zhao Jan 22, 2025
2f57972
update previous test
lf-zhao Jan 22, 2025
fee4c9f
update test
lf-zhao Jan 22, 2025
6fec032
update; not fully working with latest ver
lf-zhao Jan 22, 2025
d82f218
update test for new init
lf-zhao Jan 22, 2025
e6e3975
new utils for computing ground atom combinations without state
lf-zhao Jan 22, 2025
facabb9
major update to env: use GroundTruePredicate that directly loads from…
lf-zhao Jan 22, 2025
125a1c4
fixng saving and loading
lf-zhao Jan 22, 2025
1f99a49
udpate sacing loading logic fro perceiver
lf-zhao Jan 22, 2025
a3e2a3a
major update: loading environment state/transition system, lots of cl…
lf-zhao Jan 22, 2025
590b24e
test new perciever with the env
lf-zhao Jan 22, 2025
76295fe
minor
lf-zhao Jan 22, 2025
f414e3b
major update: wrote logsic for saving spot images [in progress]
lf-zhao Jan 22, 2025
b2a6e00
update saving paths
lf-zhao Jan 22, 2025
5026803
update test for planning
lf-zhao Jan 22, 2025
67d567a
update docs for perceiver
lf-zhao Jan 22, 2025
adcef75
note on creating new mock env tasks
lf-zhao Jan 22, 2025
8ec0416
update flag name to use VLM predicates
lf-zhao Jan 23, 2025
f2bec9f
fix saving yaml
lf-zhao Jan 23, 2025
4662c53
upate flag
lf-zhao Jan 23, 2025
f2bfddf
update docs for action wrapping for operators
lf-zhao Jan 23, 2025
4a4da9a
update fixing VLM pred evaluation
lf-zhao Jan 23, 2025
3cd3bb6
fix for vlm pred
lf-zhao Jan 23, 2025
9f7c4a9
fix for objects
lf-zhao Jan 23, 2025
2b2f0c6
update for env input
lf-zhao Jan 23, 2025
e1f39bd
update
lf-zhao Jan 23, 2025
5123b7c
update mock perceiver test, fixed vlm predicate stuf
lf-zhao Jan 23, 2025
d2ee262
major update to testing planning with perceiver on mock env
lf-zhao Jan 23, 2025
32fb086
fix timer
lf-zhao Jan 23, 2025
7be5698
fix saving non-VLM atoms, move predicate definition to the file
lf-zhao Jan 23, 2025
823fda8
moved predicate definitions
lf-zhao Jan 23, 2025
4ff16d9
update test for mock perceiver
lf-zhao Jan 23, 2025
1999b4a
update struct
lf-zhao Jan 23, 2025
7a4c5bb
minor fix
lf-zhao Jan 23, 2025
0a8f981
fix objects passing in; fix missing env names; fix task init; fix som…
lf-zhao Jan 23, 2025
604ca82
fix: add new env to factory
lf-zhao Jan 23, 2025
4944137
fix object info missing; add logging with rich table for vlm pred
lf-zhao Jan 23, 2025
81b4f2a
[in progress] test planning, need to feed right images
lf-zhao Jan 23, 2025
17ce627
update to process HEIC images from iphone
lf-zhao Jan 23, 2025
7117789
update test
lf-zhao Jan 23, 2025
c19262f
add a new "test" file for 2 cup pick place with images from iphone
lf-zhao Jan 23, 2025
c1d7a28
minor fixes and check
lf-zhao Jan 23, 2025
40ec0a7
minor bugs on action space and overrides
lf-zhao Jan 23, 2025
1596900
one mock env task for debug: for pick place 2 cups
lf-zhao Jan 23, 2025
b58697f
update openai
lf-zhao Jan 23, 2025
b66e7f8
minor
lf-zhao Jan 23, 2025
13f4632
add logging for environment side transitions
lf-zhao Jan 24, 2025
92c0c49
add information of objects into action info
lf-zhao Jan 24, 2025
8add8c0
add a flag for llm verbose
lf-zhao Jan 24, 2025
0a0606a
add test source images
lf-zhao Jan 24, 2025
d5c0172
add implementation note for mock spot env
lf-zhao Jan 24, 2025
1e64726
add implementation note for mock env creator
lf-zhao Jan 24, 2025
b7dce7f
add implementation note and docs for belief operators and predicates …
lf-zhao Jan 24, 2025
b2446a1
add more logging for llm planner
lf-zhao Jan 24, 2025
0436088
Merge remote-tracking branch 'origin/master' into lis-spot/202501-mer…
lf-zhao Jan 24, 2025
5062db6
deal with more merge+edits for VLM predicate + labeling related: my v…
lf-zhao Jan 24, 2025
209a552
update code missed in merging
lf-zhao Jan 24, 2025
0efb1ea
minor fixes
lf-zhao Jan 24, 2025
1513a2a
move creator functions to model interface, avoiding circular imports
lf-zhao Jan 24, 2025
6d354e0
add record for cost for openai models
lf-zhao Jan 24, 2025
8d5ace6
minor fix
lf-zhao Jan 24, 2025
4ec5f4b
fix vlm open loop planner
lf-zhao Jan 24, 2025
5301a44
fix wrongly init train task
lf-zhao Jan 24, 2025
026ae58
fix; no offline training for LLM learning based method - we only want…
lf-zhao Jan 24, 2025
3b6985e
update
lf-zhao Jan 24, 2025
8c435bb
fix cyclic imports for vlm creation
lf-zhao Jan 24, 2025
2e334bc
save number of actual online execution steps
lf-zhao Jan 25, 2025
ba8e3f8
update
lf-zhao Jan 25, 2025
77b3d3c
debug for random options approach
lf-zhao Jan 25, 2025
d774eed
add back
lf-zhao Jan 25, 2025
3192b50
update for saving results
lf-zhao Jan 25, 2025
1a6c59a
logging plans and replanned plans for debugging
lf-zhao Jan 25, 2025
92aa9b6
add some debug information for logging
lf-zhao Jan 25, 2025
675b772
[to check] add a vlm bilevel planning approach, not sure if correct; …
lf-zhao Jan 25, 2025
cbcaae5
[wip] add saving optimal plan for checking
lf-zhao Jan 25, 2025
3639f8e
update log
lf-zhao Jan 25, 2025
d5b3835
add note for foundation model planning approaches
lf-zhao Jan 25, 2025
6bc69ba
llm planner update v1
lf-zhao Jan 25, 2025
33afc8e
update llm planner to give oracle NSRTs for fair comparison
lf-zhao Jan 25, 2025
f8c9c50
also update vlm planner to include oracle NSRTS with pre/eff in prompt
lf-zhao Jan 25, 2025
7417818
update saving
lf-zhao Jan 25, 2025
c9ebb91
add new llm zero shot prompt
lf-zhao Jan 25, 2025
9b68d7e
update args for llm vlm planning
lf-zhao Jan 25, 2025
7cb15e3
major update: fix LLM open loop planner approach - based on latest VL…
lf-zhao Jan 27, 2025
2760952
update factory to include new task
lf-zhao Jan 27, 2025
15e37be
add new task for drawer cleaning
lf-zhao Jan 27, 2025
aa79a95
update name for pick place
lf-zhao Jan 27, 2025
997ec28
minor
lf-zhao Jan 27, 2025
8f27ac7
add a script for running all approaches on 2 cup pick place example
lf-zhao Jan 27, 2025
a7b25c7
add vlm and llm approaches for using expected atom check, otherwise t…
lf-zhao Jan 28, 2025
7e60d13
fix creation for 2 cup pick place
lf-zhao Jan 28, 2025
718b1bc
fix for cyclic import
lf-zhao Jan 28, 2025
af0fcf0
update script
lf-zhao Jan 28, 2025
b154742
major update: add CLI mock env creator, iterating over canonical pred…
lf-zhao Jan 28, 2025
3737e4b
major update: support iterating over canonical states; major update t…
lf-zhao Jan 28, 2025
e595ca4
add manual test for building drawer cleaning task - now we don't need…
lf-zhao Jan 28, 2025
6904b80
update for drawer cleaning task
lf-zhao Jan 28, 2025
2954984
upload images
lf-zhao Jan 28, 2025
d2d93ef
source images for drawer cleaning task
lf-zhao Jan 28, 2025
8d6b73a
add saving for objects in view - useful for PO tasks!
lf-zhao Jan 28, 2025
44d1287
minor fix
lf-zhao Jan 29, 2025
d442018
fix for random
lf-zhao Jan 29, 2025
e53cb1e
fix
lf-zhao Jan 29, 2025
6f51cab
minor
lf-zhao Jan 29, 2025
c234434
update/fix for llm open loop - now include state atoms in prompt
lf-zhao Jan 29, 2025
c80bd33
also the prompt
lf-zhao Jan 29, 2025
442a49d
major, WIP: add VLM perceiver and LLM text planner; add prompt
lf-zhao Jan 29, 2025
2d039a5
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 29, 2025
cc62d86
not sure why but this was probably not committed
lf-zhao Jan 29, 2025
7472ad3
push fix when visible objects are none
lf-zhao Jan 29, 2025
506e24e
update - this is not vlm, deprecate
lf-zhao Jan 29, 2025
4cd046c
add photos for task of removing empty boxes with scale
lf-zhao Jan 29, 2025
485a4f2
add action history to prompts
lf-zhao Jan 29, 2025
9b2f02c
include action history
lf-zhao Jan 29, 2025
9b4c9e3
fix action history; for LLM approach first
lf-zhao Jan 29, 2025
6ed9f04
major, finish and fix: vlm planner with action history and observatio…
lf-zhao Jan 29, 2025
ff3ede0
update prompts
lf-zhao Jan 29, 2025
8be4cae
minor update for history
lf-zhao Jan 29, 2025
116bcbf
test for vlm perceiver
lf-zhao Jan 29, 2025
a68d3de
updates
aidan-curtis Jan 29, 2025
5f2ed0b
baseline implemented
aidan-curtis Jan 29, 2025
9ea1ae1
updates to captioning baseline
aidan-curtis Jan 29, 2025
c4a2271
fix perceiver for saving image+action history
lf-zhao Jan 29, 2025
f7672cd
fix in reset as well
lf-zhao Jan 29, 2025
d39dbd1
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 29, 2025
68a14a1
searate action history maintain
lf-zhao Jan 29, 2025
2d416ee
some small updates
aidan-curtis Jan 29, 2025
e09808e
updates
aidan-curtis Jan 29, 2025
8c9b552
fix camera image history for vlm perceiver
lf-zhao Jan 29, 2025
72c9a30
we shouldn't need this anymore
lf-zhao Jan 29, 2025
abc5b4d
update vlm perceiver for image history
lf-zhao Jan 29, 2025
ade21c5
updates to experiments
aidan-curtis Jan 29, 2025
4c1231d
monir
lf-zhao Jan 30, 2025
b848f06
fix: now only use active predicates in active operators
lf-zhao Jan 30, 2025
51e2e6e
fix image load
lf-zhao Jan 30, 2025
d989d2f
major update: fix VLM belief predicates, update predicate exports, ge…
lf-zhao Jan 30, 2025
fbf9e11
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 30, 2025
b699acd
mock modifications
aidan-curtis Jan 30, 2025
0808c6f
updates
aidan-curtis Jan 30, 2025
18c1498
minor update
lf-zhao Jan 30, 2025
67feab4
update hand view
lf-zhao Jan 30, 2025
d3afb5d
[major update spot env] update predicates and operators, update task
lf-zhao Jan 30, 2025
9360c4c
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 30, 2025
6bccea5
updates
aidan-curtis Jan 30, 2025
7832600
fix bug
aidan-curtis Jan 30, 2025
99963ac
updates
aidan-curtis Jan 30, 2025
83f17cf
uncomment all
aidan-curtis Jan 30, 2025
a433ada
updates
aidan-curtis Jan 30, 2025
e232df8
updates
aidan-curtis Jan 31, 2025
a841411
save
lf-zhao Jan 31, 2025
c854a14
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 31, 2025
e31356b
minor fixes/updates
lf-zhao Jan 31, 2025
45bf321
test
lf-zhao Jan 31, 2025
755a8cd
or atoms untested
aidan-curtis Jan 31, 2025
ea80d60
good transition graph
aidan-curtis Jan 31, 2025
e598250
move this to utils
lf-zhao Jan 31, 2025
b88e064
minor update
lf-zhao Jan 31, 2025
826938b
fix
lf-zhao Jan 31, 2025
bec0f34
updates
aidan-curtis Jan 31, 2025
11f117b
Merge remote-tracking branch 'origin/lis-spot/update-fm-baselines-exp…
lf-zhao Jan 31, 2025
dc74e15
Merge branch 'lis-spot/update-fm-baselines-experiments' of https://gi…
lf-zhao Jan 31, 2025
55cc006
update perception vlm prompting
lf-zhao Jan 31, 2025
fe54333
updates
aidan-curtis Feb 1, 2025
74cace0
updates
aidan-curtis Feb 1, 2025
051cd0a
push new 2 cup 1 empty removal task
lf-zhao Feb 1, 2025
878c722
add new class for the task and also test file
lf-zhao Feb 1, 2025
1d94d27
update nsrts for it
lf-zhao Feb 1, 2025
ae5bbf0
update max token
lf-zhao Feb 1, 2025
b0fb753
updates
aidan-curtis Feb 1, 2025
edd4815
cup look env
aidan-curtis Feb 1, 2025
d179bf6
cup emptiness
aidan-curtis Feb 1, 2025
ffd2f50
deploy stuff
aidan-curtis Feb 1, 2025
f44e61d
update nsrts
lf-zhao Feb 6, 2025
0624708
update some goals
lf-zhao Feb 6, 2025
603a08d
update metadata
lf-zhao Feb 6, 2025
a215ccd
cleanup
lf-zhao Feb 6, 2025
c448c70
[in progress] update detection; fixing molmo sam detection
lf-zhao Feb 6, 2025
56b33de
update spot env: VLM only evaluate active VLM predicates; update VLM …
lf-zhao Feb 6, 2025
724a680
Merge commit 'ffd2f50ea420f88ca372194df8cde1194c3596ab' into lis-spot…
lf-zhao Feb 27, 2025
451de1d
Merge remote-tracking branch 'refs/remotes/origin/master' into lis-sp…
lf-zhao Feb 27, 2025
5a2f6ff
temp fix for merging
lf-zhao Feb 27, 2025
6dbc205
temp fix for merging - detection in minimal perceiver
lf-zhao Feb 27, 2025
3587310
Merge remote-tracking branch 'origin/master' into lis-spot/merging-li…
lf-zhao Feb 27, 2025
5d36320
create a subclass AugmentedState and initantiate it - keep original S…
lf-zhao Feb 28, 2025
ab5a259
some fixes to tests on transition graph gen
lf-zhao Mar 6, 2025
028f9c0
update using augmented state
lf-zhao Mar 17, 2025
8da7904
update getting VLM predicates more elegantly
lf-zhao Mar 17, 2025
22188f6
clean up the goal reaching check
lf-zhao Mar 17, 2025
aec26c0
update
lf-zhao Mar 17, 2025
aeace48
update token limit
lf-zhao Mar 17, 2025
23e2311
add add structure doc
lf-zhao Mar 17, 2025
091abdb
add doc for creating robot task with canonical states
lf-zhao Mar 17, 2025
e3ba695
add an overview guide for setup and running our BKLVA approach
lf-zhao Mar 17, 2025
04906ef
merge VLM predicates from Nishanth's tasks; hack to use VLM predicate…
lf-zhao Mar 23, 2025
dc3f950
minor fixes for VLM predicates
lf-zhao Mar 23, 2025
c62aa45
clean
lf-zhao Mar 23, 2025
4b2fcfc
fix: using AugmentedState in minimal perceiver
lf-zhao Mar 23, 2025
e7c3ede
update state creation helper
lf-zhao Mar 23, 2025
bbed876
update BKLVA approach doc for baselines and run commands
lf-zhao Mar 23, 2025
10ccf43
add back missing files for synthetic env data
lf-zhao Mar 24, 2025
09d8d1e
major update to mock/synthetic environment experiment running script
lf-zhao Mar 25, 2025
ae331b7
minor cleaning of the mock env file
lf-zhao Mar 25, 2025
f60ac2f
update experiment run script
lf-zhao Mar 25, 2025
d8abbf3
major, new script for running experiments locally in bash
lf-zhao Mar 25, 2025
77f944c
major, collecting results for running locally in bash/python
lf-zhao Mar 25, 2025
bcb0368
add readme and another viz script
lf-zhao Mar 25, 2025
8dd22ef
script for kill local runs
lf-zhao Mar 25, 2025
e4cf2c9
update
lf-zhao Mar 25, 2025
24f1f0f
update
lf-zhao Mar 25, 2025
247f298
update for saving a name for result collection
lf-zhao Mar 25, 2025
958b80c
fix cup emptiness one
lf-zhao Mar 25, 2025
2a9b1d9
update experiment
lf-zhao Mar 25, 2025
f8be4a2
fix OR goals in belief case
lf-zhao Mar 25, 2025
d0c9a23
similar fix for VLM planner - handling OR goals properly
lf-zhao Mar 25, 2025
938dc30
add new task: retrieving two green cups (one larger and one smaller) …
lf-zhao Mar 25, 2025
7717795
add new 2-cup picking task env; minor fixes
lf-zhao Mar 25, 2025
ab341a8
add GT factory
lf-zhao Mar 25, 2025
1e1c473
fixes for OR goal atoms; other update
lf-zhao Mar 25, 2025
55d6cf7
remove
lf-zhao Mar 25, 2025
0dca556
add new upgraded scripts for ssh (not fully tested)
lf-zhao Mar 30, 2025
4b3d1ee
add script for running local experiments
lf-zhao Mar 30, 2025
80a7995
add scripts for running ssh remote experiments
lf-zhao Mar 30, 2025
7e19337
update commands for BKLVA approach
lf-zhao Mar 30, 2025
a79fc09
minor update
lf-zhao Apr 10, 2025
b5c0f10
add a ray version of sam2
lf-zhao Apr 16, 2025
ad43d77
add server side for gemini pointing + sam2
lf-zhao Apr 16, 2025
9c2db03
add utils for gemini pointing
lf-zhao Apr 16, 2025
a40f8ce
add client side for gemini pointing + sam2
lf-zhao Apr 16, 2025

Sorry, this diff is taking too long to generate.

It may be too large to display on GitHub.