正式 GT 标注 · 设计稿 · 2026-08-02 · 未实现,待拍板项已标注
判定流程
Step 0 · 阶段判断(contact + skill 序列):物体是否在手 = 该物体处于 pick up 之后、place/release 之前的 skill 区间(注意 manipulating_object_id 是"被操作物"不是"在手"——turn on switch 的 manipulating 是开关本身,不能直接当在手判据)。
分支 A · 不在手(未拿起 / 已放下):沿用七臂比试后定版的 contact part + 高斯那一套——contact part(sim link GT)、destination 接触点弥散,全套照旧。
分支 B · 在手:多一步 VLM gate——给 1 张原 RGB + 每个涉及物体各 1 张 crop,逐物体问"本子目标的 functional / visual focus 是否不仅限于手的接触点"。否 → 回分支 A 规则;是 → 对该物体走 DINO 选 part 线,再做平滑弥散渲染(见下)。多物体 verb(2-3 个 mask)一次 call 全问,分别出结果。
数据集普查(100 task × 1 demo 抽样,skill 全量聚合)
高亮行 = 在手相位存在且初判需要 VLM gate 的 skill。"焦点位置"列为草案,待审。
| id | skill | 次数 | 在手相位 | 初判 | 焦点位置(草案) | 数据集真实例子 | 备注 |
|---|---|---|---|---|---|---|---|
| 1 | move to | 659 | 无 | 直通 | — | fridge / bottom cabinet | navigation, 无操作物 |
| 2 | pick up from | 412 | 后段才在手 | 直通 | contact part | can of soda + floors | 接近+抓取段=现行 contact 规则 |
| 10 | open door | 54 | 否 | 直通 | contact part (handle) | fridge / top cabinet | 现行规则 (sim link GT) |
| 12 | close door | 40 | 否 | 直通 | contact part (handle) | fridge / bottom cabinet | |
| 9 | open drawer | 5 | 否 | 直通 | contact part | bottom cabinet | |
| 11 | close drawer | 5 | 否 | 直通 | contact part | bottom cabinet | |
| 69 | turn on switch | 15 | 否* | 直通 | contact part (button) | washer / microwave / burner | *manipulating_object=开关本身≠在手 |
| 70 | turn off switch | 7 | 否* | 直通 | contact part (button) | table lamp / lighter | |
| 67 | press | 1 | 否* | 直通 | contact part | radio | |
| 4 | place in | 212 | 是 | VLM gate | 参考物: 容器口 | toy + toy box / hardback + storage box | 示例锚点: 放入→容器口 |
| 3 | place on | 141 | 是 | VLM gate | 参考物: 承载面落点 | sprouts + baking sheet / tupperware + countertop | |
| 91 | place on next to | 38 | 是 | VLM gate | 参考物: 落点+邻居旁侧 | gift box + floors + christmas tree | 3 物: 主物+面+邻居 |
| 92 | place in next to | 3 | 是 | VLM gate | 参考物: 容器口+邻居旁侧 | food processor + top cabinet + toaster | |
| 98 | place under | 2 | 是 | VLM gate | 参考物: 下方空隙 | bottle of detergent + sink | |
| 6 | insert | 6 | 是 | VLM gate | 双侧: 孔/缝 + 在手物插入端 | pen + pencil case / hardback + bookcase | 在手物 focus≠握持点 (笔尖/书脊) |
| 28 | pour | 11 | 是 | VLM gate | 双侧: 倾倒沿 + 目标口/落区 | taco + plate + tupperware | |
| 34 | chop | 8 | 是 | VLM gate | 双侧: 刀刃 + 切割线 | carving knife + brussels sprouts | 工具功能端 |
| 50 | sweep surface | 26 | 是 | VLM gate | 双侧: 刷毛端 + 待扫区 | broom + floors / vacuum + floors | |
| 46 | wipe hard | 9 | 是 | VLM gate | 双侧: 刷面 + 污渍区 | scrub brush + trumpet | |
| 61 | hang | 1 | 是 | VLM gate | 双侧: 挂孔/顶边 + 钉 | poster + wall nail | |
| 19 | attach | 3 | 是 | VLM gate | 双侧: 两接口配对 | digital camera + tripod / fire alarm + wall nail | |
| 88 | ignite | 1 | 是 | VLM gate | 双侧: 火焰口 + 引燃点 | lighter + firewood | |
| 5 | hand over | 19 | 是 | VLM gate | 双侧: 接收手 + 可递交端 | hinged jar + right/left hand | |
| 90 | push to | 60 | 接触中 | VLM gate | 参考物: 目的地/对齐缝 | hardback + bookcase / swivel chair + desk | 非严格在手但同构 |
| 13 | open lid | 10 | 是 | VLM gate | 参考物: 铰链/口沿 | toolbox / hinged jar | |
| 14 | close lid | 12 | 是 | VLM gate | 参考物: 口沿合缝 | car / toolbox / hinged jar | |
| 93 | turn to | 7 | 是 | 待拍板 | 待拍板: 朝向面? | french press + robot | focus 语义弱 |
| 99 | tip over | 4 | 接触中 | 待拍板 | 待拍板: 支点/倒向区? | box of oatmeal / paper bag | |
| 94 | hold | 3 | 是 | 直通 | 无 beyond-contact | broom / camera | 纯持握 |
| 8 | release | 3 | 是 | 直通 | 无 beyond-contact | broom / camera |
VLM gate prompt 草案(few-shot 例子 = brainstorm,待审)
The robot is executing the subgoal: "<subgoal>".
Held object: "<A>". Other involved objects: "<B>", "<C>"...
Image 1 = full scene. Images 2..k = one close-up crop per involved object (in the order listed).
During THIS subgoal, is the functional / visual-focus region limited to the current hand-contact
points, or does it extend beyond them? Examples of beyond-contact focus:
- place a can into a trash can -> focus = the trash can OPENING (not the grip on the can)
- insert a pen into a pencil case -> the case SLOT, plus the pen TIP (held object's functional end)
- pour from a cup onto a plate -> the cup POURING RIM, plus the plate center
- chop vegetables with a knife -> the knife BLADE EDGE, plus the cut line on the vegetable
- sweep the floor with a broom -> the broom BRISTLES, plus the dirt area
- hang a poster on a nail -> the poster hanging hole / top edge, plus the NAIL
- hand an object to someone -> the receiver's HAND, plus the object's free graspable end
Counter-examples (contact-only, answer none): merely holding or carrying an object; releasing it.
For EACH object, output whether it has a beyond-contact focus region in this subgoal and,
if yes, name that region as a short part phrase.
Output JSON: {"reason": "<one sentence>",
"objects": [{"name": "<A>", "beyond_contact": true/false, "part_phrase": "..."}, ...]}在手物 DINO part 的弥散渲染(草案,非高斯但类高斯观感)
1. 去锯齿:DINO 选中簇的 patch mask 上采样后,与原 GT mask 求交,对边界做 feather(距离变换衰减,宽度约一个 patch),patch 间锯齿一并平滑;
2. 峰值:保证聚类中心最亮——被选簇特征 medoid 对应的像素位置记为峰 1.0;
3. 弥散:簇内像素按到峰的距离(空间距离或特征距离,二选一待定)衰减到边缘 ≈0.3,再乘 feather 后的 mask → 形状贴 part、观感类高斯;
4. fallback:gate 判否或 DINO 选簇失败 → 回 contact + 高斯。