← 返回 2026-08-02 会议
迭代过程 (v1-15)方法测试(七臂比试)正式 GT

正式 GT 标注 · 设计稿 · 2026-08-02 · 未实现,待拍板项已标注

判定流程

Step 0 · 阶段判断(contact + skill 序列):物体是否在手 = 该物体处于 pick up 之后、place/release 之前的 skill 区间(注意 manipulating_object_id 是"被操作物"不是"在手"——turn on switch 的 manipulating 是开关本身,不能直接当在手判据)。

分支 A · 不在手(未拿起 / 已放下):沿用七臂比试后定版的 contact part + 高斯那一套——contact part(sim link GT)、destination 接触点弥散,全套照旧。

分支 B · 在手:多一步 VLM gate——给 1 张原 RGB + 每个涉及物体各 1 张 crop,逐物体问"本子目标的 functional / visual focus 是否不仅限于手的接触点"。否 → 回分支 A 规则;是 → 对该物体走 DINO 选 part 线,再做平滑弥散渲染(见下)。多物体 verb(2-3 个 mask)一次 call 全问,分别出结果。

数据集普查(100 task × 1 demo 抽样,skill 全量聚合)

高亮行 = 在手相位存在且初判需要 VLM gate 的 skill。"焦点位置"列为草案,待审。

idskill次数在手相位初判焦点位置(草案)数据集真实例子备注
1move to659无直通—fridge / bottom cabinetnavigation, 无操作物
2pick up from412后段才在手直通contact partcan of soda + floors接近+抓取段=现行 contact 规则
10open door54否直通contact part (handle)fridge / top cabinet现行规则 (sim link GT)
12close door40否直通contact part (handle)fridge / bottom cabinet
9open drawer5否直通contact partbottom cabinet
11close drawer5否直通contact partbottom cabinet
69turn on switch15否*直通contact part (button)washer / microwave / burner*manipulating_object=开关本身≠在手
70turn off switch7否*直通contact part (button)table lamp / lighter
67press1否*直通contact partradio
4place in212是VLM gate参考物: 容器口toy + toy box / hardback + storage box示例锚点: 放入→容器口
3place on141是VLM gate参考物: 承载面落点sprouts + baking sheet / tupperware + countertop
91place on next to38是VLM gate参考物: 落点+邻居旁侧gift box + floors + christmas tree3 物: 主物+面+邻居
92place in next to3是VLM gate参考物: 容器口+邻居旁侧food processor + top cabinet + toaster
98place under2是VLM gate参考物: 下方空隙bottle of detergent + sink
6insert6是VLM gate双侧: 孔/缝 + 在手物插入端pen + pencil case / hardback + bookcase在手物 focus≠握持点 (笔尖/书脊)
28pour11是VLM gate双侧: 倾倒沿 + 目标口/落区taco + plate + tupperware
34chop8是VLM gate双侧: 刀刃 + 切割线carving knife + brussels sprouts工具功能端
50sweep surface26是VLM gate双侧: 刷毛端 + 待扫区broom + floors / vacuum + floors
46wipe hard9是VLM gate双侧: 刷面 + 污渍区scrub brush + trumpet
61hang1是VLM gate双侧: 挂孔/顶边 + 钉poster + wall nail
19attach3是VLM gate双侧: 两接口配对digital camera + tripod / fire alarm + wall nail
88ignite1是VLM gate双侧: 火焰口 + 引燃点lighter + firewood
5hand over19是VLM gate双侧: 接收手 + 可递交端hinged jar + right/left hand
90push to60接触中VLM gate参考物: 目的地/对齐缝hardback + bookcase / swivel chair + desk非严格在手但同构
13open lid10是VLM gate参考物: 铰链/口沿toolbox / hinged jar
14close lid12是VLM gate参考物: 口沿合缝car / toolbox / hinged jar
93turn to7是待拍板待拍板: 朝向面?french press + robotfocus 语义弱
99tip over4接触中待拍板待拍板: 支点/倒向区?box of oatmeal / paper bag
94hold3是直通无 beyond-contactbroom / camera纯持握
8release3是直通无 beyond-contactbroom / camera

VLM gate prompt 草案(few-shot 例子 = brainstorm,待审)

The robot is executing the subgoal: "<subgoal>".
Held object: "<A>". Other involved objects: "<B>", "<C>"...
Image 1 = full scene. Images 2..k = one close-up crop per involved object (in the order listed).

During THIS subgoal, is the functional / visual-focus region limited to the current hand-contact
points, or does it extend beyond them? Examples of beyond-contact focus:
- place a can into a trash can -> focus = the trash can OPENING (not the grip on the can)
- insert a pen into a pencil case -> the case SLOT, plus the pen TIP (held object's functional end)
- pour from a cup onto a plate -> the cup POURING RIM, plus the plate center
- chop vegetables with a knife -> the knife BLADE EDGE, plus the cut line on the vegetable
- sweep the floor with a broom -> the broom BRISTLES, plus the dirt area
- hang a poster on a nail -> the poster hanging hole / top edge, plus the NAIL
- hand an object to someone -> the receiver's HAND, plus the object's free graspable end
Counter-examples (contact-only, answer none): merely holding or carrying an object; releasing it.

For EACH object, output whether it has a beyond-contact focus region in this subgoal and,
if yes, name that region as a short part phrase.
Output JSON: {"reason": "<one sentence>",
 "objects": [{"name": "<A>", "beyond_contact": true/false, "part_phrase": "..."}, ...]}

在手物 DINO part 的弥散渲染(草案,非高斯但类高斯观感)

1. 去锯齿:DINO 选中簇的 patch mask 上采样后,与原 GT mask 求交,对边界做 feather(距离变换衰减,宽度约一个 patch),patch 间锯齿一并平滑;
2. 峰值:保证聚类中心最亮——被选簇特征 medoid 对应的像素位置记为峰 1.0;
3. 弥散:簇内像素按到峰的距离(空间距离或特征距离,二选一待定)衰减到边缘 ≈0.3,再乘 feather 后的 mask → 形状贴 part、观感类高斯;
4. fallback:gate 判否或 DINO 选簇失败 → 回 contact + 高斯。

← 返回 方法测试