forked from huggingface/lerobot
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
1203 lines (1095 loc) · 37.4 KB
/
Copy pathindex.html
File metadata and controls
1203 lines (1095 loc) · 37.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>2D Vision Fetching</title>
<meta
name="description"
content="A project blog for 2D Vision Fetching: real-world xArm6 tabletop manipulation from dual RGB views with Action Chunking Transformers."
/>
<style>
:root {
--bg: #f7f4ef;
--paper: #fffaf2;
--ink: #191714;
--muted: #706b62;
--line: #ded5c8;
--accent: #bc5b2c;
--accent-dark: #763614;
--blue: #1f4d66;
--green: #587d4f;
--shadow: 0 20px 60px rgba(49, 38, 24, 0.12);
--radius: 28px;
font-family:
Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont,
"Segoe UI", sans-serif;
}
* {
box-sizing: border-box;
}
html {
scroll-behavior: smooth;
}
body {
margin: 0;
color: var(--ink);
background:
radial-gradient(circle at top left, rgba(188, 91, 44, 0.13), transparent 34rem),
radial-gradient(circle at 90% 10%, rgba(31, 77, 102, 0.12), transparent 28rem),
var(--bg);
line-height: 1.65;
}
a {
color: inherit;
}
.nav {
position: sticky;
top: 0;
z-index: 10;
display: flex;
justify-content: space-between;
align-items: center;
padding: 18px min(5vw, 64px);
border-bottom: 1px solid rgba(222, 213, 200, 0.7);
background: rgba(247, 244, 239, 0.86);
backdrop-filter: blur(18px);
}
.brand {
font-weight: 750;
letter-spacing: -0.03em;
}
.nav-links {
display: flex;
gap: 22px;
color: var(--muted);
font-size: 0.94rem;
}
.nav-links a {
text-decoration: none;
}
.nav-links a:hover {
color: var(--ink);
}
.hero,
.section {
width: min(1120px, calc(100% - 32px));
margin: 0 auto;
}
.hero {
padding: 92px 0 54px;
}
.eyebrow {
margin: 0 0 20px;
color: var(--accent-dark);
font-size: 0.82rem;
font-weight: 800;
letter-spacing: 0.13em;
text-transform: uppercase;
}
h1,
h2,
h3 {
margin: 0;
line-height: 1.04;
letter-spacing: -0.055em;
}
h1 {
max-width: 980px;
font-size: clamp(2.2rem, 4.8vw, 3.4rem);
font-weight: 780;
}
h2 {
max-width: 860px;
font-size: clamp(1.65rem, 2.8vw, 2.35rem);
font-weight: 760;
}
h3 {
font-size: 1.4rem;
font-weight: 760;
letter-spacing: -0.035em;
}
.subtitle {
max-width: 820px;
margin: 28px 0 0;
color: #39342e;
font-size: clamp(1.15rem, 2vw, 1.55rem);
line-height: 1.5;
}
.meta {
display: flex;
flex-wrap: wrap;
gap: 12px;
margin-top: 34px;
color: var(--muted);
}
.pill {
display: inline-flex;
align-items: center;
min-height: 34px;
padding: 5px 12px;
border: 1px solid var(--line);
border-radius: 999px;
background: rgba(255, 250, 242, 0.55);
font-size: 0.92rem;
}
.hero-card {
margin-top: 46px;
overflow: hidden;
border: 1px solid var(--line);
border-radius: var(--radius);
background: #111;
box-shadow: var(--shadow);
}
video {
display: block;
width: 100%;
background: #111;
}
.caption {
margin: 0;
padding: 16px 20px 18px;
color: #dfd7cb;
background: #15120f;
font-size: 0.96rem;
}
.section {
padding: 80px 0;
border-top: 1px solid rgba(222, 213, 200, 0.85);
}
.section-intro {
display: grid;
grid-template-columns: minmax(0, 0.85fr) minmax(280px, 1fr);
gap: clamp(28px, 6vw, 80px);
align-items: start;
}
.section-intro p,
.prose p {
margin: 0 0 20px;
color: #3d3832;
font-size: 1.08rem;
}
.lede {
color: var(--muted);
font-size: 1rem;
text-transform: uppercase;
letter-spacing: 0.11em;
font-weight: 800;
}
.stats {
display: grid;
grid-template-columns: repeat(5, 1fr);
gap: 14px;
margin-top: 44px;
}
.stat,
.card {
border: 1px solid var(--line);
border-radius: 22px;
background: rgba(255, 250, 242, 0.68);
box-shadow: 0 8px 28px rgba(49, 38, 24, 0.06);
}
.stat {
padding: 22px;
}
.stat strong {
display: block;
font-size: clamp(1.8rem, 3vw, 2.8rem);
line-height: 1;
letter-spacing: -0.06em;
}
.stat span {
display: block;
margin-top: 9px;
color: var(--muted);
font-size: 0.93rem;
}
.cards {
display: grid;
grid-template-columns: repeat(3, 1fr);
gap: 18px;
margin-top: 36px;
}
.card {
padding: 26px;
}
.card p {
margin: 14px 0 0;
color: var(--muted);
}
.number {
display: inline-grid;
place-items: center;
width: 34px;
height: 34px;
margin-bottom: 18px;
border-radius: 50%;
color: #fff;
background: var(--ink);
font-size: 0.88rem;
font-weight: 800;
}
.pipeline {
margin-top: 42px;
padding: 30px;
border: 1px solid var(--line);
border-radius: var(--radius);
background: rgba(255, 250, 242, 0.65);
}
.pipeline-grid {
display: grid;
grid-template-columns: repeat(5, 1fr);
gap: 12px;
align-items: stretch;
}
.node {
position: relative;
min-height: 120px;
padding: 18px;
border-radius: 20px;
color: #fff;
background: var(--blue);
}
.node:nth-child(2),
.node:nth-child(4) {
background: var(--accent);
}
.node:nth-child(3),
.node:nth-child(5) {
background: var(--green);
}
.node span {
display: block;
margin-bottom: 8px;
opacity: 0.75;
font-size: 0.78rem;
font-weight: 800;
letter-spacing: 0.1em;
text-transform: uppercase;
}
.node strong {
display: block;
font-size: 1.1rem;
line-height: 1.25;
}
.media-grid {
display: grid;
grid-template-columns: repeat(2, 1fr);
gap: 18px;
margin-top: 40px;
}
.media-card {
overflow: hidden;
border: 1px solid var(--line);
border-radius: var(--radius);
background: #14110f;
box-shadow: var(--shadow);
}
.media-card video {
aspect-ratio: 16 / 9;
object-fit: cover;
}
.model-grid {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 18px;
margin-top: 38px;
}
.formula {
padding: 24px;
border-radius: 22px;
color: #f7efe4;
background: #191714;
font-family: "SFMono-Regular", Consolas, "Liberation Mono", monospace;
font-size: 0.96rem;
line-height: 1.7;
overflow: auto;
}
.quote {
margin: 34px 0 0;
padding: 30px;
border-left: 6px solid var(--accent);
border-radius: 0 24px 24px 0;
background: rgba(255, 250, 242, 0.7);
color: #332e29;
font-size: clamp(1.18rem, 2vw, 1.55rem);
line-height: 1.5;
}
.comparison {
display: grid;
grid-template-columns: repeat(4, 1fr);
gap: 14px;
margin-top: 36px;
}
.metric {
padding: 20px;
border: 1px solid var(--line);
border-radius: 20px;
background: rgba(255, 250, 242, 0.72);
}
.metric strong {
display: block;
font-size: 1.05rem;
}
.metric span {
display: block;
margin-top: 7px;
color: var(--muted);
font-size: 0.93rem;
}
.flow-section {
margin-top: 42px;
}
.flow-title {
display: flex;
justify-content: space-between;
align-items: baseline;
gap: 18px;
margin-bottom: 18px;
}
.flow-title p {
margin: 0;
color: var(--muted);
font-size: 0.95rem;
}
.flow-rail {
display: grid;
grid-template-columns: repeat(5, 1fr);
gap: 12px;
}
.flow-step {
position: relative;
min-height: 158px;
padding: 22px;
border: 1px solid var(--line);
border-radius: 22px;
background: rgba(255, 250, 242, 0.75);
box-shadow: 0 8px 28px rgba(49, 38, 24, 0.06);
}
.flow-step:not(:last-child)::after {
content: "→";
position: absolute;
top: 50%;
right: -14px;
z-index: 2;
display: grid;
place-items: center;
width: 26px;
height: 26px;
border-radius: 50%;
color: #fff;
background: var(--accent);
transform: translateY(-50%);
font-weight: 800;
}
.flow-step span {
display: block;
margin-bottom: 10px;
color: var(--accent-dark);
font-size: 0.78rem;
font-weight: 850;
letter-spacing: 0.1em;
text-transform: uppercase;
}
.flow-step strong {
display: block;
font-size: 1.14rem;
line-height: 1.22;
letter-spacing: -0.035em;
}
.flow-step p {
margin: 12px 0 0;
color: var(--muted);
font-size: 0.94rem;
line-height: 1.45;
}
.detail-grid {
display: grid;
grid-template-columns: repeat(2, 1fr);
gap: 18px;
margin-top: 28px;
}
.detail-card {
padding: 26px;
border: 1px solid var(--line);
border-radius: 24px;
background: rgba(255, 250, 242, 0.7);
box-shadow: 0 8px 28px rgba(49, 38, 24, 0.06);
}
.detail-card h3 {
margin-bottom: 14px;
}
.detail-list {
display: grid;
gap: 12px;
margin: 0;
padding: 0;
list-style: none;
}
.detail-list li {
padding-left: 18px;
border-left: 3px solid rgba(188, 91, 44, 0.42);
color: #4a433b;
}
.layer-map {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 14px;
margin-top: 32px;
}
.layer {
display: grid;
grid-template-columns: 148px 1fr;
gap: 18px;
align-items: start;
padding: 20px;
border: 1px solid var(--line);
border-radius: 22px;
background: rgba(255, 250, 242, 0.66);
}
.layer-label {
color: var(--accent-dark);
font-weight: 850;
letter-spacing: -0.025em;
}
.layer p {
margin: 0;
color: var(--muted);
}
.footer {
width: min(1120px, calc(100% - 32px));
margin: 0 auto;
padding: 60px 0 80px;
color: var(--muted);
border-top: 1px solid rgba(222, 213, 200, 0.85);
}
.footer a {
color: var(--ink);
}
@media (max-width: 920px) {
.nav {
position: static;
}
.nav-links {
display: none;
}
.section-intro,
.model-grid {
grid-template-columns: 1fr;
}
.stats,
.pipeline-grid,
.comparison {
grid-template-columns: repeat(2, 1fr);
}
.flow-rail {
grid-template-columns: 1fr;
}
.cards,
.media-grid,
.detail-grid,
.layer-map {
grid-template-columns: 1fr;
}
.flow-step:not(:last-child)::after {
top: auto;
right: 50%;
bottom: -19px;
transform: translateX(50%) rotate(90deg);
}
.layer {
grid-template-columns: 1fr;
}
}
@media (max-width: 560px) {
.hero {
padding-top: 56px;
}
.stats,
.pipeline-grid,
.flow-rail,
.comparison {
grid-template-columns: 1fr;
}
.section {
padding: 56px 0;
}
}
</style>
</head>
<body>
<header class="nav">
<div class="brand">2D Vision Fetching</div>
<nav class="nav-links" aria-label="Main navigation">
<a href="#problem">Problem</a>
<a href="#system">System</a>
<a href="#code-pipeline">Code Pipeline</a>
<a href="#data">Data</a>
<a href="#model">Model</a>
<a href="#results">Results</a>
</nav>
</header>
<main>
<section class="hero">
<p class="eyebrow">ShanghaiTech SIST Robotics Course Project</p>
<h1>Teaching an xArm6 to fetch from two RGB views</h1>
<p class="subtitle">
We completed a real-robot imitation learning project for a tabletop
task: a Gello leader arm was used to teleoperate an xArm6 follower, two
RealSense RGB cameras recorded the scene, and an Action Chunking
Transformer was trained and deployed to pick up a toy kettle and place
it onto a toy stove.
</p>
<div class="meta">
<span class="pill">Final project demo · June 21, 2026</span>
<span class="pill">Jinxi Xiao, Xuebing Li, Weile Li </span>
<span class="pill">xArm6 + Gello + Dual RGB</span>
<span class="pill">Completed ACT inference on hardware</span>
</div>
<div class="hero-card">
<video controls playsinline preload="metadata" poster="">
<source src="robot_infer.mp4" type="video/mp4" />
</video>
<p class="caption">
A trained ACT policy runs on the xArm6 and executes the kettle-on-stove
task from visual observations and robot proprioception.
</p>
</div>
</section>
<section class="section" id="problem">
<div class="section-intro">
<div>
<p class="lede">Why this is hard</p>
<h2>A small tabletop task still contains the hard parts of robot manipulation.</h2>
</div>
<div class="prose">
<p>
Pick-and-place is easy to describe but brittle on real hardware. The
robot must localize the kettle, align the gripper, lift without
slipping, transport smoothly, and release at the right place. A
small error during grasping can become a large error during
placement.
</p>
<p>
We deliberately use 2D RGB observations rather than depth or a full
3D reconstruction. This makes the setup cheaper and easier to
deploy, while forcing the policy to learn useful visual and temporal
cues from partial views, camera noise, and gripper occlusion.
</p>
</div>
</div>
<div class="stats" aria-label="Project snapshot">
<div class="stat">
<strong>40</strong>
<span>expert demonstration episodes collected for the final dataset</span>
</div>
<div class="stat">
<strong>2</strong>
<span>Intel RealSense RGB cameras observing arm-side and front views</span>
</div>
<div class="stat">
<strong>30Hz</strong>
<span>target recording frequency for synchronized robot-camera data</span>
</div>
<div class="stat">
<strong>100</strong>
<span>actions predicted in each ACT action chunk during policy rollout</span>
</div>
<div class="stat">
<strong>20k</strong>
<span>training steps used for the ACT checkpoint schedule</span>
</div>
</div>
</section>
<section class="section" id="system">
<div class="section-intro">
<div>
<p class="lede">How it works</p>
<h2>One pipeline records human skill. The other turns it into a policy.</h2>
</div>
<div class="prose">
<p>
The data-collection loop reads the xArm state, grabs the latest RGB
frames from both cameras, reads the Gello leader joint positions,
sends the resulting command to the xArm follower, and writes the
synchronized observation-action frame into the LeRobot dataset.
</p>
<p>
Offline training loaded the recorded dataset as action chunks. The
ACT policy received the two-view image observation and robot
proprioception, then learned to predict a short horizon of
joint-space actions for physical rollout.
</p>
</div>
</div>
<div class="pipeline" aria-label="System pipeline">
<div class="pipeline-grid">
<div class="node">
<span>Human</span>
<strong>Gello leader arm produces teleoperation commands</strong>
</div>
<div class="node">
<span>Robot</span>
<strong>xArm6 follower executes joint and gripper targets</strong>
</div>
<div class="node">
<span>Vision</span>
<strong>Arm-side and front RGB cameras capture the workspace</strong>
</div>
<div class="node">
<span>Dataset</span>
<strong>LeRobot stores state, images, action, task, and time</strong>
</div>
<div class="node">
<span>Policy</span>
<strong>ACT executes chunked actions learned from demonstrations</strong>
</div>
</div>
</div>
</section>
<section class="section" id="code-pipeline">
<div class="section-intro">
<div>
<p class="lede">Code project pipeline</p>
<h2>The codebase turns teleoperation into an autonomous robot policy.</h2>
</div>
<div class="prose">
<p>
The implementation is built as a project-specific fork of LeRobot.
It adds the Gello leader interface, the xArm6 follower interface,
RealSense camera support, dataset recording, ACT training, and
policy rollout into one reproducible robot-learning stack.
</p>
<p>
The important design choice is separation of responsibilities:
hardware devices expose small interfaces, the recording loop only
synchronizes observation and action streams, the dataset stores a
fixed imitation-learning problem, and the policy layer consumes that
dataset without touching hardware-specific details.
</p>
</div>
</div>
<div class="flow-section">
<div class="flow-title">
<h3>Whole Project Flow</h3>
<p>From human demonstrations to real-robot inference</p>
</div>
<div class="flow-rail" aria-label="Whole code project pipeline">
<div class="flow-step">
<span>Step 1</span>
<strong>Teleoperate</strong>
<p>A human moves the Gello leader arm; seven Dynamixel readings represent six arm joints and the gripper.</p>
</div>
<div class="flow-step">
<span>Step 2</span>
<strong>Record</strong>
<p>The xArm6 executes the mapped command while robot state and two RGB camera streams are sampled at 30 Hz.</p>
</div>
<div class="flow-step">
<span>Step 3</span>
<strong>Build Dataset</strong>
<p>Each frame stores proprioception, cam-arm image, cam-front image, action, timestamp, and task label.</p>
</div>
<div class="flow-step">
<span>Step 4</span>
<strong>Train ACT</strong>
<p>The policy learns from 40 episodes and 14,320 frames, predicting 100-step action chunks.</p>
</div>
<div class="flow-step">
<span>Step 5</span>
<strong>Roll Out</strong>
<p>The trained checkpoint runs on the physical xArm6 and autonomously performs kettle placement.</p>
</div>
</div>
</div>
<div class="flow-section">
<div class="flow-title">
<h3>Data Collection Flow</h3>
<p>Human control, robot execution, visual sensing, and dataset writing happen in one loop.</p>
</div>
<div class="flow-rail" aria-label="Data collection pipeline">
<div class="flow-step">
<span>Input</span>
<strong>Gello Leader</strong>
<p>Reads normalized joint positions from the passive leader arm through the Dynamixel bus.</p>
</div>
<div class="flow-step">
<span>Control</span>
<strong>xArm6 Follower</strong>
<p>Receives joint and gripper targets through the xArm SDK and executes them without waiting for full trajectory completion.</p>
</div>
<div class="flow-step">
<span>Sensing</span>
<strong>Dual RealSense</strong>
<p>Captures arm-side and front RGB views by stable serial-number identification, avoiding camera-index ambiguity.</p>
</div>
<div class="flow-step">
<span>Sync</span>
<strong>30 Hz Frame</strong>
<p>Combines calibrated robot state, latest images, action, and task string into one aligned demonstration frame.</p>
</div>
<div class="flow-step">
<span>Storage</span>
<strong>LeRobot Dataset</strong>
<p>Stores numeric fields as structured episode data and encodes camera streams as videos for efficient replay.</p>
</div>
</div>
</div>
<div class="flow-section">
<div class="flow-title">
<h3>Training Flow</h3>
<p>The recorded dataset becomes supervised action-chunk targets.</p>
</div>
<div class="flow-rail" aria-label="ACT training pipeline">
<div class="flow-step">
<span>Dataset</span>
<strong>Episode-Aware Loading</strong>
<p>Samples stay inside episode boundaries so action chunks do not cross resets or task endings.</p>
</div>
<div class="flow-step">
<span>Observation</span>
<strong>Image + State Tokens</strong>
<p>Two RGB views are resized for the visual backbone and paired with the seven-dimensional robot state.</p>
</div>
<div class="flow-step">
<span>Encoder</span>
<strong>ResNet + Transformer</strong>
<p>Visual features, proprioception, and latent information are fused into a shared representation.</p>
</div>
<div class="flow-step">
<span>Supervision</span>
<strong>Chunk Loss</strong>
<p>The model reconstructs 100-step action sequences with an L1 term and VAE regularization.</p>
</div>
<div class="flow-step">
<span>Output</span>
<strong>Deployable Checkpoint</strong>
<p>Saved policy weights, configuration, processor settings, and normalization statistics are used for rollout.</p>
</div>
</div>
</div>
<div class="flow-section">
<div class="flow-title">
<h3>Inference Flow</h3>
<p>The deployment loop mirrors recording, but actions come from the trained policy.</p>
</div>
<div class="flow-rail" aria-label="Policy rollout pipeline">
<div class="flow-step">
<span>Observe</span>
<strong>Online Robot State</strong>
<p>The xArm wrapper gathers calibrated joint state, gripper state, cam-arm RGB, and cam-front RGB.</p>
</div>
<div class="flow-step">
<span>Prepare</span>
<strong>Normalize Inputs</strong>
<p>Preprocessors convert live observations into the same feature space used during training.</p>
</div>
<div class="flow-step">
<span>Predict</span>
<strong>ACT Action Chunk</strong>
<p>The policy predicts a queue of future robot actions instead of making an isolated one-step decision.</p>
</div>
<div class="flow-step">
<span>Schedule</span>
<strong>Chunk Execution</strong>
<p>An action interpolator and queue select the next command while preserving the target control rate.</p>
</div>
<div class="flow-step">
<span>Act</span>
<strong>xArm6 Motion</strong>
<p>The robot executes joint and gripper targets, producing the autonomous pick-and-place demonstration.</p>
</div>
</div>
</div>
<div class="layer-map" aria-label="Software architecture layers">
<div class="layer">
<div class="layer-label">Hardware Layer</div>
<p>Gello leader, xArm6 follower, gripper, and two RealSense cameras expose device-specific connect, observe, and act behavior.</p>
</div>
<div class="layer">
<div class="layer-label">Calibration Layer</div>
<p>Gello raw encoder readings are normalized into degrees, while xArm calibration maps between normalized commands and physical joint or gripper units.</p>
</div>
<div class="layer">
<div class="layer-label">Processor Layer</div>
<p>Observation and action processors keep the interface modular; in this project the mapping is mostly identity after calibration.</p>
</div>
<div class="layer">
<div class="layer-label">Dataset Layer</div>
<p>LeRobotDataset stores synchronized observation-action-task frames and preserves camera videos for later training, replay, and visualization.</p>
</div>
<div class="layer">
<div class="layer-label">Policy Layer</div>
<p>ACT uses a ResNet18 visual backbone, Transformer encoder-decoder, VAE latent variable, and 100-step action chunk output.</p>
</div>
<div class="layer">
<div class="layer-label">Rollout Layer</div>
<p>The deployment strategy repeatedly observes, preprocesses, predicts, schedules, and sends actions until the task is complete.</p>
</div>
</div>
<div class="detail-grid">
<div class="detail-card">
<h3>What the dataset contains</h3>
<ul class="detail-list">
<li>Seven-dimensional robot state: six xArm joints plus gripper position.</li>
<li>Two synchronized RGB observations: an arm-side camera and a front scene camera.</li>
<li>Seven-dimensional action targets produced by the Gello leader and executed by the xArm follower.</li>
<li>A natural-language task label, “put kettle on stove,” stored with every demonstration frame.</li>
<li>Episode-level organization that supports replay, visualization, training, and closed-loop debugging.</li>
</ul>
</div>
<div class="detail-card">
<h3>Why the implementation is robust enough for hardware</h3>
<ul class="detail-list">
<li>Camera devices are selected by RealSense serial number, so cam-arm and cam-front do not swap accidentally.</li>
<li>Background camera reading keeps the control loop close to 30 Hz instead of blocking on slow image capture.</li>
<li>xArm safety limits can cap relative target motion, reducing the risk of sudden large commands.</li>
<li>Streaming video encoding prevents episode saving from becoming the slowest part of data collection.</li>
<li>Rollout uses the same observation schema as recording, which reduces train-deploy mismatch.</li>
</ul>
</div>
<div class="detail-card">
<h3>ACT-specific details</h3>
<ul class="detail-list">
<li>The default model uses a ResNet18 image backbone with ImageNet initialization.</li>
<li>Multiple camera views are treated as separate visual streams before being fused by the Transformer.</li>
<li>The model predicts a 100-step action chunk, matching the temporal abstraction used in ACT.</li>
<li>The VAE encoder is used during training to model demonstration action sequences; inference uses the policy decoder.</li>
<li>History-frame support allows the model to receive the current frame plus recent observations when the history window is enabled.</li>
</ul>
</div>
<div class="detail-card">
<h3>Engineering lessons from the completed project</h3>
<ul class="detail-list">
<li>The most important integration point is consistent calibration between Gello degrees and xArm physical joint units.</li>
<li>Dataset quality depends on synchronized sensing, not only on whether the robot visibly completes the task.</li>
<li>Joint-space control is a practical choice for this setup because it maps cleanly to the xArm follower interface.</li>
<li>The code structure keeps data collection, training, and deployment separable, making failures easier to diagnose.</li>
<li>The final inference video demonstrates the whole pipeline rather than a single isolated model component.</li>
</ul>
</div>
</div>
</section>
<section class="section" id="data">
<div class="section-intro">
<div>
<p class="lede">Robot data</p>
<h2>We collected demonstrations directly on the physical setup.</h2>
</div>
<div class="prose">
<p>
The videos below show the human-in-the-loop data collection process.
The Gello leader provides an intuitive control interface, while the
follower robot, camera streams, and dataset writer run in one
synchronized loop.
</p>
<p>
This matters because imitation learning is only as good as the
demonstrations it receives. For a precise kettle placement task, the
dataset must capture not only the successful final pose but also the
approach, grasp, lift, transfer, release, and retreat phases.
</p>
</div>
</div>
<div class="media-grid">
<article class="media-card">
<video controls playsinline preload="metadata">
<source src="robot_data_collection_1.mp4" type="video/mp4" />
</video>
<p class="caption">
Data collection view 1: teleoperated demonstrations for the
kettle-on-stove task.
</p>