-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathparallel_tutorial.7
3940 lines (3940 loc) · 93.9 KB
/
parallel_tutorial.7
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
.\" Automatically generated by Pod::Man 4.11 (Pod::Simple 3.40)
.\"
.\" Standard preamble:
.\" ========================================================================
.de Sp \" Vertical space (when we can't use .PP)
.if t .sp .5v
.if n .sp
..
.de Vb \" Begin verbatim text
.ft CW
.nf
.ne \\$1
..
.de Ve \" End verbatim text
.ft R
.fi
..
.\" Set up some character translations and predefined strings. \*(-- will
.\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left
.\" double quote, and \*(R" will give a right double quote. \*(C+ will
.\" give a nicer C++. Capital omega is used to do unbreakable dashes and
.\" therefore won't be available. \*(C` and \*(C' expand to `' in nroff,
.\" nothing in troff, for use with C<>.
.tr \(*W-
.ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p'
.ie n \{\
. ds -- \(*W-
. ds PI pi
. if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch
. if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch
. ds L" ""
. ds R" ""
. ds C` ""
. ds C' ""
'br\}
.el\{\
. ds -- \|\(em\|
. ds PI \(*p
. ds L" ``
. ds R" ''
. ds C`
. ds C'
'br\}
.\"
.\" Escape single quotes in literal strings from groff's Unicode transform.
.ie \n(.g .ds Aq \(aq
.el .ds Aq '
.\"
.\" If the F register is >0, we'll generate index entries on stderr for
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index
.\" entries marked with X<> in POD. Of course, you'll have to process the
.\" output yourself in some meaningful fashion.
.\"
.\" Avoid warning from groff about undefined register 'F'.
.de IX
..
.nr rF 0
.if \n(.g .if rF .nr rF 1
.if (\n(rF:(\n(.g==0)) \{\
. if \nF \{\
. de IX
. tm Index:\\$1\t\\n%\t"\\$2"
..
. if !\nF==2 \{\
. nr % 0
. nr F 2
. \}
. \}
.\}
.rr rF
.\"
.\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2).
.\" Fear. Run. Save yourself. No user-serviceable parts.
. \" fudge factors for nroff and troff
.if n \{\
. ds #H 0
. ds #V .8m
. ds #F .3m
. ds #[ \f1
. ds #] \fP
.\}
.if t \{\
. ds #H ((1u-(\\\\n(.fu%2u))*.13m)
. ds #V .6m
. ds #F 0
. ds #[ \&
. ds #] \&
.\}
. \" simple accents for nroff and troff
.if n \{\
. ds ' \&
. ds ` \&
. ds ^ \&
. ds , \&
. ds ~ ~
. ds /
.\}
.if t \{\
. ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u"
. ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u'
. ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u'
. ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u'
. ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u'
. ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u'
.\}
. \" troff and (daisy-wheel) nroff accents
.ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V'
.ds 8 \h'\*(#H'\(*b\h'-\*(#H'
.ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#]
.ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H'
.ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u'
.ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#]
.ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#]
.ds ae a\h'-(\w'a'u*4/10)'e
.ds Ae A\h'-(\w'A'u*4/10)'E
. \" corrections for vroff
.if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u'
.if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u'
. \" for low resolution devices (crt and lpr)
.if \n(.H>23 .if \n(.V>19 \
\{\
. ds : e
. ds 8 ss
. ds o a
. ds d- d\h'-1'\(ga
. ds D- D\h'-1'\(hy
. ds th \o'bp'
. ds Th \o'LP'
. ds ae ae
. ds Ae AE
.\}
.rm #[ #] #H #V #F C
.\" ========================================================================
.\"
.IX Title "PARALLEL_TUTORIAL 7"
.TH PARALLEL_TUTORIAL 7 "2021-03-21" "20210222" "parallel"
.\" For nroff, turn off justification. Always turn off hyphenation; it makes
.\" way too many mistakes in technical documents.
.if n .ad l
.nh
.SH "GNU Parallel Tutorial"
.IX Header "GNU Parallel Tutorial"
This tutorial shows off much of \s-1GNU\s0 \fBparallel\fR's functionality. The
tutorial is meant to learn the options in and syntax of \s-1GNU\s0
\&\fBparallel\fR. The tutorial is \fBnot\fR to show realistic examples from the
real world.
.SS "Reader's guide"
.IX Subsection "Reader's guide"
If you prefer reading a book buy \fB\s-1GNU\s0 Parallel 2018\fR at
http://www.lulu.com/shop/ole\-tange/gnu\-parallel\-2018/paperback/product\-23558902.html
or download it at: https://doi.org/10.5281/zenodo.1146014
.PP
Otherwise start by watching the intro videos for a quick introduction:
http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
.PP
Then browse through the \fB\s-1EXAMPLE\s0\fRs after the list of \fB\s-1OPTIONS\s0\fR in
\&\fBman parallel\fR (Use \fBLESS=+/EXAMPLE: man parallel\fR). That will give
you an idea of what \s-1GNU\s0 \fBparallel\fR is capable of.
.PP
If you want to dive even deeper: spend a couple of hours walking
through the tutorial (\fBman parallel_tutorial\fR). Your command line
will love you for it.
.PP
Finally you may want to look at the rest of the manual (\fBman
parallel\fR) if you have special needs not already covered.
.PP
If you want to know the design decisions behind \s-1GNU\s0 \fBparallel\fR, try:
\&\fBman parallel_design\fR. This is also a good intro if you intend to
change \s-1GNU\s0 \fBparallel\fR.
.SH "Prerequisites"
.IX Header "Prerequisites"
To run this tutorial you must have the following:
.IP "parallel >= version 20160822" 9
.IX Item "parallel >= version 20160822"
Install the newest version using your package manager (recommended for
security reasons), the way described in \s-1README,\s0 or with this command:
.Sp
.Vb 10
\& $ (wget \-O \- pi.dk/3 || lynx \-source pi.dk/3 || curl pi.dk/3/ || \e
\& fetch \-o \- http://pi.dk/3 ) > install.sh
\& $ sha1sum install.sh
\& 12345678 3374ec53 bacb199b 245af2dd a86df6c9
\& $ md5sum install.sh
\& 029a9ac0 6e8b5bc6 052eac57 b2c3c9ca
\& $ sha512sum install.sh
\& 40f53af6 9e20dae5 713ba06c f517006d 9897747b ed8a4694 b1acba1b 1464beb4
\& 60055629 3f2356f3 3e9c4e3c 76e3f3af a9db4b32 bd33322b 975696fc e6b23cfb
\& $ bash install.sh
.Ve
.Sp
This will also install the newest version of the tutorial which you
can see by running this:
.Sp
.Vb 1
\& man parallel_tutorial
.Ve
.Sp
Most of the tutorial will work on older versions, too.
.IP "abc-file:" 9
.IX Item "abc-file:"
The file can be generated by this command:
.Sp
.Vb 1
\& parallel \-k echo ::: A B C > abc\-file
.Ve
.IP "def-file:" 9
.IX Item "def-file:"
The file can be generated by this command:
.Sp
.Vb 1
\& parallel \-k echo ::: D E F > def\-file
.Ve
.IP "abc0\-file:" 9
.IX Item "abc0-file:"
The file can be generated by this command:
.Sp
.Vb 1
\& perl \-e \*(Aqprintf "A\e0B\e0C\e0"\*(Aq > abc0\-file
.Ve
.IP "abc_\-file:" 9
.IX Item "abc_-file:"
The file can be generated by this command:
.Sp
.Vb 1
\& perl \-e \*(Aqprintf "A_B_C_"\*(Aq > abc_\-file
.Ve
.IP "tsv\-file.tsv" 9
.IX Item "tsv-file.tsv"
The file can be generated by this command:
.Sp
.Vb 1
\& perl \-e \*(Aqprintf "f1\etf2\enA\etB\enC\etD\en"\*(Aq > tsv\-file.tsv
.Ve
.IP "num8" 9
.IX Item "num8"
The file can be generated by this command:
.Sp
.Vb 1
\& perl \-e \*(Aqfor(1..8){print "$_\en"}\*(Aq > num8
.Ve
.IP "num128" 9
.IX Item "num128"
The file can be generated by this command:
.Sp
.Vb 1
\& perl \-e \*(Aqfor(1..128){print "$_\en"}\*(Aq > num128
.Ve
.IP "num30000" 9
.IX Item "num30000"
The file can be generated by this command:
.Sp
.Vb 1
\& perl \-e \*(Aqfor(1..30000){print "$_\en"}\*(Aq > num30000
.Ve
.IP "num1000000" 9
.IX Item "num1000000"
The file can be generated by this command:
.Sp
.Vb 1
\& perl \-e \*(Aqfor(1..1000000){print "$_\en"}\*(Aq > num1000000
.Ve
.IP "num_%header" 9
.IX Item "num_%header"
The file can be generated by this command:
.Sp
.Vb 2
\& (echo %head1; echo %head2; \e
\& perl \-e \*(Aqfor(1..10){print "$_\en"}\*(Aq) > num_%header
.Ve
.IP "fixedlen" 9
.IX Item "fixedlen"
The file can be generated by this command:
.Sp
.Vb 1
\& perl \-e \*(Aqprint "HHHHAAABBBCCC"\*(Aq > fixedlen
.Ve
.ie n .IP "For remote running: ssh login on 2 servers with no password in $SERVER1 and $SERVER2 must work." 9
.el .IP "For remote running: ssh login on 2 servers with no password in \f(CW$SERVER1\fR and \f(CW$SERVER2\fR must work." 9
.IX Item "For remote running: ssh login on 2 servers with no password in $SERVER1 and $SERVER2 must work."
.Vb 2
\& SERVER1=server.example.com
\& SERVER2=server2.example.net
.Ve
.Sp
So you must be able to do this without entering a password:
.Sp
.Vb 2
\& ssh $SERVER1 echo works
\& ssh $SERVER2 echo works
.Ve
.Sp
It can be setup by running 'ssh\-keygen \-t dsa; ssh-copy-id \f(CW$SERVER1\fR'
and using an empty passphrase, or you can use \fBssh-agent\fR.
.SH "Input sources"
.IX Header "Input sources"
\&\s-1GNU\s0 \fBparallel\fR reads input from input sources. These can be files, the
command line, and stdin (standard input or a pipe).
.SS "A single input source"
.IX Subsection "A single input source"
Input can be read from the command line:
.PP
.Vb 1
\& parallel echo ::: A B C
.Ve
.PP
Output (the order may be different because the jobs are run in
parallel):
.PP
.Vb 3
\& A
\& B
\& C
.Ve
.PP
The input source can be a file:
.PP
.Vb 1
\& parallel \-a abc\-file echo
.Ve
.PP
Output: Same as above.
.PP
\&\s-1STDIN\s0 (standard input) can be the input source:
.PP
.Vb 1
\& cat abc\-file | parallel echo
.Ve
.PP
Output: Same as above.
.SS "Multiple input sources"
.IX Subsection "Multiple input sources"
\&\s-1GNU\s0 \fBparallel\fR can take multiple input sources given on the command
line. \s-1GNU\s0 \fBparallel\fR then generates all combinations of the input
sources:
.PP
.Vb 1
\& parallel echo ::: A B C ::: D E F
.Ve
.PP
Output (the order may be different):
.PP
.Vb 9
\& A D
\& A E
\& A F
\& B D
\& B E
\& B F
\& C D
\& C E
\& C F
.Ve
.PP
The input sources can be files:
.PP
.Vb 1
\& parallel \-a abc\-file \-a def\-file echo
.Ve
.PP
Output: Same as above.
.PP
\&\s-1STDIN\s0 (standard input) can be one of the input sources using \fB\-\fR:
.PP
.Vb 1
\& cat abc\-file | parallel \-a \- \-a def\-file echo
.Ve
.PP
Output: Same as above.
.PP
Instead of \fB\-a\fR files can be given after \fB::::\fR:
.PP
.Vb 1
\& cat abc\-file | parallel echo :::: \- def\-file
.Ve
.PP
Output: Same as above.
.PP
::: and :::: can be mixed:
.PP
.Vb 1
\& parallel echo ::: A B C :::: def\-file
.Ve
.PP
Output: Same as above.
.PP
\fILinking arguments from input sources\fR
.IX Subsection "Linking arguments from input sources"
.PP
With \fB\-\-link\fR you can link the input sources and get one argument
from each input source:
.PP
.Vb 1
\& parallel \-\-link echo ::: A B C ::: D E F
.Ve
.PP
Output (the order may be different):
.PP
.Vb 3
\& A D
\& B E
\& C F
.Ve
.PP
If one of the input sources is too short, its values will wrap:
.PP
.Vb 1
\& parallel \-\-link echo ::: A B C D E ::: F G
.Ve
.PP
Output (the order may be different):
.PP
.Vb 5
\& A F
\& B G
\& C F
\& D G
\& E F
.Ve
.PP
For more flexible linking you can use \fB:::+\fR and \fB::::+\fR. They work
like \fB:::\fR and \fB::::\fR except they link the previous input source to
this input source.
.PP
This will link \s-1ABC\s0 to \s-1GHI:\s0
.PP
.Vb 1
\& parallel echo :::: abc\-file :::+ G H I :::: def\-file
.Ve
.PP
Output (the order may be different):
.PP
.Vb 9
\& A G D
\& A G E
\& A G F
\& B H D
\& B H E
\& B H F
\& C I D
\& C I E
\& C I F
.Ve
.PP
This will link \s-1GHI\s0 to \s-1DEF:\s0
.PP
.Vb 1
\& parallel echo :::: abc\-file ::: G H I ::::+ def\-file
.Ve
.PP
Output (the order may be different):
.PP
.Vb 9
\& A G D
\& A H E
\& A I F
\& B G D
\& B H E
\& B I F
\& C G D
\& C H E
\& C I F
.Ve
.PP
If one of the input sources is too short when using \fB:::+\fR or
\&\fB::::+\fR, the rest will be ignored:
.PP
.Vb 1
\& parallel echo ::: A B C D E :::+ F G
.Ve
.PP
Output (the order may be different):
.PP
.Vb 2
\& A F
\& B G
.Ve
.SS "Changing the argument separator."
.IX Subsection "Changing the argument separator."
\&\s-1GNU\s0 \fBparallel\fR can use other separators than \fB:::\fR or \fB::::\fR. This is
typically useful if \fB:::\fR or \fB::::\fR is used in the command to run:
.PP
.Vb 1
\& parallel \-\-arg\-sep ,, echo ,, A B C :::: def\-file
.Ve
.PP
Output (the order may be different):
.PP
.Vb 9
\& A D
\& A E
\& A F
\& B D
\& B E
\& B F
\& C D
\& C E
\& C F
.Ve
.PP
Changing the argument file separator:
.PP
.Vb 1
\& parallel \-\-arg\-file\-sep // echo ::: A B C // def\-file
.Ve
.PP
Output: Same as above.
.SS "Changing the argument delimiter"
.IX Subsection "Changing the argument delimiter"
\&\s-1GNU\s0 \fBparallel\fR will normally treat a full line as a single argument: It
uses \fB\en\fR as argument delimiter. This can be changed with \fB\-d\fR:
.PP
.Vb 1
\& parallel \-d _ echo :::: abc_\-file
.Ve
.PP
Output (the order may be different):
.PP
.Vb 3
\& A
\& B
\& C
.Ve
.PP
\&\s-1NUL\s0 can be given as \fB\e0\fR:
.PP
.Vb 1
\& parallel \-d \*(Aq\e0\*(Aq echo :::: abc0\-file
.Ve
.PP
Output: Same as above.
.PP
A shorthand for \fB\-d '\e0'\fR is \fB\-0\fR (this will often be used to read files
from \fBfind ... \-print0\fR):
.PP
.Vb 1
\& parallel \-0 echo :::: abc0\-file
.Ve
.PP
Output: Same as above.
.SS "End-of-file value for input source"
.IX Subsection "End-of-file value for input source"
\&\s-1GNU\s0 \fBparallel\fR can stop reading when it encounters a certain value:
.PP
.Vb 1
\& parallel \-E stop echo ::: A B stop C D
.Ve
.PP
Output:
.PP
.Vb 2
\& A
\& B
.Ve
.SS "Skipping empty lines"
.IX Subsection "Skipping empty lines"
Using \fB\-\-no\-run\-if\-empty\fR \s-1GNU\s0 \fBparallel\fR will skip empty lines.
.PP
.Vb 1
\& (echo 1; echo; echo 2) | parallel \-\-no\-run\-if\-empty echo
.Ve
.PP
Output:
.PP
.Vb 2
\& 1
\& 2
.Ve
.SH "Building the command line"
.IX Header "Building the command line"
.SS "No command means arguments are commands"
.IX Subsection "No command means arguments are commands"
If no command is given after parallel the arguments themselves are
treated as commands:
.PP
.Vb 1
\& parallel ::: ls \*(Aqecho foo\*(Aq pwd
.Ve
.PP
Output (the order may be different):
.PP
.Vb 3
\& [list of files in current dir]
\& foo
\& [/path/to/current/working/dir]
.Ve
.PP
The command can be a script, a binary or a Bash function if the function is
exported using \fBexport \-f\fR:
.PP
.Vb 6
\& # Only works in Bash
\& my_func() {
\& echo in my_func $1
\& }
\& export \-f my_func
\& parallel my_func ::: 1 2 3
.Ve
.PP
Output (the order may be different):
.PP
.Vb 3
\& in my_func 1
\& in my_func 2
\& in my_func 3
.Ve
.SS "Replacement strings"
.IX Subsection "Replacement strings"
\fIThe 7 predefined replacement strings\fR
.IX Subsection "The 7 predefined replacement strings"
.PP
\&\s-1GNU\s0 \fBparallel\fR has several replacement strings. If no replacement
strings are used the default is to append \fB{}\fR:
.PP
.Vb 1
\& parallel echo ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& A/B.C
.Ve
.PP
The default replacement string is \fB{}\fR:
.PP
.Vb 1
\& parallel echo {} ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& A/B.C
.Ve
.PP
The replacement string \fB{.}\fR removes the extension:
.PP
.Vb 1
\& parallel echo {.} ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& A/B
.Ve
.PP
The replacement string \fB{/}\fR removes the path:
.PP
.Vb 1
\& parallel echo {/} ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& B.C
.Ve
.PP
The replacement string \fB{//}\fR keeps only the path:
.PP
.Vb 1
\& parallel echo {//} ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& A
.Ve
.PP
The replacement string \fB{/.}\fR removes the path and the extension:
.PP
.Vb 1
\& parallel echo {/.} ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& B
.Ve
.PP
The replacement string \fB{#}\fR gives the job number:
.PP
.Vb 1
\& parallel echo {#} ::: A B C
.Ve
.PP
Output (the order may be different):
.PP
.Vb 3
\& 1
\& 2
\& 3
.Ve
.PP
The replacement string \fB{%}\fR gives the job slot number (between 1 and
number of jobs to run in parallel):
.PP
.Vb 1
\& parallel \-j 2 echo {%} ::: A B C
.Ve
.PP
Output (the order may be different and 1 and 2 may be swapped):
.PP
.Vb 3
\& 1
\& 2
\& 1
.Ve
.PP
\fIChanging the replacement strings\fR
.IX Subsection "Changing the replacement strings"
.PP
The replacement string \fB{}\fR can be changed with \fB\-I\fR:
.PP
.Vb 1
\& parallel \-I ,, echo ,, ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& A/B.C
.Ve
.PP
The replacement string \fB{.}\fR can be changed with \fB\-\-extensionreplace\fR:
.PP
.Vb 1
\& parallel \-\-extensionreplace ,, echo ,, ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& A/B
.Ve
.PP
The replacement string \fB{/}\fR can be replaced with \fB\-\-basenamereplace\fR:
.PP
.Vb 1
\& parallel \-\-basenamereplace ,, echo ,, ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& B.C
.Ve
.PP
The replacement string \fB{//}\fR can be changed with \fB\-\-dirnamereplace\fR:
.PP
.Vb 1
\& parallel \-\-dirnamereplace ,, echo ,, ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& A
.Ve
.PP
The replacement string \fB{/.}\fR can be changed with \fB\-\-basenameextensionreplace\fR:
.PP
.Vb 1
\& parallel \-\-basenameextensionreplace ,, echo ,, ::: A/B.C
.Ve
.PP
Output:
.PP
.Vb 1
\& B
.Ve
.PP
The replacement string \fB{#}\fR can be changed with \fB\-\-seqreplace\fR:
.PP
.Vb 1
\& parallel \-\-seqreplace ,, echo ,, ::: A B C
.Ve
.PP
Output (the order may be different):
.PP
.Vb 3
\& 1
\& 2
\& 3
.Ve
.PP
The replacement string \fB{%}\fR can be changed with \fB\-\-slotreplace\fR:
.PP
.Vb 1
\& parallel \-j2 \-\-slotreplace ,, echo ,, ::: A B C
.Ve
.PP
Output (the order may be different and 1 and 2 may be swapped):
.PP
.Vb 3
\& 1
\& 2
\& 1
.Ve
.PP
\fIPerl expression replacement string\fR
.IX Subsection "Perl expression replacement string"
.PP
When predefined replacement strings are not flexible enough a perl
expression can be used instead. One example is to remove two
extensions: foo.tar.gz becomes foo
.PP
.Vb 1
\& parallel echo \*(Aq{= s:\e.[^.]+$::;s:\e.[^.]+$::; =}\*(Aq ::: foo.tar.gz
.Ve
.PP
Output:
.PP
.Vb 1
\& foo
.Ve
.PP
In \fB{= =}\fR you can access all of \s-1GNU\s0 \fBparallel\fR's internal functions
and variables. A few are worth mentioning.
.PP
\&\fB\fBtotal_jobs()\fB\fR returns the total number of jobs:
.PP
.Vb 1
\& parallel echo Job {#} of {= \*(Aq$_=total_jobs()\*(Aq =} ::: {1..5}
.Ve
.PP
Output:
.PP
.Vb 5
\& Job 1 of 5
\& Job 2 of 5
\& Job 3 of 5
\& Job 4 of 5
\& Job 5 of 5
.Ve
.PP
\&\fBQ(...)\fR shell quotes the string:
.PP
.Vb 1
\& parallel echo {} shell quoted is {= \*(Aq$_=Q($_)\*(Aq =} ::: \*(Aq*/!#$\*(Aq
.Ve
.PP
Output:
.PP
.Vb 1
\& */!#$ shell quoted is \e*/\e!\e#\e$
.Ve
.PP
\&\fB\fBskip()\fB\fR skips the job:
.PP
.Vb 1
\& parallel echo {= \*(Aqif($_==3) { skip() }\*(Aq =} ::: {1..5}
.Ve
.PP
Output:
.PP
.Vb 4
\& 1
\& 2
\& 4
\& 5
.Ve
.PP
\&\fB\f(CB@arg\fB\fR contains the input source variables:
.PP
.Vb 2
\& parallel echo {= \*(Aqif($arg[1]==$arg[2]) { skip() }\*(Aq =} \e
\& ::: {1..3} ::: {1..3}
.Ve
.PP
Output:
.PP
.Vb 6
\& 1 2
\& 1 3
\& 2 1
\& 2 3
\& 3 1
\& 3 2
.Ve
.PP
If the strings \fB{=\fR and \fB=}\fR cause problems they can be replaced with \fB\-\-parens\fR:
.PP
.Vb 2
\& parallel \-\-parens ,,,, echo \*(Aq,, s:\e.[^.]+$::;s:\e.[^.]+$::; ,,\*(Aq \e
\& ::: foo.tar.gz
.Ve
.PP
Output:
.PP
.Vb 1
\& foo
.Ve
.PP
To define a shorthand replacement string use \fB\-\-rpl\fR:
.PP
.Vb 2
\& parallel \-\-rpl \*(Aq.. s:\e.[^.]+$::;s:\e.[^.]+$::;\*(Aq echo \*(Aq..\*(Aq \e
\& ::: foo.tar.gz
.Ve
.PP
Output: Same as above.
.PP
If the shorthand starts with \fB{\fR it can be used as a positional
replacement string, too:
.PP
.Vb 2
\& parallel \-\-rpl \*(Aq{..} s:\e.[^.]+$::;s:\e.[^.]+$::;\*(Aq echo \*(Aq{..}\*(Aq
\& ::: foo.tar.gz
.Ve
.PP
Output: Same as above.
.PP
If the shorthand contains matching parenthesis the replacement string
becomes a dynamic replacement string and the string in the parenthesis
can be accessed as $$1. If there are multiple matching parenthesis,
the matched strings can be accessed using $$2, $$3 and so on.
.PP
You can think of this as giving arguments to the replacement
string. Here we give the argument \fB.tar.gz\fR to the replacement string
\&\fB{%\f(BIstring\fB}\fR which removes \fIstring\fR:
.PP
.Vb 1
\& parallel \-\-rpl \*(Aq{%(.+?)} s/$$1$//;\*(Aq echo {%.tar.gz}.zip ::: foo.tar.gz
.Ve
.PP
Output:
.PP
.Vb 1
\& foo.zip
.Ve
.PP
Here we give the two arguments \fBtar.gz\fR and \fBzip\fR to the replacement
string \fB{/\f(BIstring1\fB/\f(BIstring2\fB}\fR which replaces \fIstring1\fR with
\&\fIstring2\fR:
.PP
.Vb 2
\& parallel \-\-rpl \*(Aq{/(.+?)/(.*?)} s/$$1/$$2/;\*(Aq echo {/tar.gz/zip} \e
\& ::: foo.tar.gz
.Ve
.PP
Output:
.PP
.Vb 1
\& foo.zip
.Ve
.PP
\&\s-1GNU\s0 \fBparallel\fR's 7 replacement strings are implemented as this:
.PP
.Vb 8
\& \-\-rpl \*(Aq{} \*(Aq
\& \-\-rpl \*(Aq{#} $_=$job\->seq()\*(Aq
\& \-\-rpl \*(Aq{%} $_=$job\->slot()\*(Aq
\& \-\-rpl \*(Aq{/} s:.*/::\*(Aq
\& \-\-rpl \*(Aq{//} $Global::use{"File::Basename"} ||=
\& eval "use File::Basename; 1;"; $_ = dirname($_);\*(Aq
\& \-\-rpl \*(Aq{/.} s:.*/::; s:\e.[^/.]+$::;\*(Aq
\& \-\-rpl \*(Aq{.} s:\e.[^/.]+$::\*(Aq
.Ve
.PP
\fIPositional replacement strings\fR
.IX Subsection "Positional replacement strings"
.PP
With multiple input sources the argument from the individual input
sources can be accessed with \fB{\fRnumber\fB}\fR:
.PP
.Vb 1
\& parallel echo {1} and {2} ::: A B ::: C D
.Ve
.PP
Output (the order may be different):
.PP
.Vb 4
\& A and C
\& A and D
\& B and C
\& B and D
.Ve
.PP
The positional replacement strings can also be modified using \fB/\fR, \fB//\fR, \fB/.\fR, and \fB.\fR:
.PP
.Vb 1
\& parallel echo /={1/} //={1//} /.={1/.} .={1.} ::: A/B.C D/E.F
.Ve
.PP
Output (the order may be different):
.PP
.Vb 2
\& /=B.C //=A /.=B .=A/B
\& /=E.F //=D /.=E .=D/E
.Ve
.PP
If a position is negative, it will refer to the input source counted
from behind:
.PP
.Vb 2
\& parallel echo 1={1} 2={2} 3={3} \-1={\-1} \-2={\-2} \-3={\-3} \e