Create Porting/MANIFEST.dev as a complement to MANIFEST and related infra

This file is intended to list all the files in the repo which are not
listed in the main MANIFEST file, and which are used only for
development purposes, especially those files which are only useful when
working in a git checkout of the main perl git repository.

The files it contains will NOT be added to the production tarball
release. The file has the exact same format as the main MANIFEST:
"file\t+description" or "file".

Q. Why didn't I call this Porting/MANIFEST as mentioned in the
   discussion thread that lead to this patch?

A. The main reason was that Porting/README.pod includes a list of files
   in Porting with descriptions and explanations for what the files do
   or how they are used. In several places the file refers to
   "MANIFEST", which lead to ambiguity that would have had to be
   resolved by changing all the entries to refer to "Porting/whatever"
   instead. It was much simpler to give the new file an extension, and I
   thought that '.dev' suggests it is for "development" purposes.

Q. Why isn't this using MANIFEST.skip style functionality?

A1. Various parts of our build and test process expect to read the
    MANIFEST file and then do things based on the entries contained
    within. Eg, run tests, or extract data, or compare the file list to
    content in another file. Those parts of our build process would
    break if we used a skip style list of regexen. So it would be more
    work to teach them to deal with such a file, assuming it was
    actually doable - given the additional work I have not considered it
    deeply. On the other hand teaching that logic to simply read two
    files was and is easy.

A2. I think each file we have in the repo should have a description.
    This patch currently doesn't provide a description for each, but it
    does for many, especially those migrated from MANIFEST.

A3. I think that MANIFEST.skip style files of exclusion regexens and
    globs are error prone and easy to mess up, for instance by excluding
    far more than you had intended to. They can also be annoying to get
    right, obviously not impossible, but sometimes annoying. Explicitly
    listing everything is easy in every way, especially to mechanize.

A4. I would like to be able to move verbatim entries from our existing
    MANIFEST into the new Porting/MANIFEST.dev, description and all.
    MANIFEST.skip style files do not support descriptions except as
    comments as far as I recall. That would have meant munging the data
    from MANIFEST during the move process which would be annoying.

A5. I would like to be able to reuse our sorting logic to keep the files
    nicely sorted in a way where the file is somewhat readable. A list
    of skip files would be less amenable to doing so.

Q. There is a lot of duplicated logic related to manifests, should we
   refactor it out into a module or some resuable tool set?

A. YES! We already have Porting/manifest_lib.pm, but it currently does
   not declare a package, and it only contains one function. Instead of
   adding yet more code that depends on requiring a file and having it
   inject subs into package main I decided that doing the refactoring
   could wait for a separate commit or PR. But I definitely think we
   should refactor as much of this logic as possible.

Q. Some of the test files were fairly significantly changed, are you
   sure you didn't break or drop any of the tests?

A. I am reasonably confident I did not. Secondary review appreciated.
   Some of the touched files are quite old and obviously "quick hack"
   scripts. By rewriting them quite a bit I was able to simplify and
   perform some of the tests in different ways or parts of the script.
   As far as I know I didn't drop any.

Q. Why didn't you use newer features in the rewrite?

A. I am a bit conservative in my taste, and I like build tools to be
   able to run on older perls, and for things like this I prefer to
   stick with what I know well. Patches welcome.

Q. Why didn't you move more of the stuff we shouldn't bundle with
   our releases?

A. I figured someone like Nicolas R. (who helped motivate this patch)
   would feel left out if I didn't leave him anything to do. :-)

Author: demerphq

MANIFEST

@@ -2,7 +2,6 @@
 .editorconfig			EditorConfig style file
 .lgtm.yml			LGTM.com configuration file
 .metaconf-exclusions.txt	Symbols that should ignored when generating Configure
-.travis.yml		continuous integration on github (where enabled)
 amigaos4/amigaio.c	AmigaOS4 port
 amigaos4/amigaio.h	AmigaOS4 port
 amigaos4/amigaos.c	AmigaOS4 port
@@ -5310,8 +5309,8 @@ pod/perl58delta.pod		Perl changes in version 5.8.0
 pod/perlapio.pod		Perl internal IO abstraction interface
 pod/perlartistic.pod		Perl Artistic License
 pod/perlbook.pod		Perl book information
-pod/perlboot.pod		
-pod/perlbot.pod			
+pod/perlboot.pod		Links to info on OO programming in Perl
+pod/perlbot.pod			Links to info on OO programming in Perl
 pod/perlcall.pod		Perl calling conventions from C
 pod/perlcheat.pod		Perl cheat sheet
 pod/perlclib.pod		Internal replacements for standard C library functions
@@ -5375,7 +5374,7 @@ pod/perlrecharclass.pod		Perl regular expression character classes
 pod/perlref.pod			Perl references, the rest of the story
 pod/perlreftut.pod		Perl references short introduction
 pod/perlreguts.pod		Perl regular expression engine internals
-pod/perlrepository.pod		
+pod/perlrepository.pod		Links to current info on the Perl source repository
 pod/perlrequick.pod		Perl regular expressions quick start
 pod/perlreref.pod		Perl regular expressions quick reference
 pod/perlretut.pod		Perl regular expressions tutorial
@@ -5388,9 +5387,9 @@ pod/perlsub.pod			Perl subroutines
 pod/perlsyn.pod			Perl syntax
 pod/perlthrtut.pod		Perl threads tutorial
 pod/perltie.pod			Perl objects hidden behind simple variables
-pod/perltodo.pod		
-pod/perltooc.pod		
-pod/perltoot.pod		
+pod/perltodo.pod		Link to the Perl to-do list
+pod/perltooc.pod		Links to info on OO programming in Perl
+pod/perltoot.pod		Links to info on OO programming in Perl
 pod/perltrap.pod		Perl traps for the unwary
 pod/perlunicode.pod		Perl Unicode support
 pod/perlunicook.pod		Perl Unicode cookbook
@@ -5408,14 +5407,10 @@ Porting/acknowledgements.pl	Generate perldelta acknowledgements text
 Porting/add-package.pl		Add/Update CPAN modules that are part of Core
 Porting/add-pod-file		Utility to add new pod/*.pod file to core distribution
 Porting/bench.pl		Run benchmarks against t/perf/benchmarks
-Porting/bisect.pl		A tool to make bisecting easy
-Porting/bisect-example.sh	Example script to use with git bisect run
-Porting/bisect-runner.pl	Tool to be called by git bisect run
 Porting/bump-perl-version	bump the perl version in relevant files
 Porting/check-cpan-pollution	Check for commits that may wrongly touch CPAN distros
 Porting/check83.pl		Check whether we are 8.3-friendly
 Porting/checkansi.pl		Check source code for ANSI-C violations
-Porting/checkAUTHORS.pl		Check that the AUTHORS file is complete
 Porting/checkcfguse.pl		Check that config symbols are being used
 Porting/checkcfgvar.pl		Check that config scripts define all symbols
 Porting/checkpodencoding.pl	Check POD encoding
@@ -5438,10 +5433,6 @@ Porting/exec-bit.txt		List of files that get +x in release tarball
 Porting/exercise_makedef.pl	Brute force testing for makedef.pl
 Porting/expand-macro.pl		A tool to expand C macro definitions in the Perl source
 Porting/findrfuncs		Find reentrant variants of functions used in an executable
-Porting/git-deltatool		Mark commits for perldelta in git notes
-Porting/git-find-p4-change	Find the change for a p4 change number
-Porting/git-make-p4-refs	Output git refs for each p4 change number, suitable for appending to .git/packed-refs
-Porting/GitUtils.pm		Generate the contents of a .patch file
 Porting/Glossary		Glossary of config.sh variables
 Porting/harness-timer-report.pl	Analyze the timings from the test harness
 Porting/how_to_write_a_perldelta.pod	Bluffer's guide to writing a perldelta.
@@ -5479,7 +5470,6 @@ Porting/sync-with-cpan		Sync with CPAN
 Porting/timecheck.c		Test program for the 2038 fix
 Porting/timecheck2.c		Test program for the 2038 fix
 Porting/todo.pod		Perl things to do
-Porting/updateAUTHORS.pl	Tool to automatically update AUTHORS and .mailmap from git log data
 Porting/valgrindpp.pl		Summarize valgrind reports
 Porting/vote_admin_guide.pod	Perlgov Vote Administrator guide
 pp.c				Push/Pop code

Makefile.SH

@@ -667,7 +667,11 @@ all: $(FIRSTMAKEFILE) $(MINIPERL_EXE) $(generated_pods) $(private) $(unidatafile
 
 $(MANIFEST_SRT):	MANIFEST $(PERL_EXE)
 	@$(RUN_PERL) Porting/manisort -q || (echo "WARNING: re-sorting MANIFEST"; \
-	    $(RUN_PERL) Porting/manisort -q -o MANIFEST; sh -c true)
+	    $(RUN_PERL) Porting/manisort --fix MANIFEST; sh -c true)
+	@$(RUN_PERL) Porting/manisort -q Porting/MANIFEST.dev || \
+	(echo "WARNING: re-sorting Porting/MANIFEST.dev"; \
+	    $(RUN_PERL) Porting/manisort --fix Porting/MANIFEST.dev; \
+	    sh -c true)
 	@touch $(MANIFEST_SRT)
 
 .PHONY: all utilities
@@ -1287,7 +1291,10 @@ regen_all regen-all: regen regen_meta
 
 manisort:	FORCE
 	@perl Porting/manisort -q || (echo "WARNING: re-sorting MANIFEST"; \
-		perl Porting/manisort -q -o MANIFEST; sh -c true)
+		perl Porting/manisort --fix MANIFEST; sh -c true)
+	@perl Porting/manisort -q Porting/MANIFEST.dev || \
+	    (echo "WARNING: re-sorting Porting/MANIFEST.dev"; \
+		perl Porting/manisort --fix Porting/MANIFEST.dev; sh -c true)
 
 manicheck:	FORCE
 	perl Porting/manicheck

Porting/MANIFEST.dev

@@ -0,0 +1,84 @@
+.git_patch				Format used by make_patchnum.pl
+.gitattributes				See: git --help gitattributes
+.github/ISSUE_TEMPLATE/01-perlbug-core.md
+.github/ISSUE_TEMPLATE/02-perlbug-docs.md
+.github/ISSUE_TEMPLATE/config.yml
+.github/workflows/detect-conflicts.yml
+.github/workflows/irc-notifications.yaml
+.github/workflows/testsuite.yml
+.gitignore				Generic .gitignore rules. See: git --help gitignore
+.mailmap				Map canonical name/email to those in commits
+.travis.yml				continuous integration on github (where enabled)
+cpan/.gitignore				Generic .gitignore for the cpan/ tree
+cpan/Compress-Raw-Bzip2/.gitignore	.gitignore rules for the Compress-Raw-Bzip2 cpan distribution
+cpan/Compress-Raw-Zlib/.gitignore	.gitignore rules for the Compress-Raw-Zlib cpan distribution
+cpan/DB_File/.gitignore			.gitignore rules for the DB_File cpan distribution
+cpan/Digest-MD5/.gitignore		.gitignore rules for the Digest-MD5 cpan distribution
+cpan/Digest-SHA/.gitignore		.gitignore rules for the Digest-SHA cpan distribution
+cpan/Encode/.gitignore			.gitignore rules for the Encode cpan distribution
+cpan/IO-Compress/.gitignore		.gitignore rules for the IO-Compress cpan distribution
+cpan/IPC-SysV/.gitignore		.gitignore rules for the IPC-SysV cpan distribution
+cpan/libnet/.gitignore			.gitignore rules for the libnet cpan distribution
+cpan/perlfaq/.gitignore			.gitignore rules for the perlfaq cpan distribution
+cpan/Pod-Checker/.gitignore		.gitignore rules for the Pod-Checker cpan distribution
+cpan/Pod-Perldoc/.gitignore		.gitignore rules for the Pod-Perldoc cpan distribution
+cpan/Pod-Usage/.gitignore		.gitignore rules for the Pod-Usage cpan distribution
+cpan/podlators/.gitignore		.gitignore rules for the podlators cpan distribution
+cpan/Scalar-List-Utils/.gitignore	.gitignore rules for the Scalar-List-Utils cpan distribution
+cpan/Socket/.gitignore			.gitignore rules for the Socket cpan distribution
+cpan/Sys-Syslog/.gitignore		.gitignore rules for the Sys-Syslog cpan distribution
+cpan/Unicode-Collate/.gitignore		.gitignore rules for the Unicode-Collate cpan distribution
+cpan/Win32/.gitignore			.gitignore rules for the Win32 cpan distribution
+cpan/Win32API-File/.gitignore		.gitignore rules for the Win32API-File cpan distribution
+dist/.gitignore				Generic .gitignore for the dist/ tree
+dist/base/.gitignore			.gitignore rules for the base distribution
+dist/Carp/.gitignore			.gitignore rules for the Carp distribution
+dist/Devel-PPPort/.gitignore		.gitignore rules for the Devel-PPPort distribution
+dist/ExtUtils-CBuilder/.gitignore	.gitignore rules for the ExtUtils-CBuilder distribution
+dist/IO/.gitignore			.gitignore rules for the IO distribution
+dist/lib/.gitignore			.gitignore rules for the lib distribution
+dist/Module-CoreList/.gitignore		.gitignore rules for the Module-CoreList distribution
+dist/PathTools/.gitignore		.gitignore rules for the PathTools distribution
+dist/Safe/.gitignore			.gitignore rules for the Safe distribution
+dist/Search-Dict/.gitignore		.gitignore rules for the Search-Dict distribution
+dist/Storable/.gitignore		.gitignore rules for the Storable distribution
+dist/Time-HiRes/.gitignore		.gitignore rules for the Time-HiRes distribution
+dist/Unicode-Normalize/.gitignore	.gitignore rules for the Unicode-Normalize distribution
+dist/XSLoader/.gitignore		.gitignore rules for the XSLoader distribution
+ext/.gitignore				Generic .gitignore for the ext/ tree
+ext/Amiga-ARexx/.gitignore		.gitignore rules for the Amiga-ARexx extension
+ext/Amiga-Exec/.gitignore		.gitignore rules for the Amiga-Exec extension
+ext/B/.gitignore			.gitignore rules for the B extension
+ext/DynaLoader/.gitignore		.gitignore rules for the DynaLoader extension
+ext/Errno/.gitignore			.gitignore rules for the Errno extension
+ext/Fcntl/.gitignore			.gitignore rules for the Fcntl extension
+ext/File-Glob/.gitignore		.gitignore rules for the File-Glob extension
+ext/GDBM_File/.gitignore		.gitignore rules for the GDBM_File extension
+ext/Hash-Util/.gitignore		.gitignore rules for the Hash-Util extension
+ext/I18N-Langinfo/.gitignore		.gitignore rules for the I18N-Langinfo extension
+ext/NDBM_File/.gitignore		.gitignore rules for the NDBM_File extension
+ext/ODBM_File/.gitignore		.gitignore rules for the ODBM_File extension
+ext/Pod-Functions/.gitignore		.gitignore rules for the Pod-Functions extension
+ext/Pod-Html/.gitignore			.gitignore rules for the Pod-Html extension
+ext/POSIX/.gitignore			.gitignore rules for the POSIX extension
+ext/re/.gitignore			.gitignore rules for the re extension
+ext/SDBM_File/.gitignore		.gitignore rules for the SDBM_File extension
+ext/VMS-DCLsym/.gitignore		.gitignore rules for the VMS-DCLsym extension
+ext/VMS-Stdio/.gitignore		.gitignore rules for the VMS-Stdio extension
+ext/Win32CORE/.gitignore		.gitignore rules for the Win32CORE extension
+ext/XS-APItest/.gitignore		.gitignore rules for the XS-APItest extension
+ext/XS-Typemap/.gitignore		.gitignore rules for the XS-Typemap extension
+lib/.gitignore				Generic .gitignore for the lib/ tree
+pod/.gitignore				Generic .gitignore for the pod/ tree
+Porting/bisect.pl		A tool to make bisecting easy
+Porting/bisect-example.sh	Example script to use with git bisect run
+Porting/bisect-runner.pl	Tool to be called by git bisect run
+Porting/checkAUTHORS.pl		Check that the AUTHORS file is complete
+Porting/git-deltatool		Mark commits for perldelta in git notes
+Porting/git-find-p4-change	Find the change for a p4 change number
+Porting/git-make-p4-refs	Output git refs for each p4 change number, suitable for appending to .git/packed-refs
+Porting/GitUtils.pm		Generate the contents of a .patch file
+Porting/MANIFEST.dev		Porting files - all files in repo we do not ship
+Porting/updateAUTHORS.pl	Tool to automatically update AUTHORS and .mailmap from git log data
+utils/.gitignore		Generic .gitignore for the utils/ tree
+win32/.gitignore		Generic .gitignore for the win32/ tree

Porting/README.pod

@@ -253,16 +253,39 @@ web page to use to generate the snapshot files.
 
 =head2 F<manicheck>
 
-This script outputs a list of files in F<MANIFEST> which don't exist and a
-list of files that exist and aren't in F<MANIFEST>.
+This script outputs a list of files in F<MANIFEST> and F<MANIFEST.dev>
+which don't exist and a list of files that exist and aren't in
+F<MANIFEST> or in F<MANIFEST.dev>
+
+=head2 F<MANIFEST.dev>
+
+This file lists all files that are not included in the F<MANIFEST>
+file in the root of the repo. The union of the files it contains with
+the files in C<MANIFEST> should be the same the list of file produced
+by C<git ls-files> from a git checkout.
 
 =head2 F<manifest_lib.pl>
 
-This library provides functions used in checking and sorting the F<MANIFEST>.
+This library provides functions used in checking and sorting the
+F<MANIFEST> and F<MANIFEST.dev> files. The sort order is similar to
+dictionary sort order (alphabetical case insensitive) but where path
+components and extensions are sorted independently such that the
+following files would be sorted into the following order:
+
+    Configure
+    configure
+    lib/Foo/Bar
+    lib/Foo/Bar.pm
+    lib/Foo/Bar/Alpha.pm
+    lib/Foo-Alpha/Baz
+
+Currently this code does not support EBCDIC. Patches welcome.
 
 =head2 F<manisort>
 
-This script sorts the files in F<MANIFEST>.
+This script sorts the files in F<MANIFEST> or F<MANIFEST.dev>. It uses
+F<manifest_list.pl> to do the sorting. Use C<Porting/manisort --man>
+to see instructions for use.
 
 =head2 F<mksample>
 
@@ -299,6 +322,12 @@ distribution, provides functions useful during testing.
 Generate the sections of files listed in C<%Targets> from F<pod/perl.pod>.
 Mostly these are rules in Makefiles.
 
+An example of what this tool does is to ensure that every podfile listed
+in F<pod/perl.pod> is also listed in F<MANIFEST> with the same
+description in F<pod/perl.pod>. If they differ in test mode it will
+complain. To update the manifest entries run it from the command line
+without the --test argument.
+
  --verbose gives slightly more output
  --build-all tries to build everything
  --build-foo updates foo as follows
@@ -403,4 +432,3 @@ leaks.
 Guide for Vote Administrators for running Steering Council elections.
 
 =cut
-

Porting/manicheck

@@ -4,55 +4,69 @@
 #  a) files listed in MANIFEST which don't exist
 #  b) files which exist but which aren't in MANIFEST
 
+# this file should be refactored with t/porting/manifest.t and
+# Porting/manifest_list.pm
+
 use v5.14;
 use warnings;
 use File::Find;
 use Getopt::Long;
-use constant SKIP => 125;
+use constant MAX_EXIT_CODE => 124;
 
 my $exitstatus;
 GetOptions('exitstatus!', \$exitstatus)
     or die "$0 [--exitstatus]";
 
-my %files;
-my $missing = 0;
 my $bonus = 0;
 
-open my $fh, '<', 'MANIFEST' or die "Can't read MANIFEST: $!\n";
-for my $line (<$fh>) {
-    my ($file) = $line =~ /^(\S+)/;
-    ++$files{$file};
-    next if -f $file;
-    ++$missing;
-    print "$file from MANIFEST doesn't exist\n";
+sub read_manifest {
+    my ($manifest_file) = @_;
+    open my $fh, '<', $manifest_file
+        or die "Can't read '$manifest_file': $!\n";
+    my %files;
+    my $missing = 0;
+    for my $line (<$fh>) {
+        my ($file) = $line =~ /^(\S+)/;
+        if (!defined($file)) {
+            warn "No file specified in '$manifest_file' line $.\n";
+            next;
+        }
+        $files{$file} = $.;
+        next if -f $file;
+        $missing++;
+        print "'$file' from '$manifest_file' at line $. doesn't exist\n";
+    }
+    close $fh or die "Failed to close '$manifest_file': $!\n";
+    return (\%files, $missing);
 }
-close $fh;
+
+my ($manifest_files, $manifest_missing) = read_manifest("MANIFEST");
+my ($porting_files, $porting_missing) = read_manifest("Porting/MANIFEST.dev");
 
 find {
+    no_chdir => 1,
     wanted => sub {
-        return if -d;
-        return if $_ eq '.mailmap';
-        return if $_ eq '.gitignore';
-        return if $_ eq '.gitattributes';
-        return if $_ eq '.git_patch';
-
-        my $x = $File::Find::name =~ s!^\./!!r;
-        return if $x =~ /^\.git\b/;
-        return if $x =~ m{^\.github/};
-        return if $files{$x};
+        return if -d $_;
+        my $file = $_;
+        $file =~s!^\./!!;
+        return if $file=~m!^\.git/!;
+        return if $porting_files->{$file};
+        return if $manifest_files->{$file};
         ++$bonus;
-        print "$x\t\tnot in MANIFEST\n";
+        print "$file\t\tnot in MANIFEST and not in Porting/MANIFEST.dev\n";
     },
 }, ".";
 
-my $exitcode = $exitstatus ? $missing + $bonus : 0;
+my $problems = $manifest_missing + $porting_missing + $bonus;
 
 # We can't (meaningfully) exit with codes above 255, so we're going to have to
 # clamp them to some range whatever we do. So as we need the code anyway, use
 # 124 as our maximum instead, and then we can run as a useful git bisect run
 # script if needed...
-
-$exitcode = SKIP - 1
-    if $exitcode > SKIP;
+my $exitcode = (!$exitstatus)
+               ? 0
+               : ($problems > MAX_EXIT_CODE)
+                 ? MAX_EXIT_CODE
+                 : $problems;
 
 exit $exitcode;

Porting/manifest_lib.pl

@@ -1,5 +1,4 @@
 #!/usr/bin/perl
-
 use strict;
 
 =head1 NAME
@@ -21,6 +20,9 @@ =head2 C<sort_manifest>
 
 =cut
 
+# this file should be refactored with t/porting/manifest.t and
+# Porting/manicheck
+
 # Try to get a sane sort. case insensitive, more or less
 # sorted such that path components are compared independently,
 # and so that lib/Foo/Bar sorts before lib/Foo-Alpha/Baz