Build infrastructure Perl to Python port

[abensonca]

This is a rich codebase to analyze. Let me give you a frank assessment of the Inline::Python approach and a recommended strategy instead.

---

Assessment of Inline::Python

I'd advise against using Inline::Python as the interop bridge. The key problems:

1. Data structure translation is fragile. The build system passes around deeply nested Perl data structures — blessed hashrefs representing Fortran AST nodes, anonymous sub-callbacks registered in hook hashes, etc. Inline::Python can handle scalars and simple lists, but blessed objects and complex refs translate poorly.
2. Inline::Python is poorly maintained. It has known incompatibilities with newer Perls and modern Python 3.x builds.
3. Debugging becomes a nightmare. Stack traces span two runtimes, and errors at the boundary are cryptic.
4. The hook/plugin architecture makes it harder. Modules register callbacks into %moduleHooks and %processHooks. A Python module can't natively register a callback into a Perl hash.

---

Recommended Strategy: Parallel Python Rewrite with Script-Level Cutover

Rather than interop at the module level, interop at the process boundary — keep Perl calling Perl and Python calling Python, and replace whole entry-point scripts at once.

Phase 0: Build a Python Test Harness First

Before porting anything, capture the behavior of the existing system:

• Run the build system on representative inputs and save outputs (generated Fortran/C++ files, directive parse results, etc.)
• Write a Python test framework that compares Python output to these golden files
• This gives you confidence that each ported module is behaviorally equivalent

Phase 1: Port the Infrastructure (~500 lines, ~4 modules)

These are leaf nodes with no internal dependencies — port them first and establish the Python package structure (galacticus/build/):

Module	Lines	Notes
Hooks.pm	11	Just a shared dict — trivial
SourceTree/Hooks.pm	15	Same
Dependencies.pm	132	Topological sort — graphlib in stdlib
Directives.pm	100	XML parsing — lxml or xml.etree
Make.pm	28	Trivial

[abensonca]

Phase 2: Port Core Utilities (~430 lines)

These are the most heavily reused modules (25+ and 15+ dependents respectively). Nothing else in Python can work without them:

Module	Lines	Notes
Components/Utils.pm	288	Type maps, label formatting — plain dicts/functions
Components/DataTypes.pm	139	Type system — a few dataclasses
Components/CodeGeneration.pm	50	Code string helpers

Phase 3: Port the Parsers (~1,330 lines)

The SourceTree/Parse/ modules are relatively self-contained and are prerequisites for all the Process/ modules. Port them as a group:

• Parse/Declarations.pm (338) — most critical, reused by 5+ Process modules
• Parse/ModuleUses.pm (324)
• Parse/Directives.pm (200)
• Parse/OpenMP.pm (210)
• Parse/Visibilities.pm (137)
• Parse/ModuleProcedures.pm (121)

At this point, you have a working Python parser stack. You can write standalone Python scripts that parse Fortran and verify output.

Phase 4: Port Process Modules Bottom-Up (~13,000 lines across 38 modules)

Work from smallest/simplest to largest. The natural order, grouped by size:

Small processors first (34–120 lines each): NonProcessed, DeepCopyReset, DeepCopyFinalize, Allocate, DebugMPI, OptionalArgument, ParameterMigration, HDF5FCInterop, ProfileOpenMP, StateStore, DebugHDF5, AddMetaProperty, Dependencies, SourceIntrospection, etc.

Medium processors (120–300 lines): ForEach, ConditionalCall, InputParameter, Constructors, Generics, Constants, ThreadSafeIO, DeepCopyActions, SourceDigest, ClassDocumentation, Enumeration

Large processors last: StateStorable (499), EventHooksStatic (111) + EventHooks (727), ObjectBuilder (841), and finally FunctionClass (3,742 — plan this as its own multi-week effort)

Phase 5: Port the Components Domain (~12,000 lines)

After the SourceTree domain works in Python, port Components/ and its sub-trees in dependency order:
Hierarchy/ → Properties/ → Classes/ → Implementations/ → TreeNodes/ → top-level Components.pm

Phase 6: Cut Over Entry-Point Scripts

Only at the end do you replace the Perl driver scripts (buildCode.pl, preprocess.pl, etc.) with Python equivalents. These scripts are the only interop point needed — you never need Inline::Python at all.

---

Key Principles

• Never mix runtimes within a process. The interop boundary is always a subprocess call or file.
• Port tests alongside code. Each module gets a Python test that validates against golden output.
• Keep the Perl system fully working until a whole entry-point script is ready to cut over. Don't break the build mid-migration.
• FunctionClass.pm (3,742 lines) deserves its own planning session. It's the most complex module in the repo — treat it as a mini-project.

---

What Python Libraries to Use

Perl	Python equivalent
XML::Simple / XML::SAX	lxml or xml.etree.ElementTree
Text::Template	string.Template or Jinja2
PDL (Perl Data Language)	NumPy
File::Find	os.walk or pathlib
Storable (serialization)	pickle or json
threads	concurrent.futures or multiprocessing
Topological sort	graphlib.TopologicalSorter (Python 3.9+)