Migration - Migration loadmodule : need a CopyMode.LOAD in migrate.groovy script

[FALLAI-Denis]

Hi,

We need to migrate loadmodule to Git... (even if Git is not really made to manage binaries).

It seems that the migrate.groovy procedure delivered with DBB does not provide the copyMode(CopyMode.LOAD)... only the copyMode(CopyMode.BINARY).

Is it possible to upgrade the migrate.groovy script so that it manages the loadmodule type, or tell us what changes we need to apply to it ourselves.

Thanks.

[dennis-behm]

Hi @FALLAI-Denis ,

As the documentation points out, the difference between the LOAD and BINARY copy is that LOAD is preserving ALIAS and ISPF data.

Is this a requirement for your migration of load modules? I don't know how Git will treat the additional information and if these are preserved, when you add them to the git repo. I am not aware of any tests performed in this area with load modules. (Please keep us updated about your results of this investigation).

What about working with a deputy member in the git repository, which points to the binary a binary artifact repository like Artifactory (including versioning) or a protected PDS?

I am struggling to understand the reason why you want to store the load module in git. Can you elaborate a bit more on your use case?

Anyway, the migrate.groovy script is located in the DBB toolkit installation, and you can obviously create a copy for your own use and modify it to your needs and change the copy mode in case a binary file is detected. As you will notice, the existing mappingRule and mappings don't let you specify a BINARY file, while the purpose of the migrate script is to migrate source code. I have a concern, that the migration script is the correct place to insert this logic.

You might want to create an idea on ideas.ibm.com to either extend or ship a migration script for load modules/object decks.

[FALLAI-Denis]

Hi @dennis-behm,

I think the documentation is not exhaustive.
It seems to me that a loadmodule cannot be "copied", but a loadmodule moves via a "linkedit" operation.
Under MVS, to "copy" a loadmodule by IEBCOPY we use a "COPYMOD" command instead of "COPY".
Under USS, to "copy" a loadmodule one uses the "cp -X" command, (if the -X option is not used, the loadmodule is no longer executable).

We must manage by reference components that only exist in the form of loadmodule (we also have objects in the sense of compilation result).
These loadmodules correspond either to external software packages, or to programs whose sources have been lost.
They must be versioned, and may be required at linkedit time for compiled programs (static linkedit).

We considered using Artifactory to manage binaries (loadmodules or others), but this required a very complex implementation.
With management by Git, during the DBB build all the components, text or binary sources, are copied into PDS, before launching the DBB build (prebuild step), then the DBB build is launched, exploiting and completing the PDS, then the PDS are used to create a "package" through a USS filesystem.
On deployment, the package is restored to USS, then copied back to PDS.

Using Git to manage both the versioning of these loadmodules (and other binaries) components and the provisioning for (static) linkedits seemed to us to be the simplest and most relevant solution.

At the Git level, these components are declared in .gitattributes with a binary mode.

During Git merge operations, loadmodules files (like other binary files) are treated as a complete / non-breaking element, (no line type management as with text sources)

Our (current) .gitattributes:

# Les fichiers sont gérés et édités en UTF-8 ; sur z/OS, sauf contre indication, ils sont encodés en EBCDIC IBM-1147
*          text        working-tree-encoding=utf-8     zos-working-tree-encoding=ibm-1147
*.c                                                    zos-working-tree-encoding=ibm-1047
*.cpp                                                  zos-working-tree-encoding=ibm-1047
*.groovy                                               zos-working-tree-encoding=ibm-1047
*.h                                                    zos-working-tree-encoding=ibm-1047
*.i                                                    zos-working-tree-encoding=ibm-1047
*.java                                                 zos-working-tree-encoding=ibm-1047
*.json                                                 zos-working-tree-encoding=utf-8
*.md                                                   zos-working-tree-encoding=utf-8
*.properties                                           zos-working-tree-encoding=ibm-1047
*.rexx                                                 zos-working-tree-encoding=ibm-037
*.sh                                                   zos-working-tree-encoding=ibm-1047

# Les fichiers de configuration Zowe en utf-8 et eol=lf
zowe.*          eol=lf working-tree-encoding=utf-8     zos-working-tree-encoding=utf-8

# Les fichiers de gestion Git sont gérés en ISO-8859-1 partout
.git/                  working-tree-encoding=iso8859-1 zos-working-tree-encoding=iso8859-1
*.gitattributes eol=lf working-tree-encoding=iso8859-1 zos-working-tree-encoding=iso8859-1
*.zosattributes eol=lf working-tree-encoding=iso8859-1 zos-working-tree-encoding=iso8859-1
*.gitignore     eol=lf working-tree-encoding=iso8859-1 zos-working-tree-encoding=iso8859-1

# Types Endevor binaires
*.load      binary
*.dbrm      binary
*.objet     binary
*.objetc    binary

# Fichiers bureautique
*.ppt*     binary
*.doc*     binary
*.xls*     binary
*.pdf      binary

# Images
*.png      binary
*.gif      binary
*.jpg      binary
*.jpeg     binary
*.bmp      binary
*.ico      binary

Sequence of build and packaging operations:

[FALLAI-Denis]

Hi,

Problem also on DBRM member:

+ /u/JENKINS/testgroovyz /global/SYS9/DBB/migration/bin/migrate.groovy -r /SYSTEM/var/tmp/DBB/JENKINS/workspace/migration/CEH49-CEAB3/CEAB3-Step1-Package -np warning -m MappingRule[hlq:GTEMP.MIGARTI.S9.SIRIS.REF.R,targetDir:SIRIS/DBRM,pdsMapping:false] -o /var/tmp/DBB/JENKINS/workspace/migration/outputs/S9/mappings_REF.R.SIRIS.DBRMLIB.txt -l /var/tmp/DBB/JENKINS/workspace/migration/logs/S9/logging_REF.R.SIRIS.DBRMLIB.txt DBRMLIB 
[Pipeline] echo
[INFO] ---- Retour du script:
 VERSION MYSYS
Generated mappings will be saved in /var/tmp/DBB/JENKINS/workspace/migration/outputs/S9/mappings_REF.R.SIRIS.DBRMLIB.txt
Messages will be saved in /var/tmp/DBB/JENKINS/workspace/migration/logs/S9/logging_REF.R.SIRIS.DBRMLIB.txt
Non-printable scan level is warning
Local GIT repository: /SYSTEM/var/tmp/DBB/JENKINS/workspace/migration/CEH49-CEAB3/CEAB3-Step1-Package
Using mapping rule com.ibm.dbb.migration.MappingRule to migrate the data sets
Migrating data set DBRMLIB
[WARNING] Copying GTEMP.MIGARTI.S9.SIRIS.REF.R.DBRMLIB(SA91CDP) to /SYSTEM/var/tmp/DBB/JENKINS/workspace/migration/CEH49-CEAB3/CEAB3-Step1-Package/SIRIS/DBRM/SA91CDP
 ! Possible migration issue:
      Line 1 contains non-printable characters:
        Char 0x00 at column 5
        Char 0x00 at column 6
        Char 0x00 at column 7
        Char 0x18 at column 25
        Char 0x09 at column 28
        Char 0x1A at column 29
...
        Char 0x35 at column 24
        Char 0x2D at column 26
      Line 15 contains non-printable characters:
        Char 0x00 at column 5
        Char 0x00 at column 6
        Char 0x00 at column 7
        Char 0x2C at column 8
        Char 0x00 at column 9
        Char 0x01 at column 10
        Char 0x00 at column 11
        Char 0x00 at column 12
        Char 0x00 at column 13
        Char 0x00 at column 14
        Char 0x03 at column 15
        Char 0x00 at column 17
        Char 0x00 at column 18
        Char 0x00 at column 19
        Char 0x14 at column 20
        Char 0x00 at column 21
        Char 0x00 at column 22
        Char 0x00 at column 23
        Char 0x10 at column 24
        Char 0x20 at column 30
        Char 0x39 at column 32
        Char 0x2D at column 33
        Char 0x20 at column 40
        Char 0x00 at column 41
        Char 0x00 at column 42
        Char 0x00 at column 43
        Char 0x00 at column 44
 ! Copying using BINARY mode

It may not be useful to list each non-convertible character? The script could stop at the 1st non-convertible character, or after a few non-convertible characters, but not all? (to avoid filling the logs).

Is there a way to prevent the script from testing character conversion and force it to do binary processing? (and for the record to also force it to do a load processing).