WIP: further flesh out development process and development guide

matyasselmeci · matyasselmeci · commit b2da71ed3f03 · 2025-09-08T21:46:59.000-05:00
diff --git a/docs/software/development-process.md b/docs/software/development-process.md
@@ -13,7 +13,7 @@ Overall Development Cycle
 For a typical update to an existing package, the overall development cycle is roughly as follows:
 
 1.  Download the new upstream source (tarball, source RPM, checkout) into
-    [the UW AFS upstream area](../software/rpm-development-guide.md#upstream-source-cache)
+    [the CHTC upstream area](../software/rpm-development-guide.md#upstream-source-cache)
 2.  In [a checkout of our packaging code](../software/rpm-development-guide.md#revision-control-system),
     update [the reference to the upstream file](../software/rpm-development-guide.md#upstream) and,
     as needed, [the RPM spec file](../software/rpm-development-guide.md#osg)
@@ -22,25 +22,10 @@ For a typical update to an existing package, the overall development cycle is ro
 5.  Optionally, lightly test the new RPM(s); if there are problems, redo previous steps until success
 6.  Use [osg-build](../software/osg-build-tools.md#osg-build) to perform an official build of the updated package
     (which will go into the development repos)
-7.  Perform standard developer testing of the new RPM(s) — see below for details
-8.  Obtain permission from the Software Manager to promote the package
+7.  Perform standard developer testing of the new RPMs; this generally means running the VM Universe tests - see below for details
+8.  Have another software team member review your testing and give you permission to promote the package
 9.  Promote the package to testing — see below for details
 
-Versioning Guidelines
----------------------
-
-OSG-owned software should contain three digits, X.Y.Z, where X represents the major version, Y the minor version,
-and Z the maintenance version.
-New releases of software should increment one of the major, minor, or maintenance according to the following guidelines:
-
--   **Major:** Major new software, typically (but not limited to) full rewrites, new architectures, major new features;
-    can certainly break backward compatibility (but should provide a smooth upgrade path). Worthy of introduction into Upcoming.
--   **Minor:** Notable changes to the software, including significant feature changes, API changes, etc.;
-    may break compatibility, but must provide an upgrade path from other versions within the same Major series.
--   **Maintenance:** Bug fixes, minor feature tweaks, etc.;
-    must not break compatibility with other versions within the same Major.Minor series.
-
-If you are unsure about which version number to increment in a software update, consult the Software Manager.
 
 Build Procedures
 ----------------
@@ -49,98 +34,100 @@ Build Procedures
 ### Verifying builds through GitHub Actions
 -->
 
-### Building packages for multiple OSG release series
+### Initial repo setup
 
-The OSG Software team supports multiple release series, independent but in parallel to a large degree.
-In many cases, a single package is the same across release series, and therefore we want to build the package once
-and share it among the series.
-The procedure below suggests a way to accomplish this task.
+1.  Create your own fork of the <https://github.com/osg-htc/software-packaging> repository.
+2.  Clone your fork, then add the upstream repository as a remote (replacing `<YOURNAME>` with your GitHub username):
 
-Current definitions:
+        :::console
+        $ git clone https://github.com/<YOURNAME>/software-packaging
+        $ git remote add upstream https://github.com/osg-htc/software-packaging
 
-- maintenance: OSG 3.4 ( **trunk** )
-- current: OSG 3.5 ( **branches/osg-3.5** )
 
-<!--
-- future: OSG 3.6 ( *branches/osg-3.6* )
--->
+### Basic building
 
-Procedure:
-
-
-1.  Make changes to **trunk**
-2.  Optionally, make and test a scratch build from **trunk**
-3.  Commit the changes
-4.  Make an official build from **trunk** (e.g.: `osg-build koji <PACKAGE>`)
-5.  Perform the standard 4 tests for the **current** series (see below)
-6.  Merge the relevant commits from **trunk** into the **maintenance** branch (see below for tips)
-7.  Optionally, make and test a scratch build from the **maintenance** branch
-8.  Commit the merge
-9.  Make an official build from the **maintenance** branch (e.g.: `osg-build koji --repo=3.4 <PACKAGE>`)
-10. Perform the standard 4 tests for the **maintenance** series (see below)
-11. As needed (or directed by the Software manager), perform the cross-series tests (see below)
-
-!!! note
-    Do not change the RPM Release number in the **maintenance** branch before rebuilding;
-    the `%dist` tag will differ automatically, and hence the **maintenance** and **current** NVRs will not conflict.
-
-<!--
-1. Make changes to the *trunk*
-2. Optionally, make and test a scratch build from the *trunk*
-3. Commit the changes
-4. Make an official build from the *trunk* (e.g.: `osg-build koji <PACKAGE>`)
-5. Perform the standard 4 tests for the *current* series (see below)
-6. Merge the relevant commits from the *trunk* into the *future* branch (see below for tips)
-7. Optionally, make and test a scratch build from the *future* branch
-8. Commit the merge
-9. Make an official build from the *future* branch (e.g.: `osg-build koji --repo=3.6 <PACKAGE>`)
-10. Perform install tests for the *future* series (see below)
-11. As needed (or directed by the Software manager), perform the cross-series tests (see below)
--->
+<!-- TODO flesh this out -->
+
+1.  Make changes
+2.  Make a scratch build
+3.  If you have permissions, push directly upstream
+
+        :::console
+        $ git push upstream main
 
-### Merging changes from one release series to another
+    If you do not have permissions, or want your changes reviewed,
+    push to a branch on your own fork and make a Pull Request
+4.  Make a non-scratch build
 
-These instructions assume that you are merging from `trunk` to `branches/osg-3.5`.
-They also assume that the current directory you are in is a checkout of `branches/osg-3.5`.
-I will use `$pkg` to refer to the name of your package.
 
-First, you will need the commit numbers for your changes:
+### Building packages for multiple OSG release series
+
+The OSG Software team supports two release series in parallel;
+many of the packages are identical or very similar between release series,
+and you will be making the same change to a package across both release series.
+
+We recommend using a diffing tool such as Meld (Linux) or WinMerge (Windows) that is capable of comparing
+directories and not just individual files, to make sure all the changes are carried over.
+
+This is one example workflow, using the "23-main" and "24-main" release series,
+and the package "xrdcl-pelican" which is identical between the two release series.
 
-    svn log \^/native/redhat/trunk/$pkg | less
+1.  Make changes to `24-main/xrdcl-pelican` 
+2.  Make a scratch build (e.g. `osg-build koji --scratch 24-main/xrdcl-pelican`)
+3.  Open `23-main/xrdcl-pelican` and `24-main/xrdcl-pelican` in a diffing tool, and copy
+    over all the changes
+4.  `git add` and `git commit` the changes; committing the changes to both series in the same commit
+    means you don't have to write the commit message twice,
+    and `git blame` will be accurate for both release series
+5.  `git push` to the upstream repo
+6.  Make non-scratch builds; use the following procedure to submit both at the same time:
 
-Write down the commits you want to merge.
+        :::console
+        $ osg-build koji --nowait 24-main/xrdcl-pelican
+        $ osg-build koji --nowait 23-main/xrdcl-pelican
+        $ osg-koji watch-task --mine
 
-If you only have one commit, merge that commit with -c as follows:
 
-    svn merge -c $commit_num \^/native/redhat/trunk/$pkg $pkg
+If your development process is more iterative and you need multiple commits,
+you could do something like the following (using `xrootd` as an example):
 
-Where `$commit_num` is the SVN revision number of that commit (e.g. 17000).
-Merging an individual change like this is referred to as "cherry-picking".
+1.  Make changes to `24-main/xrootd`
+2.  Make a scratch build as above
+3.  Commit your changes as a "checkpoint"
+4.  Repeat 1-3 as necessary until you think you are ready for a non-scratch build
+5.  Copy your changes to `23-main` using a diffing tool as above
+6.  You now have two options:
 
-If you have a range of commits and you wish to merge all commits within that range, then do the following:
+    -   If you haven't pushed, then you can do an interactive rebase to squash your changes to
+        `23-main` and `24-main` down to one commit
 
-    svn merge -r $start_num:$end_num \^/native/redhat/trunk/$pkg $pkg
+    -   If you have already pushed, get a log of the recent changes to `24-main/xrootd`:
 
-Where `$start_num` is the SVN revision of the commit *BEFORE* your first commit,
-and `$end_num` is the SVN revision of your last commit in that range.
-**Note:** Be very careful when merging a range from trunk into the maintenance branch
-so that you do not introduce more changes to the maintenance branch than are necessary.
+            :::console
+            $ git --no-pager log --oneline --since='last week' --reverse -- 24-main/xrootd
 
-If you have multiple commits but they are not contiguous (i.e. there are commits made by you or someone else in that range
-that you do not want to merge), you will need to cherry-pick each individual commit.
+        Copy and paste that to a text editor.
+        Then, when you write the commit message for the commit to `23-main/xrootd`,
+        reference the commits from that command.
+        For example:
 
-    svn merge -c $commit1 \^/native/redhat/trunk/$pkg $pkg
-    svn merge -c $commit2 \^/native/redhat/trunk/$pkg $pkg
-    ...
+            23-main/xrootd: Update xrootd to 5.8.4 and add various patches
 
-Where `$commit1`, `$commit2` are the commit numbers of the individual changes.
+            Includes the following commits from 24-main/xrootd:
+
+            8061d7d40 24-main/xrootd: update to 5.8.4; update patches, bring spec file closer to upstream
+            13b858dce 24-main/xrootd: add some more patches
+            773f57f21 24-main/xrootd: Add TPC worker pool patch
+
+        Keeping the Git hashes will make future archaeology easier.
 
-Note that merge tracking in recent versions of SVN (1.5 or newer) should prevent commits from accidentally being merged multiple times.
-You should still look out for conflicts and examine the changes via `svn diff` before committing the merge.
 
 Testing Procedures
 ------------------
 
+!!!note
+    This section is out of date
+
 Before promoting a package to a testing repository, each build must be tested lightly from the development repos
 to make sure that it is not completely broken, thereby wasting time during acceptance testing.
 Normally, the person who builds a package performs the development testing.
@@ -222,6 +209,9 @@ essentially restarting the development cycle.
 
 ### Preparing a Good Promotion Request
 
+!!!note
+    This section is out of date
+
 Developers must obtain permission from the OSG Software manager to promote a package from development to testing.
 A promotion request goes into at least one affected JIRA ticket and will be answered there as well.
 Below are some tips for writing a good promotion request:
diff --git a/docs/software/koji-workflow.md b/docs/software/koji-workflow.md
@@ -65,7 +65,7 @@ to OSG build services.
 Include the following information:
 
 -   your GitHub username
--   your Kerberos principal if you have one;
+-   your Kerberos principal if you already have one;
     for people with a UW NetID account, this is `<netid>@AD.WISC.EDU` and
     for people with a Fermilab account, this is `<username>@FNAL.GOV`
 
diff --git a/docs/software/rpm-development-guide.md b/docs/software/rpm-development-guide.md
@@ -100,7 +100,8 @@ For example, the package directory for `xrootd` for the `osg-24-main-*` tags are
 `24-main/xrootd`
 
 The subtree must correspond to the repos being built to.
-For example, you may not build into `osg-24-main-*` from the `23-upcoming` subtree.
+For example, you may only build into `osg-24-main-*` from the `24-main` subtree.
+(This restriction does not apply to scratch builds.)
 
 
 #### Package Directory Organization
