GH-33241: [Archery] Replace github3 with pygithub by fangchenli · Pull Request #48886 · apache/arrow

fangchenli · 2026-01-19T08:16:35Z

Rationale for this change

Archery currently uses both pygithub and github3. It's unnecessary and increases maintenance burden.

What changes are included in this PR?

Replace all github3 usage with pygithub.

Are these changes tested?

Yes, unittests added.

Are there any user-facing changes?

No.

GitHub Issue: [Archery][CI] Refactor git dependencies used on archery to be more consistent #33241

github-actions · 2026-01-19T08:17:00Z

⚠️ GitHub issue #33241 has been automatically assigned in GitHub to PR creator.

github-actions · 2026-01-19T22:13:37Z

⚠️ GitHub issue #33241 has been automatically assigned in GitHub to PR creator.

…igration

raulcd

Thanks for the PR, awesome work. The changes look great in general. I'll have to take some time to do some validation as crossbow isn't extensively tested at the moment. Can you remove this one while I find some time to test it?

arrow/ci/conda_env_archery.txt

Line 22 in 894d6a3

github3.py

raulcd · 2026-01-20T14:06:17Z

Failures are unrelated, will be fixed once the following issue closed. It has a PR ready to be merged:

[Python] Conda jobs segfault on Azure tests due to old nodejs version being installed #48894

…igration

raulcd

Changes look good to me. Thanks @fangchenli
@kou I plan to merge this and if we find any small issue not covered by tests during crossbow runs we can fix them.
I haven't validated manually the different cases but have reviewed the changes.

github-actions · 2026-02-25T01:40:02Z

⚠️ GitHub issue #33241 has been automatically assigned in GitHub to PR creator.

kou

+1

It makes sense.

@fangchenli BTW, I'm not sure why you worked on this but are you interested in GitHub API related issue? We want to complete #47340 but we need to update our GitHub API usage too. (Changing Type: XXX issue label to XXX issue type in our development scripts.) Do you want to work on it?

…igration

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR migrates Crossbow’s GitHub integration from github3.py to PyGithub, updating dependency declarations and adapting the core/CLI codepaths plus tests around releases, PRs, and asset uploads.

Changes:

Replace github3.py usage with PyGithub in Crossbow core logic and CLI flows.
Update packaging/conda environment dependencies and remove the crossbow-upload extra.
Add/expand unit tests covering the new PyGithub-based behaviors (releases, PRs, assets, statuses).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
dev/archery/setup.py	Switch Crossbow extra dependency from `github3.py` to `pygithub` and remove `crossbow-upload` extra.
dev/archery/archery/crossbow/core.py	Port GitHub API interactions (login, releases, PRs, assets, statuses) from github3.py to PyGithub.
dev/archery/archery/crossbow/cli.py	Update asset download and upload CLI commands to use PyGithub APIs; remove upload method option.
dev/archery/archery/crossbow/tests/test_core.py	Add tests for new PyGithub-based Repo/TaskStatus/TaskAssets behaviors.
dev/archery/README.md	Remove docs for the now-removed `crossbow-upload` extra.
ci/conda_env_crossbow.txt	Replace `github3.py` with `pygithub` in Crossbow conda env.
ci/conda_env_archery.txt	Remove `github3.py` from Archery conda env.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+            release = repo.get_release(tag_name)
+            release.delete_release()
+        except GithubException as e:
+            if e.status != 404:
+                raise
+
+        release = repo.create_git_release(tag_name, tag_name, "",
+                                          target_commitish=target_commitish)


+            except IOError as e:
+                # Catch network and file I/O errors (includes requests exceptions)
                logger.error(f"Attempt {i + 1} has failed with message: {e}.")


+    with mock.patch.object(Repo, 'as_github_repo') as mock_repo:
+        with mock.patch.object(Repo, 'default_branch_name', 'main'):
+            mock_pr = mock.MagicMock()


+    'crossbow': ['pygithub', jinja_req, 'pygit2>=1.14.0', 'requests',
                 'ruamel.yaml', 'setuptools_scm>=8.0.0'],


Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

        # remove the whole release if it already exists
        try:
-            release = repo.release_from_tag(tag_name)
-        except github3.exceptions.NotFoundError:
-            pass
-        else:
-            release.delete()
-
-        release = repo.create_release(tag_name, target_commitish)
+            release = repo.get_release(tag_name)
+            release.delete_release()
+        except GithubException as e:
+            if e.status != 404:
+                raise


+            except IOError as e:
+                # Catch network and file I/O errors (includes requests exceptions)
                logger.error(f"Attempt {i + 1} has failed with message: {e}.")


pitrou · 2026-06-11T09:15:08Z

@kszucs Would you like to review this?

kszucs · 2026-06-11T09:47:56Z

Back then github3.py had wider feature support and better maintenance though some features were missing hence the pygithub dependency as well (at least if I recall correctly, it was a while ago / there could have been missing packages as well). Apparently pygithub is more actively maintained nowadays so it is indeed better to keep only pygithub.

kszucs · 2026-06-11T10:16:35Z

Overall it looks good to me.

…igration

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

    def github_release(self, tag):
        repo = self.as_github_repo()
        try:
-            return repo.release_from_tag(tag)
-        except github3.exceptions.NotFoundError:
-            return None
-
-    def github_upload_asset_requests(self, release, path, name, mime,
-                                     max_retries=None, retry_backoff=None):
+            return repo.get_release(tag)
+        except GithubException as e:
+            if e.status == 404:
+                return None
+            raise


@@ -560,13 +534,14 @@ def github_overwrite_release_assets(self, tag_name, target_commitish,

        # remove the whole release if it already exists
        try:
-            release = repo.release_from_tag(tag_name)
-        except github3.exceptions.NotFoundError:
-            pass
-        else:
-            release.delete()
-
-        release = repo.create_release(tag_name, target_commitish)
+            release = repo.get_release(tag_name)
+            release.delete_release()
+        except GithubException as e:
+            if e.status != 404:
+                raise


+            except IOError as e:
+                # Catch network and file I/O errors (includes requests exceptions)
                logger.error(f"Attempt {i + 1} has failed with message: {e}.")


pitrou · 2026-06-15T14:18:37Z

@github-actions crossbow submit wheelcp313*

github-actions · 2026-06-15T14:21:09Z

Revision: b7ab6e7

Submitted crossbow builds: ursacomputing/crossbow @ actions-9dd6cd6454

Task	Status
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-musllinux-1-2-cp313-cp313-amd64
wheel-musllinux-1-2-cp313-cp313-arm64
wheel-windows-cp313-cp313-amd64

pitrou

Just two nits, I'll trust @raulcd and @kszucs otherwise

pitrou · 2026-06-15T14:19:01Z


 click
-github3.py
+pygithub


Can we keep this list alphabetically-ordered?

pitrou · 2026-06-15T14:22:19Z

 extras = {
    'benchmark': ['pandas'],
-    'crossbow': ['github3.py', jinja_req, 'pygit2>=1.14.0', 'requests',
+    'crossbow': ['pygithub>=2.5.0', jinja_req, 'pygit2>=1.14.0', 'requests',


Same nit: alphabetical order

…igration

raulcd

Thanks @fangchenli for the work here and the effort of keeping it alive. I am going to merge this. It will be tested by some CI jobs and thoroughly tested during the release (download and upload artifacts steps)

conbench-apache-arrow · 2026-06-19T17:36:42Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 98ee71e.

There were 6 benchmark results indicating a performance regression:

Commit Run on test-mac-arm at 2026-06-19 12:48:38Z
- tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-13, scale_factor=1
- tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-13, scale_factor=1
and 4 more (see the report linked below)

The full Conbench report has more details. It also includes information about 556 possible false positives for unstable benchmarks that are known to sometimes produce them.

apacheGH-33241:[Archery] Replace github3 with pygithub

91b080e

github-actions Bot added the awaiting review Awaiting review label Jan 19, 2026

fangchenli marked this pull request as ready for review January 19, 2026 22:13

fangchenli requested review from assignUser, jonkeane, kou and raulcd as code owners January 19, 2026 22:13

Merge remote-tracking branch 'upstream/main' into crossbow-pygithub-m…

ff87c22

…igration

raulcd reviewed Jan 20, 2026

View reviewed changes

Comment thread dev/archery/setup.py Outdated

clean up

654466c

github-actions Bot added the Component: Documentation label Jan 20, 2026

fangchenli added 2 commits January 20, 2026 15:18

Merge remote-tracking branch 'upstream/main' into crossbow-pygithub-m…

f89b278

…igration

Merge remote-tracking branch 'upstream/main' into crossbow-pygithub-m…

2db0368

…igration

fangchenli requested a review from raulcd February 21, 2026 00:43

raulcd approved these changes Feb 24, 2026

View reviewed changes

kou approved these changes Feb 25, 2026

View reviewed changes

Comment thread dev/archery/archery/crossbow/core.py Outdated

Comment thread dev/archery/archery/crossbow/core.py

fangchenli added 4 commits February 27, 2026 08:30

Merge remote-tracking branch 'upstream/main' into crossbow-pygithub-m…

66ea747

…igration

Merge remote-tracking branch 'upstream/main' into crossbow-pygithub-m…

05d4baa

…igration

Merge remote-tracking branch 'upstream/main' into crossbow-pygithub-m…

b73da15

…igration

remove curl

6dad897

Copilot AI review requested due to automatic review settings June 11, 2026 05:29

fangchenli requested a review from pitrou as a code owner June 11, 2026 05:29

Copilot AI reviewed Jun 11, 2026

View reviewed changes

fangchenli added 2 commits June 10, 2026 22:37

address copilot comments

5dd3298

remove access to pygithub's private requester

bf44e72

Copilot AI review requested due to automatic review settings June 11, 2026 06:02

Copilot AI reviewed Jun 11, 2026

View reviewed changes

fangchenli requested a review from kou June 11, 2026 06:07

fangchenli added 3 commits June 11, 2026 09:43

add a test

8f45e09

Merge remote-tracking branch 'upstream/main' into crossbow-pygithub-m…

f18c3a8

…igration

Merge remote-tracking branch 'upstream/main' into crossbow-pygithub-m…

b7ab6e7

…igration

Copilot AI review requested due to automatic review settings June 12, 2026 09:44

Copilot AI reviewed Jun 12, 2026

View reviewed changes

fangchenli requested a review from kszucs June 12, 2026 10:31

pitrou reviewed Jun 15, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Jun 15, 2026

fangchenli added 2 commits June 18, 2026 11:29

Merge remote-tracking branch 'upstream/main' into crossbow-pygithub-m…

374ce23

…igration

reorder deps

19cb320

raulcd approved these changes Jun 19, 2026

View reviewed changes

github-actions Bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Jun 19, 2026

raulcd merged commit 98ee71e into apache:main Jun 19, 2026
44 checks passed

raulcd removed the awaiting merge Awaiting merge label Jun 19, 2026

raulcd mentioned this pull request Jun 19, 2026

[Archery][CI] Refactor git dependencies used on archery to be more consistent #33241

Closed

		'crossbow': ['pygithub', jinja_req, 'pygit2>=1.14.0', 'requests',
		'ruamel.yaml', 'setuptools_scm>=8.0.0'],

Conversation

fangchenli commented Jan 19, 2026 • edited by kou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jan 19, 2026

Uh oh!

github-actions Bot commented Jan 19, 2026

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raulcd commented Jan 20, 2026

Uh oh!

raulcd left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 25, 2026

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

pitrou commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kszucs commented Jun 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

pitrou commented Jun 15, 2026

Uh oh!

github-actions Bot commented Jun 15, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

pitrou Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

pitrou Jun 15, 2026

Choose a reason for hiding this comment

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

conbench-apache-arrow Bot commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

fangchenli commented Jan 19, 2026 •

edited by kou

Loading

raulcd left a comment •

edited

Loading

pitrou commented Jun 11, 2026 •

edited

Loading

kszucs commented Jun 11, 2026 •

edited

Loading