Changes for handling org level rulesets

README.md

@@ -5,21 +5,29 @@
 `Safe-settings`– an app to manage policy-as-code and apply repository settings to repositories across an organization.
 
 1. In `safe-settings` all the settings are stored centrally in an `admin` repo within the organization. This is important. Unlike [Settings Probot](https://github.com/probot/settings), the settings files cannot be in individual repositories.  
-    > **Note** It is possible to override this behavior and specify a custom repo instead of the `admin` repo.<br>
+    > **Note**
+    > It is possible to override this behavior and specify a custom repo instead of the `admin` repo.<br>
     > This could be done by setting an `env` variable called `ADMIN_REPO`.
 
-1. There are 3 levels at which the settings could be managed:
+1. In `safe-settings` the settings can have 2 types of targets:
+   1. `org` - These settings are applied to the `org`. `Org`-targeted settings are defined in `.github/settings.yml` . Currently, only `rulesets` are supported as `org`-targeted settings.
+   2. `repo` - These settings are applied to `repos`
+
+2. For The `repo`-targeted settings there can be at 3 levels at which the settings could be managed:
    1. Org-level settings are defined in `.github/settings.yml`  
-       > **Note** It is possible to override this behavior and specify a different filename for the `settings` yml repo.<br>
+       > **Note**
+       > It is possible to override this behavior and specify a different filename for the `settings` yml repo.<br>
        > This could be done by setting an `env` variable called `SETTINGS_FILE_PATH`.
 
-   1. `Suborg` level settings. A `suborg` is an arbitrary collection of repos belonging to projects, business units, or teams. The `suborg` settings reside in a yaml file for each `suborg` in the `.github/suborgs` folder.
-   1. `Repo` level settings. They reside in a repo specific yaml in `.github/repos` folder
-1. It is recommended to break the settings into org-level, suborg-level, and repo-level units. This will allow different teams to define and manage policies for their specific projects or business units. With `CODEOWNERS`, this will allow different people to be responsible for approving changes in different projects.
+   2. `Suborg` level settings. A `suborg` is an arbitrary collection of repos belonging to projects, business units, or teams. The `suborg` settings reside in a yaml file for each `suborg` in the `.github/suborgs` folder.
+   3. `Repo` level settings. They reside in a repo specific yaml in `.github/repos` folder
+3. It is recommended to break the settings into org-level, suborg-level, and repo-level units. This will allow different teams to define and manage policies for their specific projects or business units. With `CODEOWNERS`, this will allow different people to be responsible for approving changes in different projects.
 
-> **Note** `Suborg` and `Repo` level settings directory structure cannot be customized.
+> **Note**
+> `Suborg` and `Repo` level settings directory structure cannot be customized.
 
-> **Note** The settings file must have a `.yml` extension only. `.yaml` extension is ignored, for now.
+> **Note**
+> The settings file must have a `.yml` extension only. `.yaml` extension is ignored, for now.
 
 ## How it works
 
@@ -36,6 +44,10 @@ The App listens to the following webhook events:
 
 - **pull_request.opened**, **pull_request.reopened**, **check_suite.requested**: If the settings are changed, but it is not in the `default` branch, and there is an existing PR, the code will validate the settings changes by running safe-settings in `nop` mode and update the PR with the `dry-run` status. 
 
+- **repository_ruleset**: If the `ruleset` settings are modified in the UI manually, `safe-settings` will `sync` the settings to prevent any unauthorized changes.
+
+- **member_change_events**: If a member is added or removed from a repository, `safe-settings` will `sync` the settings to prevent any unauthorized changes.
+
 ### Restricting `safe-settings` to specific repos
 `safe-settings` can be turned on only to a subset of repos by specifying them in the runtime settings file, `deployment-settings.yml`.  
 If no file is specified, then the following repositories -  `'admin', '.github', 'safe-settings'` are exempted by default.  
@@ -45,7 +57,8 @@ To apply `safe-settings` __only__ to a specific list of repos, add them to the `
 
 To ignore `safe-settings` for a specific list of repos, add them to the `restrictedRepos` section as `exclude` array.
 
-> **Note** The `include` and `exclude` attributes support as well regular expressions.
+> **Note**
+> The `include` and `exclude` attributes support as well regular expressions.
 
 ### Custom rules
 
@@ -210,11 +223,36 @@ The App can be configured to apply the settings on a schedule. This could be a w
 To periodically converge the settings to the configuration, set the `CRON` environment variable. This is based on [node-cron](https://www.npmjs.com/package/node-cron) and details on the possible values can be found [here](#env-variables).
 
 ### Pull Request Workflow
+It is
 `Safe-settings` explicitly looks in the `admin` repo in the organization for the settings files. The `admin` repo could be a restricted repository with `branch protections` and `codeowners`  
 
 In that set up, when changes happen to the settings files and there is a PR for merging the changes back to the `default` branch in the `admin` repo, `safe-settings` will run `checks`  – which will run in **nop** mode and produce a report of the changes that would happen, including the API calls and the payload. 
 
-The checks will fail if `org-level` branch protections are overridden at the repo or suborg level with a lesser number of required approvers.
+For e.g. If we have `override` validators that will fail if `org-level` branch protections are overridden at the repo or suborg level with a lesser number of required approvers, here is an screenshot of what users will see in the PR.
+<p>
+<img width="467" alt="image" src="https://github.com/github/safe-settings/assets/57544838/cc5d59fb-3d7c-477b-99e9-94bcafd07c0b">
+</p>
+
+> **NOTE**
+> If you don't want the PR message to have these details, it can be turned off by `env` setting `CREATE_PR_COMMENT`=`false`
+
+Here is a screenshot of what the users will see in the `checkrun` page:
+<p>
+<img width="462" alt="image" src="https://github.com/github/safe-settings/assets/57544838/c875224f-894b-45da-a9cc-4bfc75c47670">
+</p>
+
+### Error handling
+The app creates a `Check` at the end of its processing to indicate if there were any errors. The `Check` is called `safe-settings` and corrosponds to the latest commit on the `default` branch of the `admin` repo.
+
+Here is an example of a `checkrun` result:
+<p>
+<img width="944" alt="image" src="https://github.com/github/safe-settings/assets/57544838/7ccedcea-628e-4055-a5a5-b8e45123777e">
+</p>
+
+And the `checkrun` page will look like this:
+<p>
+<img width="860" alt="image" src="https://github.com/github/safe-settings/assets/57544838/893ff4e6-904c-4a07-924a-7c23dc068983">
+</p>
 
 ### The Settings file
 

docs/sample-settings/org-ruleset.json

@@ -0,0 +1,115 @@
+{
+    "name": "demo repo ruleset",
+    "target": "branch",
+    "enforcement": "disabled",
+    "bypass_actors": [
+        {
+            "actor_id": 3974045,
+            "actor_type": "Team",
+            "bypass_mode": "pull_request"
+        },
+        {
+            "actor_id": 7898,
+            "actor_type": "RepositoryRole",
+            "bypass_mode": "always"
+        },
+        {
+            "actor_id": 210920,
+            "actor_type": "Integration",
+            "bypass_mode": "always"
+        },
+        {
+            "actor_id": 1,
+            "actor_type": "OrganizationAdmin",
+            "bypass_mode": "always"
+        },
+        {
+            "actor_id": 5,
+            "actor_type": "RepositoryRole",
+            "bypass_mode": "pull_request"
+        }
+    ],
+    "conditions": {
+        "ref_name": {
+            "include": [
+                "~DEFAULT_BRANCH"
+            ],
+            "exclude": [
+                "refs/heads/oldmaster"
+            ]
+        },
+        "repository_name": {
+            "include": [
+                "test*"
+            ],
+            "exclude": [
+                "test",
+                "test1"
+            ],
+            "protected": true
+        }
+    },
+    "rules": [
+        {
+            "type": "creation"
+        },
+        {
+            "type": "update"
+        },
+        {
+            "type": "deletion"
+        },
+        {
+            "type": "required_linear_history"
+        },
+        {
+            "type": "required_signatures"
+        },
+        {
+            "type": "pull_request",
+            "parameters": {
+                "dismiss_stale_reviews_on_push": true,
+                "require_code_owner_review": true,
+                "require_last_push_approval": true,
+                "required_approving_review_count": 10,
+                "required_review_thread_resolution": true
+            }
+        },
+        {
+            "type": "commit_message_pattern",
+            "parameters": {
+                "name": "test commit_message_pattern",
+                "negate": true,
+                "operator": "starts_with",
+                "pattern": "skip*"
+            }
+        },
+        {
+            "type": "commit_author_email_pattern",
+            "parameters": {
+                "name": "test commit_author_email_pattern",
+                "negate": false,
+                "operator": "regex",
+                "pattern": "^.*@example.com$"
+            }
+        },
+        {
+            "type": "committer_email_pattern",
+            "parameters": {
+                "name": "test committer_email_pattern",
+                "negate": false,
+                "operator": "regex",
+                "pattern": "^.*@example.com$"
+            }
+        },
+        {
+            "type": "branch_name_pattern",
+            "parameters": {
+                "name": "test branch_name_pattern",
+                "negate": false,
+                "operator": "regex",
+                "pattern": "^(feature|bugfix|improvement|library|prerelease|release|hotfix)\/[a-z0-9._-]+$"
+            }
+        }
+    ]
+}

docs/sample-settings/org-ruleset.yml

@@ -0,0 +1,153 @@
+rulesets:
+- name: demo
+  # Name of the rule
+  target: branch
+  # The target of the ruleset. Can be one of:
+  #       - branch
+  #        - tag
+  enforcement: disabled
+  # The enforcement level of the ruleset. `evaluate` allows admins to test
+  # rules before enforcing them. 
+  # - disabled
+  # - active
+  # - evaluate
+  bypass_actors: 
+  # The actors that can bypass the rules in this ruleset      
+    - actor_id: 3974045
+      actor_type: Team
+      # type: The type of actor that can bypass a ruleset
+      # - RepositoryRole
+      # - Team
+      # - Integration
+      # - OrganizationAdmin
+      bypass_mode: pull_request
+      #  When the specified actor can bypass the ruleset. `pull_request`
+      #  means that an actor can only bypass rules on pull requests.
+      #  - always
+      #  - pull_request
+
+    - actor_id: 1
+      actor_type: OrganizationAdmin
+      bypass_mode: always
+
+    - actor_id: 7898
+      actor_type: RepositoryRole
+      bypass_mode: always
+
+    - actor_id: 210920
+      actor_type: Integration
+      bypass_mode: always
+
+  conditions:
+      ref_name:
+        # Parameters for a repository ruleset ref name condition
+        include: ["~DEFAULT_BRANCH"]
+        # Array of ref names or patterns to include. One of these
+        # patterns must match for the condition to pass. Also accepts
+        # `~DEFAULT_BRANCH` to include the default branch or `~ALL` to
+        # include all branches.         
+        exclude: ["refs/heads/oldmaster"]
+        # Array of ref names or patterns to exclude. The condition
+        # will not pass if any of these patterns match.
+      repository_name: 
+        include: ["test*"]
+        #                   Array of repository names or patterns to include.
+        #                   One of these patterns must match for the condition
+        #                   to pass. Also accepts `~ALL` to include all
+        #                   repositories.
+        exclude: ["test","test1"]
+        #                   Array of repository names or patterns to exclude. The
+        #                   condition will not pass if any of these patterns
+        #                   match.
+        protected: true
+        # Whether renaming of target repositories is
+        # prevented.
+
+  rules:
+  - type: creation
+  - type: update
+  - type: update_allows_fetch_and_merge
+  - type: deletion
+  - type: required_linear_history
+  - type: required_signatures
+  - type: required_deployments
+    parameters: 
+      required_deployment_environments: ["test"]
+  - type: pull_request
+    parameters: 
+      dismiss_stale_reviews_on_push: true
+      # New, reviewable commits pushed will dismiss previous pull
+      # request review approvals.
+      require_code_owner_review: true
+      # Require an approving review in pull requests that modify
+      # files that have a designated code owner
+      require_last_push_approval: true
+      # Whether the most recent reviewable push must be approved
+      # by someone other than the person who pushed it.
+      required_approving_review_count: 10
+      # The number of approving reviews that are required before a
+      # pull request can be merged.
+      required_review_thread_resolution: true
+      # All conversations on code must be resolved before a pull
+      # request can be merged.
+
+  - type: required_status_checks
+    # Choose which status checks must pass before branches can be merged
+    # into a branch that matches this rule. When enabled, commits must
+    # first be pushed to another branch, then merged or pushed directly
+    # to a branch that matches this rule after status checks have
+    # passed.
+    parameters:
+      strict_required_status_checks_policy: false
+      # Whether pull requests targeting a matching branch must be
+      # tested with the latest code. This setting will not take
+      # effect unless at least one status check is enabled.
+      required_status_checks:
+      - context: CodeQL
+        integration_id: 1234
+      - context: GHAS Compliance
+        integration_id: 1234
+
+  - type: commit_message_pattern
+    parameters:
+      name: test commit_message_pattern
+      # required:
+      #  - operator
+      #  - pattern
+      negate: true
+      operator: starts_with
+      # The operator to use for matching.
+      # - starts_with
+      # - ends_with
+      # - contains
+      # - regex
+      pattern: skip*
+      # The pattern to match with.
+
+  - type: commit_author_email_pattern
+    parameters:
+      name: test commit_author_email_pattern
+      negate: false
+      operator: regex
+      pattern: "^.*@example.com$"
+
+  - type: committer_email_pattern
+    parameters:
+      name: test committer_email_pattern
+      negate: false
+      operator: regex
+      pattern: "^.*@example.com$"
+
+  - type: branch_name_pattern
+    parameters:
+      name: test branch_name_pattern
+      negate: false
+      operator: regex
+      pattern: ".*\/.*"
+
+  - type: "tag_name_pattern"
+    parameters:
+      name: test tag_name_pattern
+      negate: false
+      operator: regex
+      pattern: ".*\/.*"