feat: enable multiple ABM and VPP tokens (#21693)

> Related issue: #9956 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Manual QA for all new/changed functionality

---------

Co-authored-by: Martin Angers <[email protected]>
Co-authored-by: Gabriel Hernandez <[email protected]>
Co-authored-by: Roberto Dip <[email protected]>
Co-authored-by: Sarah Gillespie <[email protected]>
Co-authored-by: Dante Catalfamo <[email protected]>
Co-authored-by: Roberto Dip <[email protected]>

articles/install-vpp-apps-on-macos-using-fleet.md

@@ -16,15 +16,17 @@ By integrating VPP with Fleet, organizations can seamlessly add apps to their so
 
 ## Prerequisites
 * **MDM features**: to use the VPP integration, you must first enable MDM features in Fleet. See the [MDM setup guide](https://fleetdm.com/docs/using-fleet/mdm-setup) for instructions on enabling MDM features.
-* **Teams**: Apps can only be added to a specific Team. You can manage teams by selecting your avatar in the top navigation and then **Settings > Teams**. (Note: Apps can also be added to the 'No Team' team, which contains hosts not assigned to any other team.)
+* **Teams**: Apps can only be added to a specific Team. You can manage teams by selecting your avatar in the top navigation and then **Settings > Teams**. (Note: Apps can also be added to the 'No Team' team, which contains hosts not assigned to any other team.) You can control which team uses which VPP token by assigning teams to the VPP token. Each token may have multiple teams assigned to it, but each team may be assigned to only 1 token.
 
 > As of Fleet 4.55.0, there is a [known issue](https://github.com/fleetdm/fleet/issues/20686) that uninstalled or deleted VPP apps will continue to show a status of `installed`.
 
 ## Accessing the VPP configuration
 
-1. **Navigate to the VPP integration settings page**: Click your avatar on the far right of the main navigation menu, and then **Settings > Integrations > "Volume Purchasing Program (VPP)."**
+1. **Navigate to the MDM integration settings page**: Click your avatar on the far right of the main navigation menu, and then **Settings > Integrations > "Mobile device management (MDM)"**
 
-2. **Add your VPP token**: Follow the directions on that page to get your VPP token from Apple Business Manager, and then click the "Upload" button at the bottom to upload it to Fleet.
+2. **Add your VPP token**: Scroll to the "Volume Purchasing Program (VPP)" section. Click "Add VPP", and then click "Add VPP" again on the following page. Follow the directions on the modal to get your VPP token from Apple Business Manager, and then click the "Upload" button at the bottom to upload it to Fleet.
+
+3. **Edit the team assignment for the new token**: Find the token in the table of VPP tokens. Click the "Actions" dropdown, and then click "Edit teams". Use the picker to select which team(s) this VPP token should be assigned to.
 
 ## Purchasing apps
 
@@ -76,23 +78,28 @@ To add apps to Fleet, you must first purchase them through Apple Business Manage
 
 ## Renewing an expired or expiring VPP token
 
-When your uploaded VPP token has expired or is within 30 days of expiring, you will see a warning
+When one of your uploaded VPP tokens has expired or is within 30 days of expiring, you will see a warning
 banner at the top of page reminding you to renew your token. You can do this with the following steps:
 
-1. **Navigate to the VPP integration details page**: Click your avatar on the far right of the main
-   navigation menu, and then **Settings > Integrations > "Volume Purchasing Program (VPP)."** Then
-   click on the **Edit** button to go to the VPP integration details page.
+1. **Navigate to the MDM integration settings page**: Click your avatar on the far right of the main navigation menu, and then **Settings > Integrations > "Mobile device management (MDM)"** Scroll to the "Volume Purchasing Program (VPP)" section, and click "Edit".
+
+2. **Renew the token**: Find the VPP token that you want to renew in the table. Token status is indicated in the "Renew date" column: tokens less than 30 days from expiring will have a yellow indicator, and expired tokens will have a red indicator. Click the "Actions" dropdown for the token and then click "Renew". Follow the instructions in the modal to download a new token from Apple Business Manager and then upload the new token to Fleet.
+
+## Deleting a VPP token
+
+To remove VPP tokens from Fleet:
+
+1. **Navigate to the MDM integration settings page**: Click your avatar on the far right of the main navigation menu, and then **Settings > Integrations > "Mobile device management (MDM)"** Scroll to the "Volume Purchasing Program (VPP)" section, and click "Edit". 
 
-2. **Upload a new VPP token:** Click on the **Renew token** button and follow the instructions to
-   upload a new .vpptoken file. Click the **Renew token** button when you have selected the new token.
+2. **Delete the token**: Find the VPP token that you want to delete in the table. Click the "Actions" dropdown for that token, and then click "Delete". Click "Delete" in the confirmation modal to finish deleting the token.
 
 ## Managing apps with GitOps
 
 To manage App Store apps using Fleet's best practice GitOps, check out the `software` key in the GitOps reference documentation [here](https://fleetdm.com/docs/using-fleet/gitops#software).
 
 ## REST API
 
-Fleet also provides a REST API for managing apps programmatically. You can add, install, and delete apps via this API and manage your organization’s VPP token. Learn more about Fleet's [REST API](https://fleetdm.com/docs/rest-api/rest-api).
+Fleet also provides a REST API for managing apps programmatically. You can add, install, and delete apps via this API and manage your organization’s VPP tokens. Learn more about Fleet's [REST API](https://fleetdm.com/docs/rest-api/rest-api).
 
 ## Conclusion
 

articles/macos-mdm-setup.md

@@ -20,7 +20,18 @@ To connect Fleet to APNs or renew APNs, head to the **Settings > Integrations >
 
 > Available in Fleet Premium
 
-To connect Fleet to ABM or renew ABM, head to the **Settings > Integrations > Automatic enrollment > Apple Business Manager** page.
+To connect Fleet to ABM, you have to add an ABM token to Fleet. To add an ABM token: 
+
+1. Navigate to the **Settings > Integrations > Mobile device management (MDM)** page.
+2. Under "Automatic enrollment", click "Add ABM", and then click "Add ABM" again on the next page. Follow the instructions in the modal and upload an ABM token to Fleet.
+
+When one of your uploaded ABM tokens has expired or is within 30 days of expiring, you will see a warning
+banner at the top of page reminding you to renew your token.
+
+To renew an ABM token:
+
+1. Navigate to the **Settings > Integrations > Mobile device management (MDM)** page.
+2. Under "Automatic enrollment", click "Edit", and then fin
 
 After connecting Fleet to ABM, set Fleet to be the MDM for all Macs: 
 
@@ -32,7 +43,13 @@ After connecting Fleet to ABM, set Fleet to be the MDM for all Macs:
 
 macOS, iOS, and iPadOS hosts listed in ABM and associated to a Fleet instance with MDM enabled will sync to Fleet and appear in the Hosts view with the **MDM status** label set to "Pending".
 
-macOS hosts that automatically enroll will be assigned to a default team. If no default team is set, then the host will be placed in "No team". 
+Hosts that automatically enroll will be assigned to a default team. You can configure the default team for macOS, iOS, and iPadOS hosts by:
+
+1. Navigating to the **Settings > Integrations > Mobile device management (MDM)** page and clicking "Edit" under "Automatic enrollment".
+2. Click on the "Actions" dropdown for the ABM token you want to update, and then click "Edit teams".
+3. Use the dropdowns in the modal to select the default team for each type of host, and click "Save" to save your selections.
+
+If no default team is set for a host platform (macOS, iOS, or iPadOS), then newly enrolled hosts of that platform will be placed in "No team". 
 
 > A host can be transferred to a new (not default) team before it enrolls. In the Fleet UI, you can do this under **Settings** > **Teams**.
 

changes/21177-abm-crud

@@ -0,0 +1 @@
+- Adds new endpoints and updates existing endpoints for managing multiple Apple Business Manager tokens.

changes/21178-mabm-vpp-crud

@@ -0,0 +1 @@
+- Add backend support for multiple VPP tokens

changes/21184-consolidate-all-mdm-settings-under-mdm-section

@@ -0,0 +1 @@
+- consolidates Automatic Enrollment and VPP settings under the MDM settings integration page.

changes/21185-mabm-guide-updates

@@ -0,0 +1 @@
+- Updated relevant documentation to include references to multiple ABM and VPP tokens.

changes/21186-new-abm-ui-page

@@ -0,0 +1 @@
+- add new Apple business manager page to fleet UI

changes/21187-new-vpp-page

@@ -0,0 +1 @@
+- add new vpp page to the fleet UI

changes/21273-handle-abm-terms-expired-flags

@@ -0,0 +1 @@
+* Added support to track the Apple Business Manager "terms expired" API error per token, as well as a global flag that gets set as soon as one token has its terms expired.

changes/21439-multiple-teams-vpp-token

@@ -0,0 +1 @@
+- Allow multiple teams to be assigned to the same VPP Token

cmd/fleet/cron.go

@@ -683,7 +683,11 @@ func newWorkerIntegrationsSchedule(
 		Log:       logger,
 		Commander: commander,
 	}
-	w.Register(jira, zendesk, macosSetupAsst, appleMDM)
+	dbMigrate := &worker.DBMigration{
+		Datastore: ds,
+		Log:       logger,
+	}
+	w.Register(jira, zendesk, macosSetupAsst, appleMDM, dbMigrate)
 
 	// Read app config a first time before starting, to clear up any failer client
 	// configuration if we're not on a fleet-owned server. Technically, the ServerURL
@@ -1121,29 +1125,57 @@ func newAppleMDMDEPProfileAssigner(
 ) (*schedule.Schedule, error) {
 	const name = string(fleet.CronAppleMDMDEPProfileAssigner)
 	logger = kitlog.With(logger, "cron", name, "component", "nanodep-syncer")
-	var fleetSyncer *apple_mdm.DEPService
 	s := schedule.New(
 		ctx, name, instanceID, periodicity, ds, ds,
 		schedule.WithLogger(logger),
-		schedule.WithJob("dep_syncer", func(ctx context.Context) error {
-			appCfg, err := ds.AppConfig(ctx)
-			if err != nil {
-				return ctxerr.Wrap(ctx, err, "retrieving app config")
-			}
+		schedule.WithJob("dep_syncer", appleMDMDEPSyncerJob(ds, depStorage, logger)),
+	)
 
-			if !appCfg.MDM.AppleBMEnabledAndConfigured {
-				return nil
-			}
+	return s, nil
+}
 
-			if fleetSyncer == nil {
-				fleetSyncer = apple_mdm.NewDEPService(ds, depStorage, logger)
+func appleMDMDEPSyncerJob(
+	ds fleet.Datastore,
+	depStorage *mysql.NanoDEPStorage,
+	logger kitlog.Logger,
+) func(context.Context) error {
+	var fleetSyncer *apple_mdm.DEPService
+	return func(ctx context.Context) error {
+		appCfg, err := ds.AppConfig(ctx)
+		if err != nil {
+			return ctxerr.Wrap(ctx, err, "retrieving app config")
+		}
+
+		if !appCfg.MDM.AppleBMEnabledAndConfigured {
+			return nil
+		}
+
+		// As part of the DB migration of the single ABM token to the multi-ABM
+		// token world (where the token was migrated from mdm_config_assets to
+		// abm_tokens), we need to complete migration of the existing token as
+		// during the DB migration we didn't have the organization name, apple id
+		// and renewal date.
+		incompleteToken, err := ds.GetABMTokenByOrgName(ctx, "")
+		if err != nil && !fleet.IsNotFound(err) {
+			return ctxerr.Wrap(ctx, err, "retrieving migrated ABM token")
+		}
+		if incompleteToken != nil {
+			logger.Log("msg", "migrated ABM token found, updating its metadata")
+			if err := apple_mdm.SetABMTokenMetadata(ctx, incompleteToken, depStorage, ds, logger); err != nil {
+				return ctxerr.Wrap(ctx, err, "updating migrated ABM token metadata")
 			}
+			if err := ds.SaveABMToken(ctx, incompleteToken); err != nil {
+				return ctxerr.Wrap(ctx, err, "saving updated migrated ABM token")
+			}
+			logger.Log("msg", "completed migration of existing ABM token")
+		}
 
-			return fleetSyncer.RunAssigner(ctx)
-		}),
-	)
+		if fleetSyncer == nil {
+			fleetSyncer = apple_mdm.NewDEPService(ds, depStorage, logger)
+		}
 
-	return s, nil
+		return fleetSyncer.RunAssigner(ctx)
+	}
 }
 
 func newMDMProfileManager(

cmd/fleet/cron_test.go

@@ -2,13 +2,24 @@ package main
 
 import (
 	"context"
+	"database/sql"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
 	"testing"
 
 	"github.com/stretchr/testify/require"
 
+	"github.com/fleetdm/fleet/v4/server/datastore/mysql"
+	"github.com/fleetdm/fleet/v4/server/fleet"
 	apple_mdm "github.com/fleetdm/fleet/v4/server/mdm/apple"
+	nanodep_client "github.com/fleetdm/fleet/v4/server/mdm/nanodep/client"
+	"github.com/fleetdm/fleet/v4/server/mdm/nanodep/godep"
 	"github.com/fleetdm/fleet/v4/server/mock"
 	mdmmock "github.com/fleetdm/fleet/v4/server/mock/mdm"
+	"github.com/fleetdm/fleet/v4/server/test"
+	"github.com/go-kit/log"
 	kitlog "github.com/go-kit/log"
 )
 
@@ -23,3 +34,95 @@ func TestNewMDMProfileManagerWithoutConfig(t *testing.T) {
 	require.NotNil(t, sch)
 	require.NoError(t, err)
 }
+
+func TestMigrateABMTokenDuringDEPCronJob(t *testing.T) {
+	// FIXME
+	t.Skip()
+	ctx := context.Background()
+	ds := mysql.CreateMySQLDS(t)
+
+	depStorage, err := ds.NewMDMAppleDEPStorage()
+	require.NoError(t, err)
+	// to avoid issues with syncer, use that constant as org name for now
+	const tokenOrgName = "fleet"
+
+	// insert an ABM token as if it had been migrated by the DB migration script
+	tok := mysql.SetTestABMAssets(t, ds, "")
+	// tok, err := ds.InsertABMToken(ctx, &fleet.ABMToken{EncryptedToken: abmToken, RenewAt: time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC)})
+	// require.NoError(t, err)
+	require.Empty(t, tok.OrganizationName)
+
+	// start a server that will mock the Apple DEP API
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		encoder := json.NewEncoder(w)
+		switch r.URL.Path {
+		case "/session":
+			_, _ = w.Write([]byte(`{"auth_session_token": "session123"}`))
+		case "/account":
+			_, _ = w.Write([]byte(fmt.Sprintf(`{"admin_id": "admin123", "org_name": "%s"}`, tokenOrgName)))
+		case "/profile":
+			err := encoder.Encode(godep.ProfileResponse{ProfileUUID: "profile123"})
+			require.NoError(t, err)
+		case "/server/devices":
+			err := encoder.Encode(godep.DeviceResponse{Devices: nil})
+			require.NoError(t, err)
+		case "/devices/sync":
+			err := encoder.Encode(godep.DeviceResponse{Devices: nil})
+			require.NoError(t, err)
+		default:
+			t.Errorf("unexpected request to %s", r.URL.Path)
+		}
+	}))
+	t.Cleanup(srv.Close)
+
+	err = depStorage.StoreConfig(ctx, tokenOrgName, &nanodep_client.Config{BaseURL: srv.URL})
+	require.NoError(t, err)
+	err = depStorage.StoreConfig(ctx, apple_mdm.UnsavedABMTokenOrgName, &nanodep_client.Config{BaseURL: srv.URL})
+	require.NoError(t, err)
+
+	logger := log.NewNopLogger()
+	syncFn := appleMDMDEPSyncerJob(ds, depStorage, logger)
+	err = syncFn(ctx)
+	require.NoError(t, err)
+
+	// token has been updated with its org name/apple id
+	tok, err = ds.GetABMTokenByOrgName(ctx, tokenOrgName)
+	require.NoError(t, err)
+	require.Equal(t, tokenOrgName, tok.OrganizationName)
+	require.Equal(t, "admin123", tok.AppleID)
+	require.Nil(t, tok.MacOSDefaultTeamID)
+	require.Nil(t, tok.IOSDefaultTeamID)
+	require.Nil(t, tok.IPadOSDefaultTeamID)
+
+	// empty-name token does not exist anymore
+	_, err = ds.GetABMTokenByOrgName(ctx, "")
+	require.Error(t, err)
+	var nfe fleet.NotFoundError
+	require.ErrorAs(t, err, &nfe)
+
+	// the default profile was created
+	defProf, err := ds.GetMDMAppleEnrollmentProfileByType(ctx, fleet.MDMAppleEnrollmentTypeAutomatic)
+	require.NoError(t, err)
+	require.NotNil(t, defProf)
+	require.NotEmpty(t, defProf.Token)
+
+	// no profile UUID was assigned for no-team (because there are no hosts right now)
+	profUUID, _, err := ds.GetMDMAppleDefaultSetupAssistant(ctx, nil, "")
+	require.NoError(t, err)
+	require.Equal(t, "", profUUID)
+
+	// no teams, so no team-specific custom setup assistants
+	teams, err := ds.ListTeams(ctx, fleet.TeamFilter{User: test.UserAdmin}, fleet.ListOptions{})
+	require.NoError(t, err)
+	require.Empty(t, teams)
+
+	// no no-team custom setup assistant
+	_, err = ds.GetMDMAppleSetupAssistant(ctx, nil)
+	require.ErrorIs(t, err, sql.ErrNoRows)
+
+	// no host got created
+	hosts, err := ds.ListHosts(ctx, fleet.TeamFilter{User: test.UserAdmin}, fleet.HostListOptions{})
+	require.NoError(t, err)
+	require.Empty(t, hosts)
+}

cmd/fleet/serve.go

@@ -565,7 +565,6 @@ the way that the Fleet server works.
 				err = ds.InsertMDMConfigAssets(context.Background(), []fleet.MDMConfigAsset{
 					{Name: fleet.MDMAssetABMKey, Value: appleBM.KeyPEM},
 					{Name: fleet.MDMAssetABMCert, Value: appleBM.CertPEM},
-					{Name: fleet.MDMAssetABMToken, Value: appleBM.EncryptedToken},
 				})
 				if err != nil {
 					// duplicate key errors mean that we already
@@ -577,6 +576,18 @@ the way that the Fleet server works.
 					}
 
 					level.Warn(logger).Log("msg", "Your server already has stored ABM certificates and token. Fleet will ignore any certificates provided via environment variables when this happens.")
+				} else {
+					// insert the ABM token without any metdata,
+					// it'll be picked by the
+					// apple_mdm_dep_profile_assigner cron and
+					// backfilled
+					tok := &fleet.ABMToken{
+						EncryptedToken: appleBM.EncryptedToken,
+					}
+					_, err = ds.InsertABMToken(context.Background(), tok)
+					if err != nil {
+						initFatal(err, "save ABM token")
+					}
 				}
 			}
 
@@ -609,14 +620,23 @@ the way that the Fleet server works.
 					initFatal(err, "validating MDM assets from database")
 				}
 
-				appCfg.MDM.AppleBMEnabledAndConfigured, err = checkMDMAssets([]fleet.MDMAssetName{
+				var appleBMCerts bool
+				appleBMCerts, err = checkMDMAssets([]fleet.MDMAssetName{
 					fleet.MDMAssetABMCert,
 					fleet.MDMAssetABMKey,
-					fleet.MDMAssetABMToken,
 				})
 				if err != nil {
 					initFatal(err, "validating MDM ABM assets from database")
 				}
+				if appleBMCerts {
+					// the ABM certs are there, check if a token exists and if so, apple
+					// BM is enabled and configured.
+					count, err := ds.GetABMTokenCount(context.Background())
+					if err != nil {
+						initFatal(err, "validating MDM ABM token from database")
+					}
+					appCfg.MDM.AppleBMEnabledAndConfigured = count > 0
+				}
 			}
 
 			// register the Microsoft MDM services