Skip to content

Commit 38277fc

Browse files
elliot-huffmanelliot-huffman
authored
Merge pull request #41 from Software-Hardware-Integration-Lab/feat/ko/datagateway-review
[Feature] ko/datagateway review
2 parents 011b8f7 + 5dcbc8f commit 38277fc

File tree

21 files changed

+522
-67
lines changed

21 files changed

+522
-67
lines changed

‎docs/Data-Gateway/Architecture/Infrastructure.md‎

Lines changed: 175 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,183 @@
1-
# Infrastructure Diagram
1+
# Infrastructure
22

3-
The below is the logical infrastructure interaction of the core Discover engine.
4-
Each plugin interacts with the specific service configuration that is being interpreted.
3+
This page describes how Data Gateway is composed and how the core services interact within SHI Cloud.
54

6-
## Diagram
5+
---
6+
7+
## Platform topology
78

89
```mermaid
9-
flowchart BT
10-
subgraph shiLab["SHI Lab - Trust Boundary"]
11-
AppService{{"Data Gateway"}}
12-
SqlDb[("General Data<br>(SQL DB)")]
13-
BulkData(["Bulk Data<br>(Storage Blob)"])
14-
UpdateData(["Update Data<br>(Storage Blob + Table)"])
15-
end
16-
subgraph microsoftGlobal["Microsoft Global"]
17-
entraId{"Entra ID"}
18-
shiLab
19-
end
20-
subgraph clientEnvironment["Client Environment"]
21-
client[/"REST API Client"\]
22-
end
23-
24-
AppService --> entraId
25-
AppService -- CRUD Operation + Token --> SqlDb
26-
AppService -- CRUD Operation + Token --> BulkData & UpdateData
27-
BulkData <-- Validate Auth of CRUD Operation --> entraId
28-
SqlDb <-- Validate Auth of CRUD Operation --> entraId
29-
entraId -- Public Keys --> AppService
30-
entraId <-- Log into Entra ID --> client
31-
AppService -- "CRUD Operation + Token. Response included." --> client
32-
UpdateData <-- Validate Auth of CRUD Operation --> entraId
10+
flowchart TD
11+
12+
%% Entra ID Trust Boundary
13+
subgraph CustomerTrustBoundary[Customer Environment]
14+
Client(["API Client/SDK"])
15+
Users([Users])
16+
end
17+
18+
%% Entra ID Trust Boundary
19+
subgraph EntraTrustBoundary[Microsoft Global Cloud]
20+
Entra{"Entra ID (IDP)"}
21+
end
22+
23+
%% GitHub Trust Boundary
24+
subgraph GitHubTrustBoundary[GitHub Pages]
25+
Dashboard((Data Gateway UI))
26+
end
27+
28+
%% SHI Cloud trust boundary
29+
subgraph ShiTrustBoundary[SHI Cloud]
30+
direction LR
31+
32+
API{{Data Gateway API}}
33+
34+
%% Data services
35+
subgraph DataStorage[Data Services]
36+
direction TB
37+
AzSql[("Relational Data (Azure SQL DB)")]
38+
GeneralBlob[/"Bulk data (Azure Blob)"\]
39+
UpdateBlob[/"Update packages (Azure Blob)"\]
40+
AzTable[["Update config (Azure Table)"]]
41+
OpenAi(["LLM Service (Azure OpenAI)"])
42+
end
43+
end
44+
45+
%% Relationships
46+
Users --> | HTTPS | Dashboard
47+
Client --> | Public/Secret Client Auth | Entra
48+
Entra --> | Auth Code/Access Tokens | Client
49+
Client --> API
50+
Dashboard --> | Public Client Auth | Entra
51+
Entra --> | Auth Code/Access Tokens | Dashboard
52+
Dashboard --> | HTTPS + JWT | API
53+
API --> | TDS + JWT | AzSql
54+
API --> | HTTPS + JWT | GeneralBlob
55+
API --> | HTTPS + JWT | UpdateBlob
56+
API --> | HTTPS + JWT | AzTable
57+
API --> | HTTPS + JWT | OpenAi
3358
```
3459

35-
## Threat Model
60+
!!! note "Key points"
61+
- **Identity** - Entra ID is the only identity provider. The UI authenticates users and calls the API with bearer tokens.
62+
- **Data access** - All reads/writes are brokered by the API:
63+
- **Azure SQL Database** for processed relational data
64+
- **Azure Blob Storage** for bulk reports and update packages
65+
- **Azure Table Storage** for update service configuration
66+
- **LicenseGPT** - Chat interactions are transient. Prompts are sent to the API and responses are returned in-session, never stored.
67+
68+
## SHIELD Update Service
69+
70+
Data Gateway controls update delivery using **channels** (e.g., stable, beta, alpha) and **rings** (e.g., ring 0, ring 1). Configuration is stored in **Azure Table Storage**; package files are stored in **Azure Blob Storage**.
71+
72+
### Example Channel Configuration
73+
74+
The below channel configuration demonstrates how the update system can be configured where various tenants are assigned by default to a ring in a channel.
75+
The tenant's default channel can be overridden by API call. If this happens it defaults to ring 0 on the other channel.
76+
In the below diagram, the channel is not named, all channels follow the below architecture.
77+
There can be an unlimited number of rings in a channel. `N...` represents all numbers above 1.
78+
79+
The Alpha channel is RBAC gated and is not available by default. SHI has to approve Alpha access per-tenant.
80+
81+
```mermaid
82+
flowchart TD
83+
84+
%% Tenant Configs
85+
CxTenant1[/Tenant 1\]
86+
CxTenant2[/Tenant 2\]
87+
CxTenant3[/Tenant 3\]
88+
DevTenant1[/Dev Tenant 1\]
89+
DevTenant2[/Dev Tenant 2\]
90+
91+
%% Channel Example
92+
subgraph Channel["Channel"]
93+
Ring0(("Ring 0<br>Latest"))
94+
Ring1(("Ring 1<br>Latest"))
95+
RingN(("Ring N...<br>Previous"))
96+
end
97+
98+
%% Available Versions
99+
Versions["Latest: 3.0.0<br>Previous: 2.5.0"]
100+
101+
%% Relationships
102+
Versions --> Channel
103+
Ring0 --> CxTenant1
104+
Ring1 --> CxTenant2
105+
RingN --> CxTenant3
106+
Ring0 --> DevTenant1
107+
Ring0 --> DevTenant2
108+
```
109+
110+
### Data Flow
111+
112+
The below chart demonstrates how the data storage systems relate to each other and how the configurations flow to Data Gateway.
113+
114+
```mermaid
115+
flowchart LR
116+
117+
%% Tenant and API
118+
Client([SHIELD])
119+
API{{Data Gateway API}}
120+
121+
%% Configuration in Table Storage
122+
subgraph AzureTableStorage[Azure Table Storage]
123+
ChannelConfig[["Channels"]]
124+
RingConfig[["Rings"]]
125+
TenantConfig[["Tenant config"]]
126+
end
127+
128+
%% Packages in Blob
129+
PackageStorage[/"Update Packages (Azure Blob)"\]
130+
131+
%% Relationships
132+
Client --> | Request Version | API
133+
API --> | Request Config | TenantConfig
134+
TenantConfig --> | Default channel, Ring, Alpha allowed | API
135+
API --> | Request Config | ChannelConfig
136+
ChannelConfig --> | Latest Version, Previous Version | API
137+
API --> | Request Config | RingConfig
138+
RingConfig --> | Is Latest Available | API
139+
API --> | Request Package | PackageStorage
140+
PackageStorage --> | Package Stream | API
141+
API --> | Version or Package Stream | Client
142+
```
143+
144+
### Update Selection Process
145+
146+
1. Tenant defaults define which channel and ring are used, and whether alpha builds are allowed.
147+
2. If you explicitly request a channel and it's permitted, the API applies it; otherwise the tenant default applies.
148+
3. The ring setting (latest or previous) determines the version within that channel.
149+
4. The API returns the resolved version and streams the corresponding package from Blob Storage.
150+
151+
### Data Flow Lifecycle
152+
153+
1. **Authenticate** - The UI authenticates with Entra ID and receives a token.
154+
2. **Authorize** - The UI calls the API over HTTPS with the JWT.
155+
3. **Process** - The API validates the token and performs reads/writes against SQL, Blob, or Table Storage.
156+
4. **Respond** - The API returns data or confirmation.
157+
5. **Render** - The UI presents the result to the user. *(LicenseGPT chat is not stored.)*
158+
159+
### Security and Reliability
160+
161+
- **Identity** - Entra ID is the single identity provider.
162+
- **Transport** - All calls use HTTPS.
163+
- **Isolation** - Clients never access data stores directly; all access is via the API.
164+
- **Controlled rollout** - Channels and rings enable predictable, staged updates.
165+
- **Resilience** - Operations are designed to handle transient faults.
166+
167+
### Component Map
168+
169+
| Component | Backing service(s) | Role |
170+
|---------------------|---------------------------------|----------------------------------------------------------------------|
171+
| Data Gateway UI | Web Browser, [GitHub Pages](https://docs.github.com/en/pages) | Entry point for user interactions (Tenant Manager, LicenseGPT) |
172+
| Data Gateway API | HTTPS, [Entra ID auth](https://learn.microsoft.com/en-us/entra/identity-platform/application-model#multitenant-apps) | Authenticates tokens and brokers access to data stores |
173+
| Processed Data | [Azure SQL Database](https://learn.microsoft.com/en-us/azure/azure-sql/database) | Relational/reporting data; tenant metadata |
174+
| Bulk report Data | [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/) | JSON/report payloads |
175+
| Update Packages | [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/) | Versioned package archives |
176+
| Update Config | [Azure Table Storage](https://learn.microsoft.com/en-us/azure/storage/tables/) | Channels, rings, and tenant update settings |
177+
| LicenseGPT | [Azure OpenAI Services](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/overview) | Provides LLM and Embeddings to the Data Gateway service |
36178

37-
For an annotated threat model of the application's infrastructure architecture, please see the attached Microsoft [Threat Model Tool](https://aka.ms/tmt) representation of the infrastructure architecture.
179+
## See also
38180

39-
📄 [Data Gateway - Threat Model](../assets/threat-models/Data-Gateway.tm7)
181+
- [Usage Guide](../Usage-Guide/index.md)
182+
- [Threat Model](../assets/threat-models/Data-Gateway.tm7)
183+
- [API Reference](https://specs.shilab.com)

‎docs/Data-Gateway/Architecture/index.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Architecture
2+
3+
The Architecture section describes how Data Gateway is structured, including trust boundaries, components, and service interactions.
4+
5+
- [Infrastructure](Infrastructure.md)

‎docs/Data-Gateway/Deployment/index.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Deployment
2+
3+
The Data Gateway service does not require installation in your tenant.
4+
The service is hosted by SHI and is available directly via the UI and API.
5+
6+
## Permission Grants
7+
8+
The Data Gateway SaaS requires that permissions be granted before you can access the app.
9+
To grant permissions to a principal in your tenant, please register the `SHI - Data Gateway` app so that SSO is enabled by navigating to the following URL:
10+
11+
- <https://login.microsoftonline.com/common/adminconsent?client_id=4c40281b-a305-4aaf-90a4-d5bbee6eb8ed&redirect_uri=https%3A%2F%2Fdashboard.shilab.com>
12+
13+
!!! note
14+
To register this app, you may require the [Cloud Application Administrator](https://learn.microsoft.com/en-us/entra/identity/role-based-access-control/permissions-reference#cloud-application-administrator) role or higher in your tenant depending on the admin consent settings configured.
15+
16+
When the app is registered, you will see an Enterprise App that has the name of `SHI - Data Gateway` with the Client ID of `4c40281b-a305-4aaf-90a4-d5bbee6eb8ed`.
17+
This enterprise app allows principals in your tenant to be able to generate Access Tokens (JWT) and make API calls to the Data Gateway service.
18+
This enterprise app does not allow the Data Gateway service to connect into your tenant.
19+
20+
## Getting Started
21+
22+
- Sign in with your Entra ID account.
23+
- Navigate the [Usage Guide](../Usage-Guide/index.md) for common tasks.
24+
- See [Reference](../Reference/index.md) for API integration details.

‎docs/Data-Gateway/Reference/index.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Reference
2+
3+
This page is a hub for the **Data Gateway** reference material: how to authenticate, where to find the live API documentation, and a quick tour of the most-used endpoint families.
4+
5+
## Quick links
6+
7+
<div class="grid cards" markdown>
8+
9+
- :rocket: **Getting Started**
10+
Sign in, navigate the UI, and complete common tasks.
11+
[:octicons-arrow-right-24: Usage Guide](../Usage-Guide/index.md)
12+
13+
- :triangular_ruler: **Architecture**
14+
Trust boundaries, components, and service interactions.
15+
[:octicons-arrow-right-24: Infrastructure](../Architecture/Infrastructure.md)
16+
17+
- :gear: **API Reference (Swagger)**
18+
Browse the live OpenAPI reference and try requests in your browser.
19+
[:octicons-link-24: specs.shilab.com](https://specs.shilab.com)
20+
21+
</div>
22+
23+
## Authentication
24+
25+
Data Gateway uses **Entra ID** (Microsoft identity platform) for authentication.
26+
All requests must include a valid **JSON Web Token (Bearer/Access Token)** in the `Authorization` header.
27+
28+
### Steps
29+
30+
1. Sign in with your organization's Entra ID principal to obtain an access token for the Data Gateway application.
31+
2. Include the token in each API request:
32+
33+
```bash
34+
curl -sS https://api.shilab.com/datagateway/status \
35+
-H "Authorization: Bearer <token>" \
36+
-H "Accept: application/json"
37+
```
38+
39+
### Notes
40+
41+
- Tokens are validated by the API; users do **not** access SQL or Storage directly.
42+
- Tokens expire; refresh them using your chosen auth flow (authorization code, client credentials, etc.).
43+
- LicenseGPT prompts and responses are **not persisted** - the API returns results to the UI for the current session.
44+
45+
## Endpoint Families
46+
47+
| Family | Purpose | Typical methods | Common paths* |
48+
|---|---|---|---|
49+
| **Health & metadata** | Service liveness and basic info | `GET` | `/Api/Core/Health` |
50+
| **Tenants** | Read and maintain tenant metadata (display name, parent association) | `GET`, `PATCH` | `/Api/Tenant`, `/Api/Tenant/{tenantId}` |
51+
| **LicenseGPT** | AI-assisted licensing & compliance analysis | `POST` | `/Api/Chat/LicenseGpt` |
52+
| **Updates (channels & rings)** | Resolve version and retrieve update package | `GET` (and streaming) | See Swagger (“Updates”) |
53+
54+
\* For the complete, authoritative list (parameters, schemas, and responses), use the live reference at **[specs.shilab.com](https://specs.shilab.com)**.
55+
56+
## Request & Response Basics
57+
58+
- **Protocol:** HTTPS only
59+
- **Content type:** `application/json; charset=utf-8` (unless explicitly streaming binaries)
60+
- **Date/time:** ISO 8601 in UTC unless stated otherwise
61+
- **Pagination/filters:** When applicable, filter and paging parameters are documented per endpoint in Swagger
62+
63+
## Error handling
64+
65+
The API uses standard HTTP status codes with JSON error payloads. Please see [MDN - Status Codes](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status) for more details on specific codes and how they should be interpreted.
66+
67+
!!! note
68+
The response body typically includes an explanatory message; consult Swagger for exact schemas.
69+
70+
---
71+
72+
## See also
73+
74+
- [Usage Guide](../Usage-Guide/index.md)
75+
- [Infrastructure](../Architecture/Infrastructure.md)
76+
- [API Reference (Swagger)](https://specs.shilab.com)

‎docs/Data-Gateway/Troubleshooting.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Troubleshooting
2+
3+
🚧 **This section is coming soon.**
4+
5+
Guidance for diagnosing common Data Gateway issues will be published here.

0 commit comments

Comments
 (0)