Skip to content

Commit 4c871fd

Browse files
ajitgunturiajitgunturi
committed
docs(readme): expand repo and package guides to be self-sufficient
Top-level README now carries the full OAuth/CASA walkthrough, SyncClient API, conflict strategies, per-backend quick-starts, architecture diagram, scope boundaries, migration table from drive_sync_flutter, and release workflow — so the GitHub entry point no longer punts to pub.dev or per-package READMEs for core concepts. cloud_sync_core README gains SyncClient usage, custom-adapter guide with SHA256 preservation guidance, and a conflict-resolution reference. cloud_sync_drive README grows into a standalone pub.dev landing page: scope-mode comparison, CASA cost details, visibility trap with drive.file, folder layouts, path-validation rules, scope-mismatch error mapping, and migration table.
1 parent 82f9fdd commit 4c871fd

3 files changed

Lines changed: 633 additions & 43 deletions

File tree

‎README.md‎

Lines changed: 321 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,354 @@
11
# cloud_sync
22

3-
Bidirectional file sync for Dart and Flutter, across multiple cloud storage backends. Path-based manifest diffing, SHA256 change detection, pluggable conflict resolution — implemented once in a storage-agnostic core, reused across every backend.
3+
Bidirectional file sync for Dart and Flutter, across multiple cloud storage backends. Path-based manifest diffing, SHA256 change detection, pluggable conflict resolution — implemented once in a storage-agnostic core and reused across every backend.
4+
5+
- **Storage-agnostic core** — diff, resolve, transfer logic lives in `cloud_sync_core` and has zero knowledge of any specific backend.
6+
- **One small interface per backend** — 5 methods (`ensureFolder`, `listFiles`, `uploadFile`, `downloadFile`, `deleteFile`).
7+
- **Swap adapters without touching your app code** — the same `SyncClient` works against Google Drive today, S3 tomorrow, Box the day after.
8+
- **Opaque bytes** — the engine never reads file contents. Works identically for JSON, binary, or pre-encrypted blobs.
9+
10+
---
411

512
## Packages
613

7-
| Package | Description | Status |
14+
| Package | Description | pub.dev |
815
|---|---|---|
9-
| [`cloud_sync_core`](packages/cloud_sync_core) | Core interfaces (`StorageAdapter`), sync engine, manifest differ, conflict resolver, `SyncClient`. No backend logic. | 0.1.0 |
10-
| [`cloud_sync_drive`](packages/cloud_sync_drive) | Google Drive adapter. `drive`, `drive.file`, `drive.appdata` scopes supported. | 0.1.0 |
11-
| `cloud_sync_s3` | AWS S3 + S3-compatible adapter (R2, MinIO, Backblaze B2, Wasabi, DO Spaces). | Planned |
12-
| `cloud_sync_box` | Box Content API adapter. | Planned |
16+
| [`cloud_sync_core`](packages/cloud_sync_core) | Core interfaces (`StorageAdapter`), sync engine, manifest differ, conflict resolver, `SyncClient`. No backend logic. | [![pub](https://img.shields.io/pub/v/cloud_sync_core.svg)](https://pub.dev/packages/cloud_sync_core) |
17+
| [`cloud_sync_drive`](packages/cloud_sync_drive) | Google Drive adapter. `drive`, `drive.file`, `drive.appdata` scopes supported. | [![pub](https://img.shields.io/pub/v/cloud_sync_drive.svg)](https://pub.dev/packages/cloud_sync_drive) |
18+
| [`cloud_sync_s3`](packages/cloud_sync_s3) | AWS S3 + S3-compatible adapter (R2, MinIO, Backblaze B2, Wasabi, DO Spaces). | [![pub](https://img.shields.io/pub/v/cloud_sync_s3.svg)](https://pub.dev/packages/cloud_sync_s3) |
19+
| [`cloud_sync_box`](packages/cloud_sync_box) | Box Content API adapter. | [![pub](https://img.shields.io/pub/v/cloud_sync_box.svg)](https://pub.dev/packages/cloud_sync_box) |
20+
21+
All four packages live at `0.1.x` and follow independent semver — you can pin `cloud_sync_drive` while upgrading `cloud_sync_s3`.
22+
23+
---
24+
25+
## Install
26+
27+
```yaml
28+
dependencies:
29+
cloud_sync_core: ^0.1.1
30+
cloud_sync_drive: ^0.1.1 # add the adapter(s) you need
31+
# cloud_sync_s3: ^0.1.1
32+
# cloud_sync_box: ^0.1.1
33+
```
1334

14-
## Design
35+
You only need `cloud_sync_core` + at least one adapter package. The core itself does nothing useful without an adapter.
1536

16-
One interface, many backends. Each backend package implements `StorageAdapter` (5 methods: `ensureFolder`, `listFiles`, `uploadFile`, `downloadFile`, `deleteFile`) and ships with its own auth helpers. The sync engine in `cloud_sync_core` works identically against any adapter.
37+
---
1738

18-
## Usage
39+
## Quick start — Google Drive
1940

2041
```dart
2142
import 'package:cloud_sync_core/cloud_sync_core.dart';
2243
import 'package:cloud_sync_drive/cloud_sync_drive.dart';
44+
import 'package:google_sign_in/google_sign_in.dart';
2345
46+
// 1. Authenticate with any scope that matches the adapter mode you want.
47+
final signIn = GoogleSignIn(scopes: [
48+
'https://www.googleapis.com/auth/drive.file', // for .appFiles()
49+
]);
50+
final account = await signIn.signIn();
51+
final authClient = DriveAuthClient(await account!.authHeaders);
52+
53+
// 2. Build an adapter.
2454
final adapter = DriveAdapter.appFiles(
2555
httpClient: authClient,
2656
folderName: 'MyApp',
57+
subPath: 'backups',
2758
);
59+
60+
// 3. Sync.
2861
final client = SyncClient(adapter: adapter);
2962
final result = await client.sync(localPath: '/path/to/data');
63+
print('${result.filesUploaded} uploaded, ${result.filesDownloaded} downloaded');
64+
```
65+
66+
Swap the adapter (`DriveAdapter` → `S3Adapter` → `BoxAdapter`) and the rest of the code is identical.
67+
68+
---
69+
70+
## The unified `SyncClient` API
71+
72+
`SyncClient` is the top-level type every adapter plugs into. Once you have an adapter, the operations are the same across backends:
73+
74+
```dart
75+
final client = SyncClient(
76+
adapter: adapter,
77+
defaultStrategy: ConflictStrategy.newerWins,
78+
);
79+
80+
// Bidirectional sync — new files go both ways, conflicts resolved by strategy
81+
final result = await client.sync(localPath: '/data');
82+
83+
// Push only — local overwrites remote
84+
await client.push(localPath: '/data');
85+
86+
// Pull only — remote overwrites local
87+
await client.pull(localPath: '/data');
88+
89+
// Dry-run: see what would change
90+
final status = await client.status(localPath: '/data');
91+
print('Pending: ${status.pendingChanges?.totalChanges ?? 0}');
92+
```
93+
94+
### Change detection
95+
96+
A JSON manifest (`_sync_manifest.json`) is kept alongside your local data and records `{path, sha256, lastModified}` for every synced file. On each run the engine diffs local vs. remote vs. manifest and only transfers files that actually changed. Unchanged files incur no network I/O.
97+
98+
### Conflict resolution
99+
100+
When both sides modified the same file, `SyncClient` **picks one version** — it never merges content. It compares SHA256 checksums (to detect changes) and `lastModified` timestamps (to pick a winner), so it works on binary, JSON, or encrypted blobs.
101+
102+
| Strategy | Behavior |
103+
|---|---|
104+
| `newerWins` | Most recent `lastModified` wins. Ties go to local. |
105+
| `localWins` | Always keep the local version; remote is overwritten. |
106+
| `remoteWins` | Always keep the remote version; local is overwritten. |
107+
| `askUser` | Skip the file and return it in `result.unresolvedConflicts` for your UI to handle. |
108+
109+
If you need to preserve both versions, use `askUser` and implement your own merge or backup policy.
110+
111+
---
112+
113+
## Google Drive — OAuth scopes & CASA
114+
115+
`cloud_sync_drive` supports all three Drive OAuth scopes via three factory constructors. Your choice determines what files the app can see, whether you need CASA (annual security audit), and what tradeoffs you're making. This is the area with the highest compliance blast radius, so read carefully before picking one.
116+
117+
### At a glance
118+
119+
| Factory | OAuth scope | App sees | User sees in Drive UI | CASA needed? |
120+
|---|---|---|---|---|
121+
| `DriveAdapter.userDrive(basePath:)` | `drive` (full) | Everything in the user's Drive | Files visible | **Yes** for public distribution |
122+
| `DriveAdapter.appFiles(folderName:)` | `drive.file` | Only files this app created | Files visible | No |
123+
| `DriveAdapter.appData(subPath:)` | `drive.appdata` | Only contents of hidden `appDataFolder` | **Nothing** (folder hidden) | No |
124+
125+
### Which should I pick?
126+
127+
- **`.appFiles()`** — The 80% case and lowest compliance burden. Your app is the *only* writer. No CLI, no companion web app, no Drive Desktop drops into the sync folder.
128+
- **`.userDrive()`** — Multiple OAuth clients write to the same folder (e.g. a CLI tool on a laptop plus a mobile app, or Drive Desktop drops that the app needs to read). Full `drive` is the only scope that lets the app see files created by other identities. Restricted-scope: public distribution requires OAuth verification *plus* annual CASA (details below).
129+
- **`.appData()`** — Internal state the user should never see — app config, caches, encrypted blobs. The `appDataFolder` is invisible in the Drive UI, quota-separate from the user's Drive, and strictly scoped to this OAuth client ID.
130+
131+
### The visibility trap with `.appFiles()`
132+
133+
`drive.file` is scoped by **creating OAuth client ID**, not by path. If anything other than your Flutter app writes into the folder — the user manually, Drive Desktop, a companion CLI — those files are **invisible** to your app's `listFiles()`, even if they live in the same folder. You'll only discover this in production, when a user says "where are my plans?"
134+
135+
If your architecture has multiple writers, use `.userDrive()` instead.
136+
137+
### CASA and the restricted-scope tax
138+
139+
`.userDrive()` uses the `drive` scope, which Google classifies as *restricted*. Public distribution requires:
140+
141+
1. **Google OAuth verification** — one-time review, free, takes 1–4 weeks. Brand/domain verification + privacy policy review + scope-justification video.
142+
2. **Annual CASA** (Cloud Application Security Assessment) — third-party security audit by a Google-approved lab. Tier 2 is the common minimum: ~$5K–$20K/year. Covers pen test, SAST/DAST scan, token-storage review, deletion-flow review.
143+
144+
**Can you skip CASA?** Yes, by keeping your OAuth client in **Testing** publishing status:
145+
146+
- Up to 100 test users (listed by Gmail address).
147+
- Users see a "Google hasn't verified this app" warning on first sign-in.
148+
- Refresh tokens for restricted scopes expire every 7 days — users re-sign-in weekly.
149+
150+
Testing mode is the legitimate path for personal apps, family tools, and small-circle distribution. **Workspace escape hatch:** if all users belong to a Google Workspace domain, set the consent screen user type to `Internal` and skip verification + CASA + the 100-user cap entirely.
151+
152+
### Folder layouts
153+
154+
```
155+
.userDrive(basePath: '.app/longeviti', subPath: 'plans')
156+
└── User's Google Drive
157+
└── .app/
158+
└── longeviti/
159+
└── plans/ ← synced files here
160+
161+
.appFiles(folderName: 'MyApp', subPath: 'backups')
162+
└── User's Google Drive
163+
└── MyApp/
164+
└── backups/ ← synced files here
165+
(visible to user; app sees only its own files)
166+
167+
.appData(subPath: 'cache')
168+
└── Hidden appDataFolder (invisible to user)
169+
└── cache/ ← synced files here
170+
```
171+
172+
### Scope-mismatch errors
173+
174+
If the auth client's actual scope doesn't match the adapter's declared scope, the first Drive API call returns 403. The library catches this and re-raises as `DriveScopeError` with a remediation message pointing at the expected scope.
175+
176+
---
177+
178+
## Amazon S3 + S3-compatible services
179+
180+
`cloud_sync_s3` supports AWS S3 and every S3-compatible service we've tested: Cloudflare R2, MinIO (local dev), Backblaze B2, Wasabi, DigitalOcean Spaces. See [`cloud_sync_s3/README.md`](packages/cloud_sync_s3/README.md) for endpoint-by-endpoint config snippets.
181+
182+
```dart
183+
import 'package:cloud_sync_core/cloud_sync_core.dart';
184+
import 'package:cloud_sync_s3/cloud_sync_s3.dart';
185+
186+
final adapter = S3Adapter(
187+
config: S3Config(region: 'us-east-1', bucket: 'my-sync-bucket'),
188+
credentials: S3Credentials(accessKeyId: 'AKIA...', secretAccessKey: '...'),
189+
);
190+
final client = SyncClient(adapter: adapter);
191+
await client.sync(localPath: '/data');
192+
```
193+
194+
The adapter ships a SigV4 implementation validated against AWS's official test vector and preserves SHA256 via `x-amz-meta-sha256`.
195+
196+
---
197+
198+
## Box
199+
200+
`cloud_sync_box` speaks the Box Content API and caches path-to-ID resolution on first use (Box's API is ID-based; the sync contract is path-based).
201+
202+
```dart
203+
import 'package:cloud_sync_core/cloud_sync_core.dart';
204+
import 'package:cloud_sync_box/cloud_sync_box.dart';
205+
206+
final authClient = BoxAuthClient(accessToken: 'your-oauth2-access-token');
207+
final adapter = BoxAdapter(
208+
config: BoxConfig(rootFolderId: '0'), // "0" = user's Box root
209+
httpClient: authClient,
210+
);
211+
final client = SyncClient(adapter: adapter);
212+
await client.sync(localPath: '/data');
213+
```
214+
215+
Box provides SHA1 natively; `cloud_sync_box` preserves SHA256 by stashing it under custom metadata (`/files/{id}/metadata/global/properties`). See [`cloud_sync_box/README.md`](packages/cloud_sync_box/README.md) for details.
216+
217+
---
218+
219+
## Architecture
220+
221+
```
222+
SyncClient <- high-level API (sync/push/pull/status)
223+
└─ SyncEngine <- orchestrates diff → resolve → transfer
224+
├─ ManifestDiffer <- compares file states
225+
├─ ConflictResolver <- applies conflict strategy
226+
└─ StorageAdapter <- backend interface (5 methods)
227+
├─ DriveAdapter (cloud_sync_drive)
228+
├─ S3Adapter (cloud_sync_s3)
229+
└─ BoxAdapter (cloud_sync_box)
30230
```
31231

32-
Swap the adapter, not the client. Everything downstream stays the same.
232+
Everything above `StorageAdapter` is backend-free. Everything below is backend-specific. Implementing a new adapter means filling in 5 methods against `StorageAdapter`.
233+
234+
---
235+
236+
## Scope & boundaries
237+
238+
### What cloud_sync does
239+
240+
- Syncs **files** (any format — JSON, YAML, images, binary, encrypted blobs) between a local directory and a cloud folder.
241+
- Detects changes via SHA256 — only transfers files that actually differ.
242+
- Resolves conflicts by strategy when both sides modified the same file.
243+
- Creates nested folder hierarchies on the remote automatically.
244+
- Tracks sync state via a local manifest file (`_sync_manifest.json`).
245+
- Validates paths structurally — rejects traversal, absolute paths, empty segments; escapes backend query strings.
246+
247+
### What cloud_sync does **not** do
248+
249+
- **No encryption.** Files are transferred as-is. Encrypt before syncing and decrypt after pulling if you need it.
250+
- **No content merging.** Conflict resolution picks one version; it never merges file contents.
251+
- **No authentication.** You supply an authenticated `http.Client` / credentials.
252+
- **No background sync.** Sync is triggered explicitly by your code.
253+
- **No partial / resumable transfers.** Files are up/downloaded in full. ~50MB per file is the practical ceiling in v1.
254+
- **No file locking or concurrency control.** Designed for single-device / single-writer use.
255+
256+
### Who handles what
257+
258+
| Concern | Who |
259+
|---|---|
260+
| OAuth flow (sign-in, token refresh) | **You** — use `google_sign_in` or equivalent |
261+
| Providing an authenticated HTTP client / credentials | **You** — wrap with `DriveAuthClient` / `S3AuthClient` / `BoxAuthClient` or your own |
262+
| Encryption of sensitive data | **You** — encrypt before sync, decrypt after pull |
263+
| File format and schema validation | **You** — library treats files as opaque bytes |
264+
| Retry logic on network failure | **You** — library returns errors in `SyncResult.errors` |
265+
| Background/periodic sync scheduling | **You** — call `sync()` when appropriate |
266+
| Change detection (SHA256) | Library |
267+
| Manifest tracking | Library |
268+
| Conflict resolution | Library (configurable strategy) |
269+
| Backend CRUD (list/upload/download/delete) | Library (per-adapter) |
270+
| Path validation & query-injection prevention | Library (`PathValidator` in `cloud_sync_core`) |
271+
| Per-file error reporting | Library (`SyncResult.errors`) |
272+
273+
---
274+
275+
## Migration from `drive_sync_flutter`
276+
277+
[`drive_sync_flutter`](https://pub.dev/packages/drive_sync_flutter) is frozen at 1.2.0. `cloud_sync_drive` is its successor. The public surface was renamed to drop the Drive-specific prefix and make room for other backends.
278+
279+
| `drive_sync_flutter` (old) | `cloud_sync_*` (new) | Lives in |
280+
|---|---|---|
281+
| `DriveSyncClient` | `SyncClient` | `cloud_sync_core` |
282+
| `DriveAdapter` (interface) | `StorageAdapter` | `cloud_sync_core` |
283+
| `GoogleDriveAdapter` | `DriveAdapter` | `cloud_sync_drive` |
284+
| `GoogleAuthClient` | `DriveAuthClient` | `cloud_sync_drive` |
285+
| `SandboxValidator` | `PathValidator` | `cloud_sync_core` |
286+
| `DriveScope`, `DriveScopeError` | (unchanged) | `cloud_sync_drive` |
287+
288+
**Dropped**: the deprecated `GoogleDriveAdapter.sandboxed()`, positional `GoogleDriveAdapter()`, and `.withPath()` constructors. Use `DriveAdapter.userDrive()`, `.appFiles()`, or `.appData()` explicitly — clean-slate API.
289+
290+
**Steps**:
291+
292+
1. In `pubspec.yaml`, replace `drive_sync_flutter: ^1.2.0` with `cloud_sync_core: ^0.1.1` + `cloud_sync_drive: ^0.1.1`.
293+
2. Rename imports: `package:drive_sync_flutter/...` → `package:cloud_sync_core/cloud_sync_core.dart` + `package:cloud_sync_drive/cloud_sync_drive.dart`.
294+
3. Rename types per the table above.
295+
4. Replace deprecated constructors with the equivalent `.userDrive()` / `.appFiles()` / `.appData()` call.
296+
297+
Behavior is unchanged — same manifest format, same conflict strategies, same scope semantics. An existing `_sync_manifest.json` from `drive_sync_flutter` is readable by `cloud_sync_core` without modification.
298+
299+
---
33300

34301
## Development
35302

303+
This is a [melos](https://melos.invertase.dev/)-managed monorepo.
304+
36305
```bash
37306
dart pub global activate melos
38-
melos bootstrap
39-
melos run analyze
40-
melos run test
307+
melos bootstrap # link workspace packages
308+
melos run analyze # dart analyze across all packages
309+
melos run test # dart test across all packages
310+
melos run format # check formatting
311+
```
312+
313+
Or via the top-level `Makefile`:
314+
315+
```bash
316+
make bootstrap
317+
make analyze
318+
make test
319+
make pre-release # analyze + test + publish dry-run — run before tagging
320+
```
321+
322+
`make help` lists every target.
323+
324+
### Tests
325+
326+
157 tests across 4 packages:
327+
328+
- `cloud_sync_core`: 62 — manifest diffing, conflict resolution, sync engine flows, path validation.
329+
- `cloud_sync_drive`: 28 — all three scope modes, scope-mismatch error mapping, query injection prevention, `DriveSyncClient` lifecycle.
330+
- `cloud_sync_s3`: 48 — SigV4 signing (against AWS's official test vector), adapter CRUD, SHA256 preservation, all S3-compatible endpoints.
331+
- `cloud_sync_box`: 19 — path-to-ID cache, metadata-backed SHA256, adapter CRUD.
332+
333+
---
334+
335+
## Release workflow
336+
337+
Each package releases independently. Tagging is gated on a clean tree, synced main, and a pubspec version that isn't already on pub.dev.
338+
339+
```bash
340+
# 1. Bump version in packages/<pkg>/pubspec.yaml + update CHANGELOG.md
341+
# 2. git commit + git push main
342+
# 3. Preflight (analyze + test + publish dry-run):
343+
make pre-release
344+
# 4. Tag + push; publish.yaml on GitHub Actions publishes to pub.dev:
345+
make release PKG=drive # or: core | s3 | box
41346
```
42347

43-
## History
348+
The tag pattern is `<package>-v<semver>` (e.g. `cloud_sync_drive-v0.1.1`). [`.github/workflows/publish.yaml`](.github/workflows/publish.yaml) parses the tag, re-runs analyze + test, verifies the pubspec version matches, and publishes via pub.dev OIDC trusted publishing — no long-lived secrets.
44349

45-
This repository supersedes [`drive_sync_flutter`](https://pub.dev/packages/drive_sync_flutter). That package is frozen at 1.2.0; new development lives here.
350+
---
46351

47352
## License
48353

49-
MIT
354+
MIT. See [`LICENSE`](LICENSE).

0 commit comments

Comments
 (0)