Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
a47f699
feat(file/gcs): add GCS support with Go 1.25 and fix requested changes
Suryakantdsa Sep 3, 2025
d3dd786
added gcs filestore and some testcase
Suryakantdsa Aug 2, 2025
652914d
Added GCS FIlestore
Suryakantdsa Aug 2, 2025
8fafe1a
added testcase for Stat
Suryakantdsa Aug 2, 2025
e481b69
format code using goimports and fix lint issues via golangci-lint
Suryakantdsa Aug 3, 2025
0a2f863
fix lint issue
Suryakantdsa Aug 16, 2025
4528c4b
refactor: update docs, logging, and contribution guidelines based on …
Suryakantdsa Aug 21, 2025
0e468c5
test: add missing coverage for GCS
Suryakantdsa Oct 3, 2025
a367972
docs(gcs): add EndPoint field in config example
Suryakantdsa Oct 3, 2025
d1d73b4
Merge branch 'development' into feat/gcs-file-provider
Umang01-hash Oct 6, 2025
802e568
Merge branch 'development' into feat/gcs-file-provider
Umang01-hash Oct 8, 2025
7b1006c
Merge branch 'development' into feat/gcs-file-provider
Umang01-hash Oct 8, 2025
441b089
refactor(tests): remove if-else by splitting into separate test funct…
Suryakantdsa Oct 11, 2025
5dd78e1
Merge branch 'feat/gcs-file-provider' of https://github.com/Suryakant…
Suryakantdsa Oct 11, 2025
ec5ce82
feat(file/gcs): add GCS support with Go 1.25 and fix requested changes
Suryakantdsa Sep 3, 2025
f30f9a1
Added GCS FIlestore
Suryakantdsa Aug 2, 2025
d38cb41
added testcase for Stat
Suryakantdsa Aug 2, 2025
c2c6a31
format code using goimports and fix lint issues via golangci-lint
Suryakantdsa Aug 3, 2025
e43697e
fix lint issue
Suryakantdsa Aug 16, 2025
d81576f
refactor: update docs, logging, and contribution guidelines based on …
Suryakantdsa Aug 21, 2025
2d8782a
docs(gcs): add EndPoint field in config example
Suryakantdsa Oct 3, 2025
1d243d8
refactor(tests): remove if-else by splitting into separate test funct…
Suryakantdsa Oct 11, 2025
60aa36f
Merge branch 'development' into feat/gcs-file-provider
Umang01-hash Oct 13, 2025
23fe152
feat(file/gcs): add GCS support with Go 1.25 and fix requested changes
Suryakantdsa Sep 3, 2025
b820a8d
added gcs filestore and some testcase
Suryakantdsa Aug 2, 2025
2d49098
Added GCS FIlestore
Suryakantdsa Aug 2, 2025
64eaf78
added testcase for Stat
Suryakantdsa Aug 2, 2025
ad4e0b8
format code using goimports and fix lint issues via golangci-lint
Suryakantdsa Aug 3, 2025
f119426
fix lint issue
Suryakantdsa Aug 16, 2025
391facb
test: add missing coverage for GCS
Suryakantdsa Oct 3, 2025
b9d4d87
docs(gcs): add EndPoint field in config example
Suryakantdsa Oct 3, 2025
05d41fb
refactor(tests): remove if-else by splitting into separate test funct…
Suryakantdsa Oct 11, 2025
ea87e09
feat(file/gcs): add GCS support with Go 1.25 and fix requested changes
Suryakantdsa Sep 3, 2025
d026aeb
Added GCS FIlestore
Suryakantdsa Aug 2, 2025
cff2eb7
added testcase for Stat
Suryakantdsa Aug 2, 2025
4692870
format code using goimports and fix lint issues via golangci-lint
Suryakantdsa Aug 3, 2025
e084087
fix lint issue
Suryakantdsa Aug 16, 2025
006af26
refactor: update docs, logging, and contribution guidelines based on …
Suryakantdsa Aug 21, 2025
02b872f
docs(gcs): add EndPoint field in config example
Suryakantdsa Oct 3, 2025
e2fb9b3
refactor(tests): remove if-else by splitting into separate test funct…
Suryakantdsa Oct 11, 2025
dfc13b4
fix: handle nil metrics in Connect
Suryakantdsa Oct 13, 2025
071e2f9
fix: handle nil metrics in Connect
Suryakantdsa Oct 13, 2025
f8643ce
Merge branch 'development' into feat/gcs-file-provider
Umang01-hash Oct 14, 2025
f8e6fac
Merge branch 'development' into feat/gcs-file-provider
Umang01-hash Oct 15, 2025
fa26ed1
extract out logger and metrics and use them for diff file providers
Umang01-hash Oct 16, 2025
762c927
use LogFileoperation method to log and record metrics for gcs impleme…
Umang01-hash Oct 16, 2025
dda5b26
rename common Log method and resolve linters
Umang01-hash Oct 16, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ docker run -d --name db -p 8091-8096:8091-8096 -p 11210-11211:11210-11211 couchb
docker login container-registry.oracle.com
docker pull container-registry.oracle.com/database/free:latest
docker run -d --name oracle-free -p 1521:1521 -e ORACLE_PWD=YourPasswordHere container-registry.oracle.com/database/free:latest
docker run -it --rm -p 4443:4443 -e STORAGE_EMULATOR_HOST=0.0.0.0:4443 fsouza/fake-gcs-server:latest
```

> [!NOTE]
Expand Down Expand Up @@ -144,4 +145,4 @@ docker run -d --name oracle-free -p 1521:1521 -e ORACLE_PWD=YourPasswordHere con
- Use trailing white space or the <br> HTML tag at the end of the line.
- Use "`" sign to add single line code and "```" to add multi-line code block.
- Use relative references to images (in `public` folder as mentioned above.)
* The [gofr.dev documentation](https://gofr.dev/docs) site is updated upon push to `/docs` path in the repo. Verify your changes are live after next GoFr version.
* The [gofr.dev documentation](https://gofr.dev/docs) site is updated upon push to `/docs` path in the repo. Verify your changes are live after next GoFr version.
118 changes: 100 additions & 18 deletions docs/advanced-guide/handling-file/page.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,15 @@ GoFr simplifies the complexity of working with different file stores by offering

By default, local file-store is initialized and user can access it from the context.

GoFr also supports FTP/SFTP file-store. Developers can also connect and use their AWS S3 bucket as a file-store. The file-store can be initialized as follows:
GoFr also supports FTP/SFTP file-store. Developers can also connect and use their cloud storage bucket as a file-store. Following cloud storage options are currently supported:

- **AWS S3**
- **Google Cloud Storage (GCS)**

The file-store can be initialized as follows:

### FTP file-store

```go
package main

Expand All @@ -34,6 +40,7 @@ func main() {
```

### SFTP file-store

```go
package main

Expand All @@ -60,8 +67,7 @@ func main() {
### AWS S3 Bucket as File-Store

To run S3 File-Store locally we can use localstack,
``docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack``

`docker run --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack`

```go
package main
Expand Down Expand Up @@ -90,17 +96,76 @@ func main() {
app.Run()
}
```
> Note: The current implementation supports handling only one bucket at a time,
> as shown in the example with `gofr-bucket-2`. Bucket switching mid-operation is not supported.

> Note: The current implementation supports handling only one bucket at a time,
> as shown in the example with `gofr-bucket-2`. Bucket switching mid-operation is not supported.

### Google Cloud Storage (GCS) Bucket as File-Store

To run GCS File-Store locally we can use fake-gcs-server:
`docker run -it --rm -p 4443:4443 -e STORAGE_EMULATOR_HOST=0.0.0.0:4443 fsouza/fake-gcs-server:latest`

```go
package main

import (
"gofr.dev/pkg/gofr"

"gofr.dev/pkg/gofr/datasource/file/gcs"
)

func main() {
app := gofr.New()

// Option 1: Using JSON credentials with local emulator
app.AddFileStore(gcs.New(&gcs.Config{
EndPoint: "http://localhost:4566", // for LocalStack S3-compatible endpoint
BucketName: "my-bucket",
CredentialsJSON: readFile("gcs-credentials.json"),
ProjectID: "my-project-id",
}))

// Option 2: Using default credentials (GOOGLE_APPLICATION_CREDENTIALS)
// app.AddFileStore(gcs.New(&gcs.Config{
// BucketName: "my-bucket",
// ProjectID: "my-project-id",
// }))

// Option 3: Direct connection to real GCS (no EndPoint)
// app.AddFileStore(gcs.New(&gcs.Config{
// BucketName: "my-bucket",
// CredentialsJSON: readFile("prod-creds.json"),
// ProjectID: "my-project-id",
// }))

app.Run()
}

// Helper function to read credentials file
func readFile(filename string) []byte {
data, err := os.ReadFile(filename)
if err != nil {
log.Fatalf("Failed to read credentials file: %v", err)
}
return data
}

```

> **Note:** When connecting to the actual GCS service, authentication can be provided via CredentialsJSON or the GOOGLE_APPLICATION_CREDENTIALS environment variable.
> When using fake-gcs-server, authentication is not required.
> Currently supports one bucket per file-store instance.

### Creating Directory

To create a single directory

```go
err := ctx.File.Mkdir("my_dir",os.ModePerm)
```

To create subdirectories as well

```go
err := ctx.File.MkdirAll("my_dir/sub_dir", os.ModePerm)
```
Expand All @@ -114,27 +179,32 @@ currentDir, err := ctx.File.Getwd()
### Change current Directory

To switch to parent directory

```go
currentDir, err := ctx.File.Chdir("..")
```

To switch to another directory in same parent directory

```go
currentDir, err := ctx.File.Chdir("../my_dir2")
```

To switch to a subfolder of the current directory

```go
currentDir, err := ctx.File.Chdir("sub_dir")
```

> Note: This method attempts to change the directory, but S3's flat structure and fixed bucket
> make this operation inapplicable.
> make this operation inapplicable. Similarly, GCS uses a flat structure where directories are simulated through object prefixes.

### Read a Directory

The ReadDir function reads the specified directory and returns a sorted list of its entries as FileInfo objects. Each FileInfo object provides access to its associated methods, eliminating the need for additional stat calls.
The ReadDir function reads the specified directory and returns a sorted list of its entries as FileInfo objects. Each FileInfo object provides access to its associated methods, eliminating the need for additional stat calls.

If an error occurs during the read operation, ReadDir returns the successfully read entries up to the point of the error along with the error itself. Passing "." as the directory argument returns the entries for the current directory.

```go
entries, err := ctx.File.ReadDir("../testdir")

Expand All @@ -143,12 +213,13 @@ for _, entry := range entries {

if entry.IsDir() {
entryType = "Dir"
}
}

fmt.Printf("%v: %v Size: %v Last Modified Time : %v\n", entryType, entry.Name(), entry.Size(), entry.ModTime())
}
```
> Note: In S3, directories are represented as prefixes of file keys. This method retrieves file

> Note: In S3 and GCS, directories are represented as prefixes of file keys/object names. This method retrieves file
> entries only from the immediate level within the specified directory.

### Creating and Save a File with Content
Expand All @@ -163,14 +234,15 @@ _, _ = file.Write([]byte("Hello World!"))
```

### Reading file as CSV/JSON/TEXT

GoFr support reading CSV/JSON/TEXT files line by line.

```go
reader, err := file.ReadAll()

for reader.Next() {
var b string

// For reading CSV/TEXT files user need to pass pointer to string to SCAN.
// In case of JSON user should pass structs with JSON tags as defined in encoding/json.
err = reader.Scan(&b)
Expand All @@ -179,10 +251,12 @@ for reader.Next() {
}
```


### Opening and Reading Content from a File

To open a file with default settings, use the `Open` command, which provides read and seek permissions only. For write permissions, use `OpenFile` with the appropriate file modes.

> Note: In FTP, file permissions are not differentiated; both `Open` and `OpenFile` allow all file operations regardless of specified permissions.

```go
csvFile, _ := ctx.File.Open("my_file.csv")

Expand All @@ -205,6 +279,7 @@ if err != nil {
### Getting Information of the file/directory

Stat retrieves details of a file or directory, including its name, size, last modified time, and type (such as whether it is a file or folder)

```go
file, _ := ctx.File.Stat("my_file.text")
entryType := "File"
Expand All @@ -215,10 +290,12 @@ if entry.IsDir() {

fmt.Printf("%v: %v Size: %v Last Modified Time : %v\n", entryType, entry.Name(), entry.Size(), entry.ModTime())
```
>Note: In S3:

> Note: In S3 and GCS:
>
> - Names without a file extension are treated as directories by default.
> - Names starting with "0" are interpreted as binary files, with the "0" prefix removed.
>
> - Names starting with "0" are interpreted as binary files, with the "0" prefix removed (S3 specific behavior).
>
> For directories, the method calculates the total size of all contained objects and returns the most recent modification time. For files, it directly returns the file's size and last modified time.

### Rename/Move a File
Expand All @@ -234,18 +311,23 @@ err := ctx.File.Rename("old_name.text", "new_name.text")
### Deleting Files

`Remove` deletes a single file
> Note: Currently, the S3 package supports the deletion of unversioned files from general-purpose buckets only. Directory buckets and versioned files are not supported for deletion by this method.

> Note: Currently, the S3 package supports the deletion of unversioned files from general-purpose buckets only. Directory buckets and versioned files are not supported for deletion by this method. GCS supports deletion of both files and empty directories.

```go
err := ctx.File.Remove("my_dir")
```

The `RemoveAll` command deletes all subdirectories as well. If you delete the current working directory, such as "../currentDir", the working directory will be reset to its parent directory.
> Note: In S3, RemoveAll only supports deleting directories and will return an error if a file path (as indicated by a file extension) is provided.

> Note: In S3, RemoveAll only supports deleting directories and will return an error if a file path (as indicated by a file extension) is provided for S3.
> GCS handles both files and directories.

```go
err := ctx.File.RemoveAll("my_dir/my_text")
```

> GoFr supports relative paths, allowing locations to be referenced relative to the current working directory. However, since S3 uses
> a flat file structure, all methods require a full path relative to the S3 bucket.
> GoFr supports relative paths, allowing locations to be referenced relative to the current working directory. However, since S3 and GCS use
> a flat file structure, all methods require a full path relative to the bucket.

> Errors have been skipped in the example to focus on the core logic, it is recommended to handle all the errors.
1 change: 1 addition & 0 deletions go.work
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ use (
./pkg/gofr/datasource/elasticsearch
./pkg/gofr/datasource/file/ftp
./pkg/gofr/datasource/file/s3
./pkg/gofr/datasource/file/gcs
./pkg/gofr/datasource/file/sftp
./pkg/gofr/datasource/influxdb
./pkg/gofr/datasource/kv-store/badger
Expand Down
Loading
Loading