Skip to content

API.md

Latest commit

 

History

History
335 lines (236 loc) · 9.13 KB

API.md

File metadata and controls

335 lines (236 loc) · 9.13 KB
Raw

FerroS3 API Reference

FerroS3 is an S3-compatible file proxy. It exposes configured local folders as buckets and serves objects directly from the filesystem.

Base URL

The server listens on the endpoint and port from config.yaml.

Example:

http://127.0.0.1:8080

Configuration Model

The runtime configuration is read from config.yaml in the current working directory.

Field Type Required Description
port integer yes TCP port to bind.
endpoint string yes Interface/address to bind.
verbose boolean yes Enables request logging in the auth middleware.
cache_size integer no In-memory stat cache size. Defaults to 10000.
auth object no Enables request authentication when present.
auth.access_key string yes, when auth exists Access key used by simple auth and SigV4.
auth.secret_key string yes, when auth exists Secret key used by SigV4.
buckets array yes Bucket-to-directory mapping.
buckets[].name string yes Public bucket name.
buckets[].storage string yes Local filesystem directory for the bucket.

Authentication

Authentication is optional until auth is configured in config.yaml.

When auth is configured, requests are accepted if they use one of these forms:

  1. Authorization: <access_key> for simple scripts and tests.
  2. AWS SigV4 header authentication with Authorization: AWS4-HMAC-SHA256 ....
  3. A presigned URL generated by POST /_admin/presign.
  4. HTTP Basic auth with access_key as the username and secret_key as the password.

SigV4 details

The implementation currently uses:

  • Region: us-east-1
  • Service: s3
  • Payload hash: UNSIGNED-PAYLOAD for presigned URLs
  • Signed headers: at least host for presigned URLs

Endpoint Summary

Method Path Purpose
GET / List configured buckets
HEAD /:bucket Check whether a bucket exists
GET /:bucket/ List objects in a bucket
GET /:bucket/*key Download an object
HEAD /:bucket/*key Fetch object metadata
PUT /:bucket/*key Upload or replace an object
DELETE /:bucket/*key Delete an object
POST /_admin/presign Generate a presigned URL

AWS CLI Example

FerroS3 works with S3-style clients. For the AWS CLI, use path-style addressing and point the client at the local endpoint.

aws configure set aws_access_key_id test_key
aws configure set aws_secret_access_key test_secret
aws configure set default.region us-east-1
aws configure set default.s3.addressing_style path

aws --endpoint-url http://127.0.0.1:8080 s3 ls
aws --endpoint-url http://127.0.0.1:8080 s3 cp ./local-file.txt s3://my-bucket/path/to/file.txt
aws --endpoint-url http://127.0.0.1:8080 s3 cp s3://my-bucket/path/to/file.txt ./downloaded-file.txt

GET /

Lists the buckets configured in config.yaml.

Response

  • 200 OK
  • Content-Type: application/xml

Example

curl -H "Authorization: test_key" http://127.0.0.1:8080/

Example XML

<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Owner>
    <ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID>
    <DisplayName>Owner</DisplayName>
  </Owner>
  <Buckets>
    <Bucket>
      <Name>my-bucket</Name>
      <CreationDate>2006-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
  </Buckets>
</ListAllMyBucketsResult>

HEAD /:bucket

Checks whether a bucket exists in the configured bucket map.

Path parameters

Name Type Description
bucket string Bucket name defined in config.yaml.

Responses

  • 200 OK when the bucket exists
  • 404 Not Found with NoSuchBucket when it does not

GET /:bucket/

Lists objects in a bucket.

Path parameters

Name Type Description
bucket string Bucket name defined in config.yaml.

Query parameters

Name Type Required Description
prefix string no Return only keys that start with this prefix.
delimiter string no Group keys by delimiter and emit CommonPrefixes.
marker string no Accepted for compatibility; echoed back in the response.
max-keys integer no Maximum number of objects to return. Defaults to 1000.
list-type integer no Accepts 2 for a simplified ListObjectsV2-style response.
continuation-token string no Accepted for compatibility; echoed back in the response.

Behavior notes

  • The listing is backed by filesystem traversal of the bucket directory.
  • delimiter splits results into CommonPrefixes.
  • max-keys stops after the requested number of objects.
  • list-type=2 is accepted, but the current implementation does not implement full S3 pagination semantics.
  • is_truncated, next_marker, and next_continuation_token are currently returned with fixed or empty values.

Example

curl -H "Authorization: test_key" \
  "http://127.0.0.1:8080/my-bucket/?prefix=logs/&delimiter=/&max-keys=100"

Example XML

<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>my-bucket</Name>
  <Prefix>logs/</Prefix>
  <Marker/>
  <MaxKeys>100</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>
</ListBucketResult>

GET /:bucket/*key

Downloads an object.

Path parameters

Name Type Description
bucket string Bucket name defined in config.yaml.
key string Object key, URL-encoded when needed.

Request headers

Header Required Description
Range no Supports bytes=start-end and bytes=start-.

Responses

  • 200 OK with the full object
  • 206 Partial Content when a valid Range header is supplied
  • 404 Not Found with NoSuchKey if the object is missing
  • 400 Bad Request if the object key cannot be decoded

Example

curl -H "Authorization: test_key" \
  -H "Range: bytes=0-99" \
  "http://127.0.0.1:8080/my-bucket/path/to/file.txt"

HEAD /:bucket/*key

Returns object metadata without the body.

Response headers

  • Content-Length
  • ETag
  • Last-Modified

Responses

  • 200 OK
  • 404 Not Found with NoSuchKey
  • 400 Bad Request if the key cannot be decoded

PUT /:bucket/*key

Uploads or replaces an object.

Request body

  • application/octet-stream

Behavior notes

  • Parent directories are created automatically.
  • Existing metadata cache entries for the object are invalidated.

Responses

  • 200 OK
  • 404 Not Found with NoSuchBucket if the bucket does not exist
  • 500 Internal Server Error if file creation or writes fail

Example

curl -X PUT \
  -H "Authorization: test_key" \
  --data-binary @local-file.txt \
  "http://127.0.0.1:8080/my-bucket/path/to/file.txt"

DELETE /:bucket/*key

Deletes an object.

Responses

  • 204 No Content
  • 404 Not Found with NoSuchBucket if the bucket does not exist
  • 400 Bad Request if the key cannot be decoded

Notes

  • The handler returns 204 even if the file is already missing.
  • Cache entries for the object are removed on delete.

Example

curl -X DELETE \
  -H "Authorization: test_key" \
  "http://127.0.0.1:8080/my-bucket/path/to/file.txt"

POST /_admin/presign

Generates a presigned URL for a bucket object.

Query parameters

Name Type Required Description
bucket string yes Target bucket name.
key string yes Object key.
method string yes HTTP method to sign, for example GET or PUT.
expires integer yes Expiration time in seconds.

Responses

  • 200 OK with a plain-text URL
  • 500 Internal Server Error if auth is not configured

Example

curl -X POST -H "Authorization: test_key" \
  "http://127.0.0.1:8080/_admin/presign?bucket=my-bucket&key=path/to/file.txt&method=GET&expires=300"

Example output

http://127.0.0.1:8080/my-bucket/path/to/file.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...&X-Amz-Date=...&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=...

Error Responses

All S3-style errors are returned as XML.

HTTP Status S3 Code Meaning
403 Forbidden AccessDenied Authentication failed or was missing.
404 Not Found NoSuchBucket Bucket is not configured.
404 Not Found NoSuchKey Object does not exist.
500 Internal Server Error InternalError Unhandled storage or server failure.

Example XML error

<Error>
  <Code>NoSuchKey</Code>
  <Message>The specified key does not exist.</Message>
  <Resource>path/to/file.txt</Resource>
  <RequestId>...</RequestId>
</Error>

What Is Not Implemented

The service intentionally keeps a small surface area.

  • Bucket creation and deletion endpoints are not exposed.
  • Multipart upload is not implemented.
  • ACL, lifecycle, replication, notification, and policy APIs are not implemented.
  • The ListObjects V2 implementation is simplified rather than fully S3-complete.