S3 (Simple Storage Service)

Overview

DevCloud S3 provides basic object storage using the filesystem for object data and SQLite for metadata. Objects are stored as files on disk; bucket and object metadata (names, sizes, ETags, content types, timestamps) are tracked in a SQLite database with WAL mode enabled.

Supported APIs

These 43 operations are hand-verified, implemented by the provider, not by the CRUD engine. A sub-resource this provider does not serve (?lifecycle, ?encryption, …) returns a clean AWS error rather than being quietly answered as a bucket listing; fidelity-manifest.md is the per-operation answer.

Notification configuration is the one of those that has behaviour behind it: PutObject, CopyObject, CompleteMultipartUpload and DeleteObject emit ObjectCreated:* / ObjectRemoved:Delete events and deliver them to the configured SQS queue or Lambda function, which is what makes the S3 -> Lambda integration work.

UploadPartCopy uses the existing multipart upload ID, destination bucket/key and local account. It copies the source bytes, preserves the source object and returns CopyPartResult with a quoted MD5 ETag and LastModified. ListParts and completion use those same stored bytes and metadata, including after restart. A copied part does not emit an object-created notification; completion does.

The URL-encoded copy-source header is decoded once; literal plus signs, Unicode and encoded percent/question/hash characters remain part of the object key. CopySourceRange accepts one inclusive bytes=first-last range on sources larger than 5 MiB. Only the selected range is read. A copied part may be at most 5 GiB; the existing multipart completion permits small final parts.

ETag/date copy conditions are checked before replacing a part, with ETag taking precedence over its paired date condition. Quoted and existing unquoted local ETags are accepted. Owner mismatches fail with AccessDenied. Invalid part numbers, ranges and conditions leave previous parts intact. Plain and copied part writes share serialized, atomic file replacement; a metadata failure restores the prior file (or removes a newly created part). Completion, abort and part listing share the multipart lock, and part mutations recheck upload existence under that lock. Explicit UploadPartCopy requests with missing required fields cannot fall through to PutObject, CopyObject, UploadPart or CreateBucket.

Conditional copies hold a consistent source metadata/byte snapshot against local PutObject, CopyObject, completion and deletion. Object writers also restore prior bytes when their metadata write fails. These writes are serialized within the provider. Filesystem and SQLite writes are not a crash-atomic transaction, and persistent filesystem failure can prevent rollback.

Source version selectors, access points and encryption headers return NotImplemented rather than copying the current unencrypted local object. This operation inherits the existing single-account storage, filesystem key mapping, stored-only versioning/policies and in-memory buffering of the selected part.

Directory buckets and RenameObject

CreateBucket records a directory bucket only when its XML configuration specifies Bucket.Type=Directory, DataRedundancy=SingleAvailabilityZone and Location.Type=AvailabilityZone. The location name must match the bucket suffix --<zone>--x-s3; the local region is fixed to us-east-1. An ordinary bucket with a similar name remains ordinary. Bucket kind is immutable; configured recreation returns BucketAlreadyOwnedByYou. Directory deletion rejects remaining objects, orphan files and multipart uploads, and removes the incarnation’s sessions and rename receipts only on successful deletion.

CreateSession returns real random AccessKeyId, SecretAccessKey, SessionToken and Expiration fields. Sessions expire in five minutes and persist across normal restarts. The token digest, bucket incarnation, access key, mode and expiry are checked for supported directory data operations. ReadOnly permits GetObject, HeadObject, ListObjectsV2, ListParts and ListMultipartUploads; mutations require ReadWrite. HeadBucket and bucket control operations use the existing credentials; CopyObject and UploadPartCopy remain exempt from directory session admission. Multipart upload IDs must match the request bucket, key and local account before parts can be read, replaced, completed or aborted. An upload from a directory bucket cannot be accessed through an ordinary bucket or another bucket session. Explicit copy-part method conflicts return MethodNotAllowed; an empty copy header cannot exempt an ordinary object write from admission. Both the issued access key in Authorization and x-amz-s3session-token are required for protected requests. This adds local session admission; inherited SigV4 signatures and IAM/bucket policies are still not evaluated.

RenameObject requires a directory bucket and a ReadWrite session. It moves the existing file without reading its payload and preserves size, content type, ETag, LastModified and object tags. HeadObject returns StorageClass=EXPRESS_ONEZONE for directory objects. RenameSource is URL-decoded exactly once and accepts a key, leading slash plus key, or current bucket plus key. Unicode, plus signs and encoded percent/question/hash characters are literal key contents. Keys must be 1-1024 UTF-8 bytes with safe filesystem components and no trailing slash. Symlinks, path aliases and nonregular files are rejected. A same-key rename is a validated no-op. Success is HTTP 200 with an empty body; general buckets return InvalidRequest rather than performing PutObject or CopyObject.

SourceIfMatch, SourceIfNoneMatch, SourceIfModifiedSince and SourceIfUnmodifiedSince, and the four corresponding Destination conditions, are validated before mutation. Strong quoted ETags, unquoted local ETags, lists and standalone * are supported. HTTP dates compare at second precision. IfMatch suppresses its paired unmodified date; IfNoneMatch suppresses its paired modified date. Source and destination conditions are independent. A missing destination satisfies IfNoneMatch, including *; a failed condition returns PreconditionFailed and leaves both objects intact.

An optional ClientToken is 1-64 ASCII characters in the range 33-126 (no spaces). A successful receipt is persisted with the canonical keys and conditions, scoped to account and bucket incarnation. An equal retry returns 200 before checking source existence or conditions, even after objects change or disappear; changed parameters return IdempotencyParameterMismatch. Credentials and the token itself are excluded from the canonical request. Receipts have no TTL and live until successful bucket deletion. Failures do not consume a token. SDK-generated tokens follow this same contract.

One provider lock covers session admission and all native S3 requests, including reads, tags and multipart operations. Rename commits object rows, tags and receipt in one SQLite transaction. An overwritten destination is staged outside the object tree; a precommit error rolls SQL and file moves back. Recovery errors are joined with the original failure and returned as InternalError. Backup cleanup after a successful commit logs failures without changing success. Filesystem and SQLite updates are not crash atomic; persistent filesystem failures may prevent rollback, and multiple servers must not share the same data directory. Migration 12 adds directory/session/receipt tables without rewriting legacy rows.

LocalZone, source versions, access points and encryption options are unsupported; explicit encryption/session/rename errors do not fall through to generic success. There is no cloud zone lookup, encryption, CloudTrail or virtual-host DNS. Use a numeric loopback endpoint with path addressing so boto3’s S3 Express endpoint rules stay on the local server. The control client’s Config(s3={"addressing_style": "path", "disable_s3_express_session_auth": True}) uses existing test credentials for CreateBucket/CreateSession/DeleteBucket. The data client uses Config(s3={"addressing_style": "path"}) and automatically obtains its session; both use http://127.0.0.1:4747 and us-east-1. An explicitly issued mode client must send its issued token as x-amz-s3session-token without allowing an automatic replacement ReadWrite session. See the real boto3 tests.

boto3 Examples

Create a bucket and upload an object

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://localhost:4747",
    aws_access_key_id="test",
    aws_secret_access_key="test",
    region_name="us-east-1",
)

# Create bucket
s3.create_bucket(Bucket="my-bucket")

# Upload object
s3.put_object(Bucket="my-bucket", Key="hello.txt", Body=b"Hello, DevCloud!")

# Download object
response = s3.get_object(Bucket="my-bucket", Key="hello.txt")
print(response["Body"].read().decode())  # Hello, DevCloud!

List buckets and objects

# List all buckets
buckets = s3.list_buckets()
for b in buckets["Buckets"]:
    print(b["Name"])

# List objects in a bucket
objects = s3.list_objects(Bucket="my-bucket", Prefix="hello")
for obj in objects.get("Contents", []):
    print(obj["Key"], obj["Size"])

AWS CLI Examples

# Create bucket
aws --endpoint-url http://localhost:4747 s3 mb s3://my-bucket

# Upload file
aws --endpoint-url http://localhost:4747 s3 cp file.txt s3://my-bucket/

# List objects
aws --endpoint-url http://localhost:4747 s3 ls s3://my-bucket/

# Download file
aws --endpoint-url http://localhost:4747 s3 cp s3://my-bucket/file.txt ./downloaded.txt

Known Limitations

The pattern below is worth reading once: several configuration sub-resources round-trip faithfully but change nothing about how requests are served.

• Versioning is a stored status, not versions. PutBucketVersioning records Enabled, and GetBucketVersioning reads it back, but no version IDs are assigned and overwriting an object still destroys the previous one.
• ACLs and bucket policies are stored, never evaluated. No request is ever denied by one. There is no object lock.
• CORS configuration is stored, never applied. DevCloud does not emit Access-Control-* response headers or answer preflight requests from it.
• Presigned URLs are not verified. SigV4 signatures are not validated anywhere, so a presigned request is served like any other and an expired or forged one succeeds for ordinary buckets. Protected directory data operations additionally require the local session admission described above.
• No server-side encryption (SSE-S3, SSE-KMS): ?encryption is unimplemented
• No lifecycle policies or replication; directory HeadObject reports the fixed EXPRESS_ONEZONE class
• Single account model (account ID: 000000000000)