S3 Baby-server is a file server by AWS-S3 protocol. It is designed to share existing files in a filesystem via S3. In contrast, most full-fledged servers store files in chunks (of manageable sizes) and are not adequate for this purpose. Baby-server is similar to "rclone serve s3".
Baby-server can be used in combination with "Lens3" to run multiple servers at a single http endpoint. See for Lens3 https://github.com/RIKEN-RCCS/lens3.
Prepare Golang. Then, build. It builds the binary "v1/s3-baby-server".
cd v1
make get
make
Or, less likely,
go install github.com/RIKEN-RCCS/s3-baby-server/v1@v1.3.1
./s3-baby-server serve 127.0.0.1:9000 ~/pool --cred s3baby,s3babybaby
where "~/pool" specifies a pool directory where buckets are created. Existing directories in the pool are considered as buckets. "--cred" specifies a credential pair separated by a comma (access-key,secret-access-key). A credential pair can be specified by the environment variable "S3BBS_CRED", too (in the same comma-separated pair format).
-
Object names cannot begin with a dot ("."). Such names are internally used as hidden metainfo.
-
Object names cannot be end with "/".
-
Bucket names cannot include any dots ("."). It is more restrictive compared to other S3 servers.
-
Object versions are not supported at all.
-
Copying by "CopyObject" is only allowed inside a bucket.
-
Symbolic links in a filesystem are not followed. It is an error when an object name (a path) includes symbolic links. It is to avoid files are stored in unexpected paths.
-
Baby-server does not return owner information. "Ower" in responses is always missing in ListObjects, etc. The value of query "fetch-owner" is ignored. Configuration on "accept_fetch_owner" changes "fetch-owner" to be an error.
-
Tags are not supported on buckets. Tags on a CreateBucket request are ignored.
-
ETags are always strong. ETags are MD5 and cost to calculate. So, Baby-server records an ETag in object's metainfo.
-
GetObjectAttributes returns no "ObjectParts" infomation. Baby-server does not retain parts information after finishing multi-part uploads.
-
Badly formatted http-date in http-headers "if-modified-since", "if-unmodified-since", and "x-amz-if-match-last-modified-time" invokes an error, although they should be ignored.
-
ContentType of a response is "binary/octet-stream". It would better be "application/octet-stream".
-
Files are created with the mode of process's umask. Set the umask before running Baby-server if neccessary.
-
The size of XML parameters passed in the message body is limited (2MB). Multipart uploads, object deletion, or setting tags may be affected. The size can be changed by configuration "xml_parameter_size_limit".
- Baby-server stores metadata information in a file "." + object-name + "@meta" in the same directory as an object. Metadata includes an ETag, a checksum value, tags, and meta-headers. Metainfo files will be created even by operations like listing objects.
-
Baby-server prints access logs to stdout by "-log-access" option.
-
Baby-server also stores access logs in the directory ".s3bbs/log" when it exists in the pool directory. The file name is ".s3bbs/log/access-log".
- Errors returned to a client do not contain information from OS such as directory paths or user id, because they can be something that should not be disclosed to a client.
An implementation note of Baby-server is design.md.
Restrictions on names for buckets and objects are stricter than AWS-S3, Google GCS, or MinIO.
Characters of bucket names are from the set "[a-z0-9-]". Bucket names should not start or end with a minus sign ("-"). DOTS (".") ARE FORBIDDEN.
Characters of object names are from the set "[a-zA-Z0-9!$&'()+,-./;=@_]" plus utf-8 characters.
Baby-server only distinguishes between ctime and mtime on buckets. It uses mtime for objects.
The mtime of an object may not correct when copying. As Baby-server performs simple copying by making a hard-link of a file, mtime is not updated.
Baby-server does not check the message digest in signing. It uses the given hash value without checking it.
Baby-server is based on 2019-03-27 Release of AWS-S3 API. Baby-server implements the following actions.
- AbortMultipartUpload
- CompleteMultipartUpload
- CopyObject
- CreateBucket
- CreateMultipartUpload
- DeleteBucket
- DeleteObject
- DeleteObjects
- DeleteObjectTagging
- GetObject
- GetObjectAttributes
- GetObjectTagging
- HeadBucket
- HeadObject
- ListBuckets
- ListMultipartUploads
- ListObjects
- ListObjectsV2
- ListParts
- PutObject
- PutObjectTagging
- UploadPart
- UploadPartCopy