Skip to content

Integration testing with Testcontainers#294

Open
NathanBaulch wants to merge 7 commits intoC2FO:mainfrom
NathanBaulch:testcontainers
Open

Integration testing with Testcontainers#294
NathanBaulch wants to merge 7 commits intoC2FO:mainfrom
NathanBaulch:testcontainers

Conversation

@NathanBaulch
Copy link
Contributor

This project would benefit greatly from using Testcontainers to reliably provision ephemeral backend servers for integration testing. This makes it quick and easy to run integration tests against real servers running locally in Docker without the hassle of creating cloud accounts and managing credentials, etc.

This PR duplicates the two existing vfsintegration test suites and modifies them to run their tests against the following local Docker containers:

I've created this sub-module in a folder called "testcontainers" but happy to rename/move.
I've included two backends for S3 since both are quite popular and it's easy enough to test both.
All containers are created using the latest image versions, but there's no reason why they couldn't be pinned to one or more specific versions if needed.

Currently 15 of 340 tests are failing due to bugs/inconsistencies in certain backends. These should be fixed before this PR is merged.

@c2fo-cibot c2fo-cibot bot added the size/XXL Denotes a PR that changes 1000+ lines label Oct 13, 2025
@NathanBaulch NathanBaulch changed the title Integration testing with testcontainers Integration testing with Testcontainers Oct 17, 2025
@NathanBaulch
Copy link
Contributor Author

I was happy to see some work has gone into compliance recently, an area I've also been tackling in this PR. I've merged these improvements and after a few iterations have managed to get all CI checks passing, including the new testcontainers package tests. 🎉

However this PR includes some temporary fixes and is not yet ready to merge in its current state...

  • ftp: Exists should return true for the root / location. Previously it would try searching the parent folder (which makes no sense) to see if / exists (impossible).
  • az: Unlike all other backends, writing without first seeking causes a partial write because the remote file is re-downloaded into tempfile first. I've fixed this behavior to match the other backends.
  • az: Seeking a file that doesn't exist should throw os.ErrNotExist like every other backend.
  • ftp/sftp: These backends URI encode the paths returned from the location/file String func, which causes problems in the "MoveToFile with spaces in path" tests. I've hacked a workaround for now but a more permanent decision should be made on whether the URI and/or String funcs should be URI encoded.
  • ftp: The FTP protocol does not support partial writes, so "Write, Seek, Write" tests cannot be supported. I've hacked in a fix that follows the new ConformanceOptions pattern.
  • ftp: Set the SkipTouchTimestampTest flag since the fauria/vsftpd server doesn't support MDTM or MFMT.
  • Some backends return EOF when reading up to the end of a file (and no further). ExecuteSequence was not handling this non-error state correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XXL Denotes a PR that changes 1000+ lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant