upload-directory and webdav-server on linux

[cbertelli]

Hello,
thank you for providing this client. If you need a client on an old machine, it's not easy to get node installed while it's easy to find python 3.8+.
I got it running on RH8 without issues and with no hassle.
Then I tried to use it to upload entire directories from Google Drive to Internxt. I didn't succeed to get the work done as expected.
I found two issues:

• I can upload a file, but not a directory
• using rclone on top of webdav-server seems to work (from GDrive to Internxt directrly), but when I try to copy files and folders preserving dates, by using the --metadata flag, it stops because the webdav server is not able to handle this request.

Is there a way to overcome these limitations, maybe substituting the webdav server? Should I accept to lose metadata with Internxt?

[CrispStrobe]

thanks for your interest in this and your feedback.
latest changes include directory upload:

Usage: cli.py upload [OPTIONS] [SOURCES]...

  Encrypts and uploads local files/folders to a remote path.

  SOURCES... can be one or more local file or directory paths. Wildcards (*)
  are supported. Use --target to specify the destination folder (defaults to
  root /).

Options:
  -t, --target TEXT               Destination path on Internxt Drive (default:
                                  /)
  -r, --recursive                 Upload directories recursively
  --on-conflict [overwrite|skip]  Action if target exists (overwrite/skip)
  --help                          Show this message and exit.

i have not yet tested this thoroughly. please report errors you might encounter.

[cbertelli]

Thank you very much for the new feature. It works well, with a very detailed log.
Unfortunately it does not solve the issue about the modification date of the file. I don't know if the issue is due to the client or to the API itself.
If I evaluate the rationale of the process, I see the original file with its creation and modification date is substituted with a new locally encrypted file which has a new creation and modification date that is then transferred to Internxt drive.
This means the transferred file loses the original date that is substituted with the date of upload.
Obviously this is bad and unfortunately Internxt command line does the same thing. I hope you can "convince" the API to accept the original date for files and hopefully even for directories.

[CrispStrobe]

ah yes i think we can include timestamps in the payload (i just had not thought about that). current push includes this, -p parameter = preserve timestamps. it is experimental atm - i had no time to test this. so very helpful if you would check it.

[cbertelli]

I tried the current version. Thank you very much for the enhancement.
The environment is python 3.13.7 with virtual env.
I did all per the instructions.
At start it asked for the missing cheeroot module. Waitress was installed but it didn't notice it. After installation it run well. I attach the log of my upload

intrtnxt-cli-xx.log

maybe it took some more time to upload-then-add_metadata.

After loading, I verified the result and I couldn't detect if the data was changed.
/opt/internxt-cli/cli.py list-path -d oradiag_xx/diag/clients/user_xx/host_1389007998_107/trace
did not show the expected date, but the current date. Maybe this is only the upload date and the creation/modification date is available only via webdav.

So I started webdav (cheeroot – I had no choice even if waitress was installed), and I got:

🚀 Starting WebDAV server...
📡 Protocol: HTTPS
🔌 Port: 3005
⏰ Timeout: 0 minutes
🕐 Preserve Timestamps: Yes
🕐 InternxtDAVProvider initialized with timestamp preservation: True
🔍 WEBDAV: Initializing InternxtDAVProvider...
✅ WEBDAV: Provider initialized with credentials for: xx@mail.com
❌ Error starting WebDAV server: Invalid configuration:
  - Deprecated option 'enable_loggers': use 'logging.enable_loggers' instead.
Traceback (most recent call last):
  File "/opt/internxt-cli/services/webdav_server.py", line 173, in start
    app = WsgiDAVApp(config)
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/wsgidav_app.py", line 155, in __init__
    _check_config(config)
    ~~~~~~~~~~~~~^^^^^^^^
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/wsgidav_app.py", line 129, in _check_config
    raise ValueError("Invalid configuration:\n  - " + "\n  - ".join(errors))
ValueError: Invalid configuration:
  - Deprecated option 'enable_loggers': use 'logging.enable_loggers' instead.
❌ Failed to start server: Invalid configuration:
  - Deprecated option 'enable_loggers': use 'logging.enable_loggers' instead.

I added the required "logging." to /opt/internxt-cli/services/webdav_server.py and restarted webdav. I got:

🚀 Starting WebDAV server...
📡 Protocol: HTTPS
🔌 Port: 3005
⏰ Timeout: 0 minutes
🕐 Preserve Timestamps: Yes
🕐 InternxtDAVProvider initialized with timestamp preservation: True
🔍 WEBDAV: Initializing InternxtDAVProvider...
✅ WEBDAV: Provider initialized with credentials for: carlo.bertelli@gmail.com
✅ WebDAV server started successfully
🌐 URL: https://localhost:3005/
👤 Username: internxt
🔑 Password: internxt-webdav
🕐 Timestamp Preservation: Enabled

Timestamp Preservation in webdav is wonderful! (I was not able to verify them).
When trying to mount the webdav drive, I couldn't use https, but http instead. No problem, the credentials remain inside my machine.
The webdav server didn't run in background, so I got warnings and errors on screen:

14:51:21.185 - ERROR   : Traceback (most recent call last):
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/dav_provider.py", line 614, in get_properties
    value = self.get_property_value(name)
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/dav_provider.py", line 647, in get_property_value
    refUrl = self.get_ref_url()
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/dav_provider.py", line 389, in get_ref_url
    return quote(self.provider.share_path + self.get_preferred_path())
                 ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'share_path'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/error_printer.py", line 49, in __call__
    for v in app_iter:
             ^^^^^^^^
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/request_resolver.py", line 224, in __call__
    yield from app_iter
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/request_server.py", line 124, in __call__
    app_iter = provider.custom_request_handler(environ, start_response, method)
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/dav_provider.py", line 1619, in custom_request_handler
    return default_handler(environ, start_response)
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/request_server.py", line 361, in do_PROPFIND
    propList = child.get_properties("named", name_list=propNameList)
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/dav_provider.py", line 620, in get_properties
    if self.provider.verbose >= 2:
       ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'verbose'

14:51:21.187 - ERROR   : Caught HTTPRequestException(HTTP_INTERNAL_ERROR)
Traceback (most recent call last):
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/error_printer.py", line 82, in __call__
    raise as_DAVError(e) from None
wsgidav.dav_error.DAVError: 500

14:51:21.187 - ERROR   : e.src_exception:
'NoneType' object has no attribute 'verbose'
14:51:22.019 - ERROR   : Traceback (most recent call last):
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/dav_provider.py", line 614, in get_properties
    value = self.get_property_value(name)
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/dav_provider.py", line 647, in get_property_value
    refUrl = self.get_ref_url()
  File "/opt/internxt-cli/lib/python3.13/site-packages/wsgidav/dav_provider.py", line 389, in get_ref_url
    return quote(self.provider.share_path + self.get_preferred_path())
                 ^^^^^^^^^^^^^^^^^^^^^^^^

(this was repeated over and over).
Unfortunately the mounted disk seems empty. Maybe a less recent version of python works better. I will try with version 3.9, if you think this is better.

[CrispStrobe]

thank you for the excellent report. i fixed the list(-path) -d to show the preserved modified timestamp.
i cannot check the webdav issues right now but attempted some quick fixes on the fly. will come back to that asap.

[cbertelli]

Amazing! Now the files retain the dates.
I don't know if it is possible, but I think that for backups even the dates of the directories are mandatory. Can you do something even for these?

About the webdav-start, I get this error:

🚀 Starting WebDAV server...
📡 Protocol: HTTPS
🔌 Port: 3005
⏰ Timeout: 0 minutes
🕐 Preserve Timestamps: Yes
🕐 InternxtDAVProvider initialized with timestamp preservation: True
🔍 WEBDAV: Initializing InternxtDAVProvider...
✅ WEBDAV: Provider initialized with credentials for: carlo.bertelli@gmail.com
🔐 SSL certs not found, generating new ones...
🔐 Generating new SSL certificate for WebDAV server...
📁 Certificates saved to: /Users/carlo/.internxt-cli/webdav-ssl
✅ SSL certificate generated and saved successfully
📋 Certificate valid until: 2026-10-29 12:05:07 UTC
🔐 SSL (Waitress) enabled: /Users/carlo/.internxt-cli/webdav-ssl/cert.crt
✅ WebDAV server starting...
🌐 URL: https://localhost:3005/
👤 Username: internxt
🔑 Password: internxt-webdav
🕐 Timestamp Preservation: Enabled
🚀 Using server: waitress
❌ Error starting WebDAV server: Unknown adjustment 'ssl_certificate'
Traceback (most recent call last):
  File "/opt/internxt-cli/services/webdav_server.py", line 221, in start
    serve(
    ~~~~~^
        self.app,
        ^^^^^^^^^
    ...<3 lines>...
        ssl_private_key=str(NetworkUtils.WEBDAV_SSL_KEY_FILE),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/internxt-cli/lib/python3.13/site-packages/waitress/__init__.py", line 13, in serve
    server = _server(app, **kw)
  File "/opt/internxt-cli/lib/python3.13/site-packages/waitress/server.py", line 49, in create_server
    adj = Adjustments(**kw)
  File "/opt/internxt-cli/lib/python3.13/site-packages/waitress/adjustments.py", line 318, in __init__
    raise ValueError("Unknown adjustment %r" % k)
ValueError: Unknown adjustment 'ssl_certificate'
❌ Failed to start server: Unknown adjustment 'ssl_certificate'

I think that getting a certificate and setting a hard-coded username and password is somehow contradictory, even if SSL is much more than managing credentials.

[CrispStrobe]

made a few quick fixes, maybe that works for you now. you can choose by --server (waitress or cheroot) what to use. only cheroot supports ssl. we could i think make folders with timestamp payload. you need this for e.g. rsync over webdav? because upload -r should work anyway with our cli.

[cbertelli]

upload -r should work anyway with our cli

I did that, upload -r -p and I get the date of upload when listing with list-path -d. Maybe I do something wrong.

you need this for e.g. rsync over webdav?

Yes, it would be convenient to mount the remote directory via webdav to make temporary copies while working.

Thank you for -- server.

[cbertelli]

By the way, deleting a folder with a 3 level deep structure on webdav (waitress) takes a lot of time and a lot of work, with repetitive listings (minutes).

[CrispStrobe]

folders should now have the timestamps on upload creation.
as for the recursive delete, that might come from your webdav client walking the entire tree with PROPFINDs and DELETEs instead of a once DELETE of the top folder, plus we cannot change the <= 50 items pagination limit. we can increase cache timeout but i suspect that will not help much. would have to try though.

[cbertelli]

Thank you for the relentlesss development of this tool.
Now the folders are created with the proper date, but the files are not transferred. The procedure ends with this message:

========================================
📊 Upload Summary:
  ✅ Uploaded: 4
  ⏭️  Skipped:  0
  ❌ Errors:   258
========================================

The complete log is attached.

intrtnxt-cli-xx2.log

[CrispStrobe]

ah i added better caching in order to speed things up and probably overlooked a race condition. could you try that again?

[cbertelli]

I tried again after git-pulling (services/drive.py changed), Unfortunately I got the same errors.