installed 3007 TypeError: object NoneType can't be used in 'await' expression

[z900collector]

Upgraded 3006.7 to 3007 on both salt-master and a single salt-minion
Salt minion log is recording the following errors constantly

/var/log/salt/minion
2024-03-07 15:53:49,411 [salt.transport.zeromq:396 ][ERROR ][2127705] Exception while running callback
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 394, in consume
await callback(msg)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 484, in wrap_callback
await callback(decoded)
TypeError: object NoneType can't be used in 'await' expression

/var/log/salt/master
2024-03-07 15:54:59,908 [salt.loader.lazy :531 ][ERROR ][1084696] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'
2024-03-07 15:54:59,979 [salt.loader.lazy :531 ][ERROR ][1084696] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'
2024-03-07 15:54:59,980 [salt.loader.lazy :531 ][ERROR ][1084696] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'

Initially spotted when doing a manual salt-call
looked in /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/ and spotted an old file after upgrade:

-rw-r--r-- 1 root root 9832 Mar 7 14:12 user.cpython-310.pyc
-rw-r--r-- 1 root root 440 Mar 7 14:12 value.cpython-310.pyc
-rw-r--r-- 1 root root 14016 Feb 26 09:08 vault.cpython-310.pyc
-rw-r--r-- 1 root root 15913 Mar 7 14:12 verify.cpython-310.pyc
-rw-r--r-- 1 root root 14925 Mar 7 14:12 versions.cpython-310.pyc

Salt Version:
Salt: 3007.0

Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]

Dependency Versions:
cffi: 1.16.0
cherrypy: 18.8.0
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4

Salt Package Information:
Package Type: onedir

System Versions:
dist: oracle 8.9
locale: utf-8
machine: x86_64
release: 5.15.0-101.103.2.1.el8uek.x86_64
system: Linux
version: Oracle Linux Server 8.9

[doesitblend]

Can you try stopping your Salt master and minion, then run find /opt/saltstack/salt -type f -iname "*.pyc" -print -delete?Then start them back up. I believe that will resolve this issue.

[tjyang]

@doesitblend , thanks for the oneliner script to cleanup the .pyc.
I am still getting this error message from minion startup. Any suggestion ?

salt-minion[1567838]: [ERROR ] Publish server binding pub to /var/run/salt/minion/minion_event_0f4efbad65_pub.ipc ssl=None

[z900collector]

Tried the one liner and now the minion reports:

2024-03-07 22:32:33,860 [salt.transport.zeromq:396 ][ERROR ][1756164] Exception while running callback
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 394, in consume
await callback(msg)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 484, in wrap_callback
await callback(decoded)
TypeError: object NoneType can't be used in 'await' expression

The master reports (lots and lots of these):
2024-03-07 22:23:13,633 [salt.transport.tcp:311 ][WARNING ][1699260] TCP Publish Client encountered an exception while connecting to /var/run/salt/master/master_event_pub.ipc: StreamClosedError('Stream is closed'), will reconnect in 1 seconds - File "/bin/salt-run", line 11, in
sys.exit(salt_run())

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/scripts.py", line 456, in salt_run
client.run()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/cli/run.py", line 34, in run
ret = runner.run(full_return=True)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/runner.py", line 303, in run
ret = self._proc_function(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/mixins.py", line 546, in _proc_function
return instance.low(fun, low, full_return=full_return)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/client/mixins.py", line 383, in low
data["return"] = func(*args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in call
ret = self.loader.run(run_func, *args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1233, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1248, in _run_as
return _func_or_method(*args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/runners/state.py", line 310, in event
return statemod["state.event"](

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in call
ret = self.loader.run(run_func, *args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1233, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1248, in _run_as
return _func_or_method(*args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 2576, in event
with salt.utils.event.get_event(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 127, in get_event
return MasterEvent(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 928, in init
super().init(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 265, in init
self.connect_pub()

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 323, in connect_pub
self.subscriber = salt.utils.asynchronous.SyncWrapper(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 77, in init
self.obj = cls(*args, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 210, in ipc_publish_client
return publish_client(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 152, in publish_client
return salt.transport.tcp.PublishClient(

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/tcp.py", line 219, in init
super().init(opts, io_loop, **kwargs)

File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/base.py", line 398, in init
super().init()

[tjyang]

@z900collector , the one-liner worked for me to remove *.pyc files and I don't have your message above.
I only have following error message from "systemctl status salt-minion" per another ticket #66179

salt-minion[1567838]: [ERROR   ] Publish server binding pub to /var/run/salt/minion/minion_event_0f4efbad65_pub.ipc ssl=None

[doc75]

After running the oneliner on both master and minion, it is working again.
Nevertheless I can still see a lot of occurence like this one on minion side:

2024-03-10 19:22:50,833 [salt.transport.zeromq:396 ][ERROR   ][32857] Exception while running callback
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 394, in consume
    await callback(msg)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 484, in wrap_callback
    await callback(decoded)
TypeError: object NoneType can't be used in 'await' expression

But it does not prevent the minion from returning properly.

[ZsBT]

I'm facing the same issue, after deleting .pyc files the minions get online. However these error lines still flood the minion logs.

[tjyang]

rocky9t01a test VM upgraded from 3006.6 to 3007.0.
I thought next version 3006.8 will be latest one with this bug fix but latest one is 3007 without this bug fix.

[me@rocky9t01 salt]$ sudo salt -L rocky9t01 cmd.run "/usr/bin/tail -10 /var/log/salt/minion"
rocky9t01:
    2024-04-09 16:40:43,823 [salt.loaded.int.module.cmdmod:1347][ERROR   ][571240] Command 'systemctl' failed with return code: 1
    2024-04-09 16:40:43,823 [salt.loaded.int.module.cmdmod:1352][ERROR   ][571240] output: Job for salt-minion.service canceled.
    2024-04-09 16:40:44,840 [salt.transport.tcp:1405][ERROR   ][571328] Publish server binding pub to /var/run/salt/minion/minion_event_69362de666_pub.ipc ssl=None
    2024-04-09 16:40:49,491 [salt.transport.zeromq:396 ][ERROR   ][571328] Exception while running callback
    Traceback (most recent call last):
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 394, in consume
        await callback(msg)
      File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 484, in wrap_callback
        await callback(decoded)
    TypeError: object NoneType can't be used in 'await' expression
[me@rocky9t01 salt]$ 

Details

[me@rocky9t01 salt]$ sudo salt-minion -V Salt Version: Salt: 3007.0

Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]

Dependency Versions:
cffi: 1.16.0
cherrypy: 18.8.0
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4

Salt Package Information:
Package Type: onedir

System Versions:
dist: rocky 9.3 Blue Onyx
locale: utf-8
machine: x86_64
release: 5.14.0-362.18.1.el9_3.0.1.x86_64
system: Linux
version: Rocky Linux 9.3 Blue Onyx

[me@rocky9t01 salt]$

[SwimGeek]

find /opt/saltstack/salt -type f -iname "*.pyc" -print -delete

This worked, but only for two days, errors returned on three minions so far.

[aloosnetmatch]

Hi all.

I have the same issues with saltminion 3007.0-0

2024-03-12 10:27:41,901 [salt.transport.tcp:1410][ERROR ][16652] Publish server binding pub to 127.0.0.1:4510 ssl=None

Both on Linux as on windows

[SwimGeek]

Hi

I see there are comments that this is a 'log flood' issue. I think it's a bit more than that - my minions disconnect from the master.

[tjyang]

This issue has been opened for two weeks and it is still not gaining enough attention to be classified as bug.

[SwimGeek]

Hi, my minions seem to become slow to respond and commands seem to have timeouts.

When I delete the .pyc files all is fast and happy. A few days later things go slow and time out again.

Log when minion does not respond, doing basic test.ping:

2024-03-19 14:58:45,810 [salt.transport.zeromq:396 ][ERROR   ][817] Exception while running callback
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 394, in consume
    await callback(msg)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 484, in wrap_callback
    await callback(decoded)
TypeError: object NoneType can't be used in 'await' expression
2024-03-19 14:58:55,939 [salt.minion      :2763][ERROR   ][817] Timeout encountered while sending {'cmd': '_return', 'id': 'cpt-ter-blahblah1.atomic.ac', 'success': True, 'return': True, 'retcode': 0, 'jid': '20240319125840713889', 'fun': 'test.ping', 'fun_args': [], 'user': 'root', '_stamp': '2024-03-19T12:58:40.915616', 'nonce': '39f602421d8c4c9bb844bca46a7a8f83'} request