Kernel: Out of memory

[goldenhub77]

Upon running the server and streaming/recording one camera for 10 minutes the server process crashes due to running out of memory. Upon reviewing the logs I am seeing the python3 main.py process incrementally grows by 200-700MB of memory until the process kills due to running out of system memory. Any thoughts?

kernel: [13104.512298] Out of memory: Killed process 52834 (python3) total-vm:14380312kB, anon-rss:13377752kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:27244kB oom_score_adj:0

[goldenhub77]

I am also noticing many ERROR:camera:Exception occured: timed out Trying to reconnect the camera stream. It always reconnects successfully, but continues to occur once every 10 seconds to sometimes once every 10 minutes. I am also noticing periodic ERROR:libav.h264:error while decoding MB 109 33, bytestream -16 but those are few and far between.

[goldenhub77]

@jdillenkofer You can replicate this issue when you open two concurrent streams on the same host and cancel loading one of them.

[goldenhub77]

@jdillenkofer See below for ERROR:Exception details:

Exception in thread CameraThread:
Traceback (most recent call last):
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/camera_proxy/src/camera_stream_manager.py", line 30, in start_camera
camera.start(lambda data: decoder.handle_packet(data))
File "/camera_proxy/src/camera.py", line 24, in start
self._start_stream(handle_video_stream)
File "/camera_proxy/src/camera.py", line 66, in _start_stream

File "/camera_proxy/src/baichuan_control_layer.py", line 178, in recv_packet
packet = self._recv_next_udp_packet()
File "/camera_proxy/src/baichuan_control_layer.py", line 143, in _recv_next_udp_packet
packet = self.udp_layer.recv_packet()
File "/camera_proxy/src/baichuan_udp_layer.py", line 112, in recv_packet
data, sender = self.socket.recvfrom(4096)
socket.timeout: timed out

[goldenhub77]

@jdillenkofer Directly after the Exception the additional following errors are thrown upon attempt to reconnect:

[goldenhub77]

ERROR:libav.h264:missing picture in access unit with size 40000
ERROR:libav.h264:No start code is found.
ERROR:libav.h264:Error splitting the input into NAL units.
[Errno 1094995529] Invalid data found when processing input; last error log: [h264] Error splitting the input into NAL units.

[jdillenkofer]

I don't know yet what those libav errors are, i have them too.
I think the program sometimes reads invalid data into the h264 stream.
Not sure where and why this happens.

But I found and fixed the memory leak. The problem was that i didn't remove the frame_callback from the decoder
when a stream is cancelled.
It is now fixed in the newest version in the develop branch.