Memory profiling tool

The Lyft memory profiling tool is based on tracemalloc. To capture the memory trace for a given gunicorn process, we registered the worker process to listen to USR2 signal during the application initialization phase.

# app/__init__.py

MemoryProfiler().register_handlers()

# mem_profiler.py pseudo code

class MemoryProfiler: 
    def __init__(self) -> None:
        self._state_machine = self._profiling_state_machine()

     def register_handlers(self) -> None:
        # Register gunicorn worker to listen to USR2 to dump traces
        signal.signal(signal.SIGUSR2, self.handle_signal)  

    def handle_signal(self, signum: signal.Signals, frame: FrameType) -> None:
        next(self._state_machine)

    def _profiling_state_machine(self) -> Generator[None, None, None]:
        while True:
            try:
                self.start_tracing() # tracemalloc.start()
                self.memory_dump() # Create snapshot1
                yield
                self.memory_dump() # Create snaphot2,compare with snapshot1, and dump the difference in a file 
            finally:
                if tracemalloc.is_tracing():
                    tracemalloc.stop()

Let’s start the tracing!

Ok, now that we had the memory profiler setup, we are ready for some tracing to find the source of the leak. To start the tracing, we send USR2 signal to the gunicorn process in the K8s pod to start tracing and send the signal again after some time interval to capture the stack trace with highest memory usage.

ps aux

Now, we will send a USR2 signal to worker with pid 12

kill -USR2 12

Upon checking the process list again….

ps aux

… we observed that the gunicorn process we planned to trace got killed 🙁

It took several hours of debugging and a journey back to one of my favorite class to find the root of the issue — preload. To understand why preload caused the process to be killed, we first need to understand how gunicorn works.

Gunicorn

Gunicorn works on the pre-fork model. There is a leader process which forks a bunch of workers. There are two ways to fork the workers:

No Preload

When the leader process forks a worker, the worker has its own application code. This results in the worker process having a larger memory footprint than the leader.

smem -a - sort=pid -k

With Preload

Preload is a memory optimization based on the concept of copy-on-write. Essentially, the workers share the imports and application code with the leader and only modified pages are written to the worker’s memory.

smem -a - sort=pid -k

So how does preload play a role with USR2 signal killing the process?

If you remember, we registered the signal during the app initialization by calling register_handlers().

# app/__init__.py

MemoryProfiler().register_handlers()  


# mem_profiler.py

class MemoryProfiler:

    def register_handlers(self) -> None:
        # Register gunicorn worker to listen to USR2 to dump traces
        signal.signal(signal.SIGUSR2, self.handle_signal)

Since the app had preload=True, only the leader process was registering the USR2 signal to handle the tracing. The worker process did not register due to copy-on-write and that causes any kill -USR2 to actually kill the process!