Viztracer + torch.compile never exits

[mwlon]

Here's a simple script that runs forever when run with viztracer -- <this file>

import torch

@torch.compile
def add(x, y):
  return x + y

It appears this is due to the torch compile inductor workers: I believe this is where they're spawned, so they only die on parent process exit. But viztracer waits for its children to die first, so we get stuck in an infinite loop.

[mwlon]

Maybe one solution would be to just track subprocesses recursively, and combine their traces once globally? That way we wouldn't need each python process to wait for its children to exit.

[gaogaotiantian]

Which Python version are you using? I can't repro it in 3.11. Are you using Windows/MacOS where the default context is spawn?

Maybe one solution would be to just track subprocesses recursively, and combine their traces once globally?

That's exactly how viztracer collects traces, but you need the process to exit to produce trace. The main process needs to wait for all its descendants to finish to collect everything.

[mwlon]

I'm on Linux and using

Python 3.10.13
torch 2.5.0
viztracer 1.0.2

So I believe the default start method is fork.

And here's what I see when the process is stuck:

 $ pstree 1711115
python─┬─python─┬─32*[python───{python}]
       │        └─3*[{python}]
       └─{python}

The 32* python processes are torch inductor workers. If you don't get a bunch of these when you launch torch, I'm guessing you have an older version of torch or some environment variable like TORCHINDUCTOR_COMPILE_THREADS=0 that tells torch inductor to run in-process instead. If you do get these processes, but it doesn't get stuck, then I'm confused as to what's different.

[gaogaotiantian]

I did trace multiple processes on my machine. I'm using 3.11 + torch 2.7.0 + viztracer 1.0.3.

[mwlon]

I confirmed this works in torch 2.7.0, so torch's behavior must have changed between 2.5.0 and 2.7.0. Hopefully you can reproduce this if you downgrade to 2.5.0.

Now this seems like a smaller issue, but it would still be nice to fix - it's a bit scary if viztracer can cause deadlocks depending on the program it's tracing.

[gaogaotiantian]

It's not really scary - it's normal. You can do it with any daemon process. If you don't explicitly shutdown your sub-process and rely on your main process to exit to bring down your sub-process, you'll have this issue. viztracer needs the main process to live to collect all the traces from sub-processes. If the program requries the other way around, that's a deadlock and it's not really solvable. I would guess that's what happened.

Some program would start a process pool and make it daemon so it will be automatically shutdown when the main process exits - I would consider that a bad design overall.

[gaogaotiantian]

It's possible this issue is fixed by #585 - I did not get the chance to try it out.