How to debug load_dataset?

[micklexqg]

I want to figure out the parameters of load_dataset and set breakpoints in load_dataset function , but it does not stop at any breakpoints, so "load_dataset" function can not be stepped into ?

[Hassanmahmood4]

load_dataset() itself is definitely debuggable, so if your breakpoints inside datasets/load.py are never hit, it's usually due to one of the following reasons:

1. You're debugging a different Python environment than the one actually executing the code.

   For example, check:

   import datasets
   print(datasets.__file__)

   and verify that it points to the same load.py file where you placed the breakpoints.

2. The debugger is configured to skip library code.

   In VS Code, make sure options such as:

   "justMyCode": false

   are enabled in your debug configuration so that you can step into third-party packages.

3. The call may be returning from cache before reaching the code path you're interested in.

   Try forcing execution through the full loading process:

   from datasets import load_dataset, DownloadMode
   
   dataset = load_dataset(
       "imdb",
       download_mode=DownloadMode.FORCE_REDOWNLOAD
   )

4. If multiprocessing (num_proc) is involved, execution may happen in child processes that your debugger isn't attached to.

   Try reproducing the issue with:

   num_proc=None

   or disabling multiprocessing entirely while debugging.

So in short, load_dataset() can absolutely be stepped into. I would first verify that datasets.__file__ matches the source file you're editing and check whether your debugger is skipping external library code via a setting like justMyCode=True.