Unable to simulate large GEMMs using example experiment

[ncrago]

Hello,

I am trying to do some GEMM sweeps using the experiments/gemm.py script (via the docker container provided by the repo). When the GEMM sizes are large (e.g. MNK=16384), PyTorchSim crashes, and I am not sure how to proceed. My understanding is that the configuration (systolic_ws_128x128_c2_simple_noc_tpuv4.json) configures 32 channels x 8Gb (32GB total), and that the size should fit just fine. Can you please suggest next steps? Attached is the full log from the run.

Thank you,
Neal Crago

gemm16384.log

[YWHyuk]

Hello @ncrago

Thank you for reaching out and for your interest in PyTorchSim.

The crash you're seeing with large GEMM sizes (MNK=16384) is an issue in the current master image, specifically during the linking process. Fortunately, this has already been resolved in our latest development version.

Please try pulling the following image:

docker pull ghcr.io/psal-postech/torchsim-test-2-8:0f7f5425a489a11da35fb3970941c5b2a6561f87

Please let me know if you still encounter any issues with this version.

[ncrago]

Hi Wonhyuk,

Thanks for responding to my issue. I'm using the new docker image, and I cannot run the experiments/ - getting this error message when running "python experiments/layernorm.py" (and also for the gemm.py):

Traceback (most recent call last):
File "/workspace/PyTorchSim/experiments/layernorm.py", line 48, in
run_layernorm(size, config)
File "/workspace/PyTorchSim/experiments/layernorm.py", line 11, in run_layernorm
scheduler = Scheduler(num_request_queue=1, engine_select=Scheduler.FIFO_ENGINE, togsim_config=config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/PyTorchSim/Scheduler/scheduler.py", line 354, in init
self.tog_simulator.interactive_simulation()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'TOGSimulator' object has no attribute 'interactive_simulation'

Any help is appreciated.

Thanks,
Neal

[ncrago]

Just a followup on this issue. None of the experiments/ work in this new docker, though the tests/ work. There seems to be big changes in the simulator.py file, and no changes to the scheduler.py file. There's no longer interactive_simulation() or simulation() defined in simulator.py. The experiments/ all create schedulers, add work to them and run - which is now broken.

Can you please advise on what next steps should be to run gemm experiments?

Thanks,
Neal

[YWHyuk]

Hi @ncrago

Thank you for your patience, and sorry for the inconvenience. This was something I had missed. I really appreciate you taking the time to report it.

I've fixed the issue (1ae39bf). The scripts under experiments/ and experiments/artifact/ should now be runnable, and I've verified that they pass in CI(Link).

Also, please use the updated image below (the previously provided one had a bug affecting simulation cycle counts)

docker pull ghcr.io/psal-postech/torchsim-test-2-8:68b88238149aef2fc99dbc39858ad8f8921c7077

Please give it a try and let me know if you run into any issues. Sorry for all the bugs!