Simulator for Tenstorrent architecture

This is a software simulator for the Tenstorrent architecture in large part based upon the Tenstorrent ISA Documentation. It currently provides an (incomplete) implementation of the Wormhole, but has been developed so that it can be easily enhanced and support a range of future architectures too. Note that it is not cycle accurate (or intended to be), although this could be added in the future. Instead, the idea was to try and create a software similator from the public documentation to enable developers to experiment with writing TT-Metal code without needing to run on physical hardware.

Optional diagnostic information can be provided by each RISC-V baby core, the NoC, the Tensix co-processor and memory, enabling tracing of the execution of a program. This is currently at the instruction and architectural state level, but could be enhanced in the future to provide feedback to developers around potential code bottlenecks or other issues.

This is written in Python, mainly to make it easy for people to hackaround and experiment with things. If you want to add some functionality, or fix a bug, then please feel free to go ahead and raise a PR. This is still work in progress, so also please raise issues etc as you find them!

Contents

• Installation
• Using tt-sim as a simulator for tt-metal
• Repository overview
• Key parts of the simulator

Installation

tt-sim is pure Python and requires Python ≥ 3.10. Clone the repository and install it in editable mode, which pulls in its dependencies (numpy, pyyaml, pynng, flatbuffers, pyarrow, pyelftools):

git clone https://github.com/mesham/tt-sim.git
cd tt-sim
pip install -e .           # add [dev] for the ruff + pytest tooling: pip install -e .[dev]

This installs the tt_sim simulator library. The example drivers live under driver/ and are run from the repository root, so keep the repo root on your PYTHONPATH when invoking them directly (the tt-metal flow below wires this up for you automatically):

export PYTHONPATH=$PWD:$PYTHONPATH

Using tt-sim as a simulator for tt-metal

tt-metal's UMD has a "simulation" chip backend: point TT_METAL_SIMULATOR at the driver/wormhole directory and UMD launches tt-sim in place of real silicon, so an ordinary tt-metal program runs against the simulator over a socket — no code changes, exactly as it would run on hardware. The minimal setup:

export TT_METAL_RUNTIME_ROOT=/path/to/tt-metal          # your built tt-metal checkout
export TT_METAL_SIMULATOR="$PWD/driver/wormhole"        # the switch: route execution to tt-sim
export TT_METAL_SLOW_DISPATCH_MODE=1                    # the only launch path the sim models
export LD_LIBRARY_PATH="$TT_METAL_RUNTIME_ROOT/build/lib:$LD_LIBRARY_PATH"

# then run any tt-metal program, e.g. an upstream example:
cd "$TT_METAL_RUNTIME_ROOT/build/programming_examples"
./metal_example_add_2_integers_in_compute

A set of ready-to-run example programs, plus a build-and-run test suite, lives under driver/wormhole/tests. For the full walkthrough — building the examples, the multi-tile TT_SIM_TENSIX_COORDS knob, diagnostics, structured tracing and the deadlock watchdog — see driver/wormhole/README.md.

Repository overview

The simulator implementation is in the tt_sim directory, with the driver directory providing a range of examples that illustrate running the simulator. These are individually documented, but to summarise:

• wormhole is where you likely want to go to. It provides an implementation of a Wormhole, currently with one DRAM tile and one Tensix block (although this is easy to expand, although will likely be slow!) It holds a set of example tt-metal programs that are built against a local tt-metal checkout and run against the simulator — exactly as they would run on hardware, only the device is the simulator. Each validates its own results, so the examples double as a test suite. See its README for how to build and run them.
• simple are very basic examples, demonstrating the memory subsystem and running codes on a vanilla RV32IM CPU.

Key parts of the simulator

There are a few of key components which are worth highlighting:

• tt_device.Wormhole creates the Wormhole, currently with a single DRAM tile and a tensix tile. Here you can see the NoC coordinates specified of each, and also the booleans provided to TensixTile are whether to report diagnostic information (the first five for each RISC-V baby core, then next two for each NoC and then the separate Tensix co-processor choices). In the wormhole tt-metal flow these are set from the TT_SIM_DIAG_* environment variables (see that directory's README).
• tt_device.TensixTile plumbs everything together within a Tensix tile, setting all the memory addresses and ranges for each individual component. These are all based on the ISA documentation memory map.
• tt_noc is the implementation of the NoC, it is not yet complete with all the functionality but is sufficient to communicate between tiles and, for example, read and write between DRAM and the Tensix tile.
• rv provides the RV32IM implementation with .ttinsn extension. This is fairly self explanatory, providing a pluggable approach to combining ISAs. The BabyRISCV ties this together for the baby RISC-V cores in the Tensix tile, for instance determining the initial PC value after a soft reset as per here.
• tensix is the implementation of the Tensix coprocessor as per here. This is not fully complete, but enough to run a range of codes that use the matrix, vector and scalar units (and all the associated unit implementation required to enable this).