Hi,
I’m having an issue with my Tenstorrent N300 GPU instance where the kernel driver sees the device, but userspace tools cannot enumerate it.
Important: This was working fine a few days ago. The issue started after my account was suspended for verification. I completed the verification process and the account is now active, but the TT device detection has been broken since then.
What’s working:
- Kernel driver loads successfully: Loading Tenstorrent AI driver module v2.5.0
- Device is found: Found a Tenstorrent Wormhole device at bus 0000:02
- sysfs entry exists: /sys/class/tenstorrent/tenstorrent!0
- Card type correctly shows: n300
- PCI device accessible: vendor=0x1e52, device=0x401e
- Can mmap BAR0 and read hardware registers
What’s NOT working:
- pyluwen.detect_chips() returns
- pyluwen.pci_scan() returns
- ttnn.open_device(0) fails with: TT_FATAL: No chips detected in the cluster
- tt-smi shows: No Tenstorrent devices detected
Environment:
- TTNN version: 0.65.1rc16
- pyluwen version: 0.8.1
- Kernel: 6.1.62-tenstorrent-gpu
- Driver: tenstorrent v2.5.0
What I’ve tried:
- Multiple redeploys
- Creating /dev/tenstorrent0 device node manually (c 246 0)
- Setting up hugepages
- Various environment variables
The kernel sees the device but pyluwen/UMD cannot enumerate it at the userspace level. Any help appreciated!
Thanks!