I have recently been having issues with running the official Tenstorrent inference server. It turns out the Docker container doesn’t have enough access to the GPUs to properly use them in edge case scenarios. I’ve had some help from a staff member, and they told me this:
On Koyeb's tt-ssh container, the `/dev/tenstorrent/*` device nodes are coming up with mode 000 (`c---------`) instead of the expected 666 (`crw-rw-rw-`):
Koyeb tt-ssh: c--------- 1 root root 246, 0 ...
bare-metal: crw-rw-rw- 1 root root 510, 0 ...
This works for tools run directly inside tt-ssh (e.g. `tt-smi`) because root in that container can bypass file modes via CAP_DAC_OVERRIDE. But it breaks
nested-Docker workflows: Tenstorrent's inference server spawns the vLLM workload in a nested container via `docker run`, the nested container inherits the 000
mode bits, and it cannot open the devices — even with `--privileged`.
Fix needed on Koyeb's side: ensure `/dev/tenstorrent/*` device nodes are created with mode 666 (e.g. a udev rule on the host, or a chmod in the tt-ssh
entrypoint), so they are usable both by the outer tt-ssh container and any nested workloads.
If the Koyeb team could take a look at this, it would be greatly appreciated.