NCCL tests¶
This example shows how to run distributed NCCL tests with MPI using dstack
.
Running as a task¶
Here's an example of a task that runs AllReduce test on 2 nodes, each with 4 GPUs (8 processes in total).
type: task
name: nccl-tests
nodes: 2
startup_order: workers-first
stop_criteria: master-done
env:
- NCCL_DEBUG=INFO
commands:
- |
if [ $DSTACK_NODE_RANK -eq 0 ]; then
mpirun \
--allow-run-as-root \
--hostfile $DSTACK_MPI_HOSTFILE \
-n $DSTACK_GPUS_NUM \
-N $DSTACK_GPUS_PER_NODE \
--bind-to none \
/opt/nccl-tests/build/all_reduce_perf -b 8 -e 8G -f 2 -g 1
else
sleep infinity
fi
# Uncomment if the `kubernetes` backend requires it for `/dev/infiniband` access
#privileged: true
resources:
gpu: nvidia:1..8
shm_size: 16GB
Default image
If you don't specify image
, dstack
uses its base Docker image pre-configured with
uv
, python
, pip
, essential CUDA drivers, mpirun
, and NCCL tests (under /opt/nccl-tests/build
).
Privileged
In some cases, the backend (e.g., kubernetes
) may require privileged: true
to access the high-speed interconnect (e.g., InfiniBand).
Apply a configuration¶
To run a configuration, use the dstack apply
command.
$ dstack apply -f examples/clusters/nccl-tests/.dstack.yml
# BACKEND REGION INSTANCE RESOURCES SPOT PRICE
1 aws us-east-1 g4dn.12xlarge 48xCPU, 192GB, 4xT4 (16GB), 100.0GB (disk) no $3.912
2 aws us-west-2 g4dn.12xlarge 48xCPU, 192GB, 4xT4 (16GB), 100.0GB (disk) no $3.912
3 aws us-east-2 g4dn.12xlarge 48xCPU, 192GB, 4xT4 (16GB), 100.0GB (disk) no $3.912
Submit the run nccl-tests? [y/n]: y
Source code¶
The source-code of this example can be found in
examples/clusters/nccl-tests
.
What's next?¶
- Check dev environments, tasks, services, and fleets.