8卡机器只有一张卡被使用

[KZF-kzf]

我启动提供的示例demo，Flow-Factory/examples/nft/lora/flux2_klein_base.yaml，我将其中的i2i数据换成了t2i"dataset/pickscore_custom"。其他未做任何修改。训练曲线看起来没什么问题，但是不管我怎么设置都只使用了单卡训练，请问您知道是什么问题吗？

[Jayce-Ping]

我这里都是正常的。我感觉是不是环境变量有问题呢？能否提供一下命令输出的前几行，ff-train启动时会有如下类似的输出：

[2026-03-12 20:01:28,880] [INFO] [flow_factory.cli]: ============================================================
[2026-03-12 20:01:28,880] [INFO] [flow_factory.cli]: Flow-Factory Launch Configuration
[2026-03-12 20:01:28,880] [INFO] [flow_factory.cli]:   Num machines:       1
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]:   Machine rank:       0
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]:   Num processes:      8
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]:   Master node:        localhost:29500
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]:   Mixed precision:    bf16
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]:   Accelerate config:  config/deepspeed/deepspeed_zero2.yaml
[2026-03-12 20:01:28,882] [INFO] [flow_factory.cli]: ============================================================
[2026-03-12 20:01:34,428] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0312 20:01:35.317000 2305 site-packages/torch/distributed/run.py:774] 
W0312 20:01:35.317000 2305 site-packages/torch/distributed/run.py:774] *****************************************
W0312 20:01:35.317000 2305 site-packages/torch/distributed/run.py:774] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0312 20:01:35.317000 2305 site-packages/torch/distributed/run.py:774] *****************************************
[2026-03-12 20:01:45] [Rank 0] [INFO] [flow_factory.hparams.training_args]: World Size:8
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: ====================================================================================================
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: Flow-Factory Training Initialized
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: World Size: 8
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: ====================================================================================================
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: Config: examples/grpo/lora/sd3_5_nocfg.yaml
[2026-03-12 20:01:45,232] [INFO] [flow_factory.train]: 

如果只跑了一张卡，在这里的相关部分应该会报如下warning:

    if num_procs > gpu_count * max(num_machines, 1) and gpu_count > 0:
        logger.warning(
            f"Requested {num_procs} processes but {num_machines} node(s) x "
            f"{gpu_count} GPU(s)/node = {num_machines * gpu_count} GPUs available."
        )

[daooshee]

For my case, in src/flow_factory/cli.py, cmd becomes
[sys.executable, "-m", "flow_factory.train", *script_args]
instead of
cmd = ["accelerate", "launch", "--num_processes", str(num_procs), "--num_machines", str(num_machines), "--machine_rank", str(machine_rank), "--main_process_port", str(main_process_port), "--mixed_precision", str(mixed_precision)].

The problem is in:
https://github.com/X-GenGroup/Flow-Factory/blob/main/src/flow_factory/cli.py#L164

Deleting os.environ.get("RANK") is not None solves the problem.

[KZF-kzf]

我这里都是正常的。我感觉是不是环境变量有问题呢？能否提供一下命令输出的前几行，ff-train启动时会有如下类似的输出：

[2026-03-12 20:01:28,880] [INFO] [flow_factory.cli]: ============================================================
[2026-03-12 20:01:28,880] [INFO] [flow_factory.cli]: Flow-Factory Launch Configuration
[2026-03-12 20:01:28,880] [INFO] [flow_factory.cli]: Num machines: 1
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]: Machine rank: 0
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]: Num processes: 8
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]: Master node: localhost:29500
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]: Mixed precision: bf16
[2026-03-12 20:01:28,881] [INFO] [flow_factory.cli]: Accelerate config: config/deepspeed/deepspeed_zero2.yaml
[2026-03-12 20:01:28,882] [INFO] [flow_factory.cli]: ============================================================
[2026-03-12 20:01:34,428] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
W0312 20:01:35.317000 2305 site-packages/torch/distributed/run.py:774]
W0312 20:01:35.317000 2305 site-packages/torch/distributed/run.py:774] *****************************************
W0312 20:01:35.317000 2305 site-packages/torch/distributed/run.py:774] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0312 20:01:35.317000 2305 site-packages/torch/distributed/run.py:774] *****************************************
[2026-03-12 20:01:45] [Rank 0] [INFO] [flow_factory.hparams.training_args]: World Size:8
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: ====================================================================================================
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: Flow-Factory Training Initialized
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: World Size: 8
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: ====================================================================================================
[2026-03-12 20:01:45,229] [INFO] [flow_factory.train]: Config: examples/grpo/lora/sd3_5_nocfg.yaml
[2026-03-12 20:01:45,232] [INFO] [flow_factory.train]:
如果只跑了一张卡，在这里的相关部分应该会报如下warning:

if num_procs > gpu_count * max(num_machines, 1) and gpu_count > 0:
    logger.warning(
        f"Requested {num_procs} processes but {num_machines} node(s) x "
        f"{gpu_count} GPU(s)/node = {num_machines * gpu_count} GPUs available."
    )

ff的输出是8个任务，但是实际只使用了一张卡。不过我现在在启动脚本前面加上unset RANK解决了这个bug
如：
unset RANK
HF_ENDPOINT=https://hf-mirror.com CUDA_VISIBLE_DEVICES=1 ff-train Flow-Factory/examples/nft/lora/flux2_t2i.yaml