Skip to content

Gyan output is empty apart from the headers #4

@rahul003

Description

@rahul003

I'm starting my job as follows

LD_PRELOAD=~/mpi-tools/mpi_t/gyan/libgyan.so mpirun -np 8 -hostfile hosts -mca plm_rsh_no_tree_spawn 1 -bind-to socket -map-by slot -x HOROVOD_HIERARCHICAL_ALLREDUCE=1 -x HOROVOD_FUSION_THRESHOLD=16777216 -x NCCL_MIN_NRINGS=4 -x LD_LIBRARY_PATH -x PATH -mca pml ob1 -mca btl ^openib -x TF_CPP_MIN_LOG_LEVEL=0 python -W ignore train_imagenet_resnet_hvd.py --synthetic -b 256 --num_batches 200 --clear_log

Output is

----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions