Reproducing REC results

[ryohachiuma]

Hi, @machuofan

I tried to reproduce REC results on the RefCOCOg dataset.
I downloaded the model weights from here and followed the instructions at here by running the following command.

python torchrun --nnodes=1 --nproc_per_node=8 groma/eval/eval_rec.py --model-name ./groma-7b-finetune --img-prefix /workspace/dataset/MSCOCO/train2017 --ann-file ./rec_refcocog_val.json

However, i got the results below. Am i missing sth?

rec_refcocog_val.json
[email protected] accu: 0.21956699346405228
m_iou: 0.20178459516537736
missing percentage: 0.0

Thanks in advance

[pengzhansun]

I got similar problems actually. And I found out that only use one GPU can get the correct results:

| distributed init (rank 0): env://
Using distributed mode: 1
| distributed init (rank 3): env://
| distributed init (rank 2): env://
| distributed init (rank 1): env://
loading annotations into memory...
Done (t=2.67s)
creating index...
index created!
refcoco_val.json
[email protected] accu: 0.4452649067749677
m_iou: 0.4139149429746631
missing percentage: 0.0

Versus

Using distributed mode: 1
| distributed init (rank 0): env://
loading annotations into memory...
Done (t=0.20s)
creating index...
index created!
refcoco_val.json
[email protected] accu: 0.8940372900129223
m_iou: 0.8308635033517168
missing percentage: 0.0