troubles when reproducing #2

Activity

1. NingYuanxiang commented on Jul 22, 2022

   

   NingYuanxiang

   on Jul 22, 2022

   Last edited by NingYuanxiang

   More actions

   Hi, thanks for the wonderful work.

   But I had some troubles when trying to reproduce the results with command:

   python3 projects/IDOL/train_net.py --config-file projects/IDOL/configs/ytvis19_r50.yaml --num-gpus 8 MODEL.WEIGHTS projects/IDOL/weights/cocopretrain_R50.pth SOLVER.IMS_PER_BATCH 16

   The results is 46.96, which is lower than provided results 49.5. I'm using Torch 1.9.0, and batchsize was set to 16 instead of 32.

   Is there something I missed? Looking forward to your reply.

   跑的这么快，几张卡啊大哥。STEPS、 MAX_ITER增大了吗？
2. HanGuangXin commented on Jul 23, 2022

   

   HanGuangXin

   on Jul 23, 2022

   Author

   More actions

   @NingYuanxiang I then doubled MAX_ITER as I cut IMS_PER_BATCH by half. The results is 0.4710, which is still much lower than provided results 49.5. I'll take a look at STEPS.
3. wjf5203 commented on Jul 23, 2022

   

   wjf5203

   on Jul 23, 2022

   Last edited by wjf5203

   Collaborator

   More actions

   Thanks for your attention!

   After seeing your issue, I conducted more experiments to provide some reference results, hoping it can help.
   First of all, in order to achieve the results reported in the paper, we need to infer at 480p resolution according to the description in Table 3, so please set 'MIN_SIZE_TEST' to 480.
   Directly reducing batchsize will result in insufficient usage of the dataset, so I have made three settings for IMS_PER_BATCH and MAX_ITER, and each setting is repeated five times.
   The settings and results are as follows:

   Original:
   IMS_PER_BATCH: 32
   STEPS: (8000,)
   MAX_ITER: 12000
   Results (mAP): [50.5, 49.4, 49.8, 49.7, 48.8 ]

   Setting1:
   IMS_PER_BATCH: 16
   STEPS: (8000,)
   MAX_ITER: 12000
   Results (mAP): [ 47.5, 47.1, 47.9, 47.4, 48.1]

   Setting2:
   IMS_PER_BATCH: 16
   STEPS: (16000,)
   MAX_ITER: 24000
   Results (mAP): [49.0, 47.9, 48.5, 49.1, 48.6]

   So, the reported results can be stably reproduced by default setting.
   If you reduce the batchsize from 32 to 16, you may need to adjust the number of iterations appropriately.
   Welcome to continue exploring the impact of different settings on training results and discuss here.
4. FoundationVision

   locked and limited conversation to collaborators on Jul 23, 2022
5. wjf5203

   converted this issue into a discussion #3 on Jul 23, 2022