Repository navigation
Some questions about training #18
Description
Hello! Very great work. I have some problems with the reproduction process.
- is it normal to have a large number of nan values for loss values when you first start training.
- could you please tell me approximately how long an epoch takes when you train audio-visual information at the same time.
Activity
burchim commentedon Nov 26, 2024
Hi, thanks for using the repo.
The nans problem may be related to this issue
It may be fixed by computing attention scores with float32 precision.
It took us around 40 minutes per epoch when using 4 A100 GPUs, (see issue 9).
You may need to lower the batch size if your GPUs have less than 40GB of memory.