Unable to reproduce results on SimplerEnv

[Boltzmachine]

Hi, I try to run the inference on SimplerEnv by following the instruction, but get 0.3889 on apple_in_drawer/matching_avg, which is much lower than reported in the paper.

[arnoldland]

Hi, may I ask whether the results can be correctly reproduced on other tasks?

[Boltzmachine]

Yes the other 7 tasks are good.

[Boltzmachine]

Hi just a follow up on this issue. This is the only task that is more than 10% behind what was reported.

[Boltzmachine]

Do you really comment the last two envs out?

CogACT/sim_cogact/scripts/cogact_put_in_drawer_visual_matching.sh

Lines 8 to 12 in b174a1b

	declare -a env_names=(
	PlaceIntoClosedTopDrawerCustomInScene-v0
	# PlaceIntoClosedMiddleDrawerCustomInScene-v0
	# PlaceIntoClosedBottomDrawerCustomInScene-v0
	)

Or all three env names should be evaluated?

[arnoldland]

I apologize for the late response. I did indeed comment out the last two lines. I have rechecked, and the success rate for place apple is indeed 50.9%.

Different versions of the simulation environment and random seeds may lead to a few percentage points of variation, but the average success rate across all tasks remains very similar. However, a success rate of 39% does seem unusually low.