The residual part in the decoder was perhaps implemented wrong?

[eduOS]

This is that in the official implementaion:

        # residual
        x = (x + residual) * math.sqrt(0.5)

And the following is that in this project:

# add res connections

next_layer += (next_layer + res_inputs) * tf.sqrt(0.5)

Am I right?

[crazygirlfym]

can you explain why * tf.sqrt(0.5) in residual operation

[eduOS]

@crazygirlfym The 3.2.1 of Attention Is All You Need may provide you some hints.

[StevenWD]

@eduOS Have you figure out why we use "+ =" instead of "+"? Thanks.

[eduOS]

@StevenWD Nein, leider nicht.

But I got the expected result by dropping the additional "+". I have not tried the +=, and don't know if it will be better. You can try to compare these two methods.