RELU before Linear layer in MLP in MetricLossOnly.ipynb example

[ajktym94]

Hi,

In the MLP class definition in the MetricLossOnly.ipynb example, I see a RELU layer being added before a Linear layer. My understanding is that an activation layer should always appear after a linear layer. Is the ReLU-before-Linear ordering intentional? It differs from the usual MLP convention.

class MLP(nn.Module):
    # layer_sizes[0] is the dimension of the input
    # layer_sizes[-1] is the dimension of the output
    def __init__(self, layer_sizes, final_relu=False):
        super().__init__()
        layer_list = []
        layer_sizes = [int(x) for x in layer_sizes]
        num_layers = len(layer_sizes) - 1
        final_relu_layer = num_layers if final_relu else num_layers - 1
        for i in range(len(layer_sizes) - 1):
            input_size = layer_sizes[i]
            curr_size = layer_sizes[i + 1]
            if i < final_relu_layer:
                layer_list.append(nn.ReLU(inplace=False))
            layer_list.append(nn.Linear(input_size, curr_size))
        self.net = nn.Sequential(*layer_list)
        self.last_linear = self.net[-1]

    def forward(self, x):
        return self.net(x)

[KevinMusgrave]

Thanks for pointing this out. You’re right that the MLP helper appends the ReLU before the Linear layer, which is surprising and not the usual MLP ordering.

In this notebook’s current usage, the embedder is instantiated with only one linear layer, so that ReLU branch is not actually used. However, the implementation is still misleading and would behave unexpectedly if hidden layers were added.