How to handle FC layers with non-constant inputs?

[jinevening]

Let's discuss how to handle FC layers with non-const inputs.

Current status

--replace_non_const_fc_with_batch_matmul pass converts FC with non-const weights (regardless of whether ifm is const or not) into batch matmul. This is for ease of quantization (circle quantizer does not support FC with non-const weights).

• Before

ifm (const) -------+--> FC --> ofm
wgt (non-const)--+

• After

ifm (const) -------+--> BMM --> ofm
wgt (non-const)--+

In this way, ifm is quantized per-tensor, because it is treated as an activation tensor.

Alternative

We can do as follows.

1. Update --replace_non_const_fc_with_batch_matmul to do conversion only when ifm is non-const -> This will allow FC with const ifm and non-const weights.
2. Add a pass (e.g., --replace_non_const_fc_with_transposed_fc) which does the below conversion for FC with const ifm and non-const weights.

• Before

ifm (const) -------+--> FC --> ofm
wgt (non-const)--+

• After (Note that ifm and wgt are switched)

wgt (non-const) --> Transpose ----+--> FC --> Transpose -> ofm
ifm (const) --> Transpose ----------+

This will allow per-channel quantization of ifm, but Transpose Ops are added. So, the alternative can have a better accuracy with potential performance penalty (Transpose Ops).

If the Transpose Ops can be canceled out with adjacent operators, this idea seems cool.

The alternative was proposed by @parjong

CC @parjong @ejjeong

[jinevening]

After offline discussion, we decided to do the first item of the alternative.

The second item may be discussed later, when it can bring a clear benefit.