- ViTs are not as efficient as EfficientNet in terms of its number of parameters.
- DeiT suggested a new simple architecture that simply added an additional distillation token on top of ViT and utilized knowledge distillation technique to boost performance of ViTs. The effect is so significant that DeiT is now as efficient as EfficientNet.
- One interesting thing to note is that CNNs work better than transformers as a teacher model.
