On the other hand, linear probing, a standard transfer learning method, can sometimes become the best approach. We propose a log-likelihood ratio (LLR) approach to analyze the comparative benefits of ...

We first combine the pre-trained models and the linear head with the highest accuracy obtained on the linear probing experiment. You can look at the self-attention of the [CLS] token on the different ...

For example, the following command will automatically evaluate the models on K-NN and linear probing benchmark after the pre-training with student and teacher model distributed across 2 nodes: ...