Which *BERT? A Survey Organizing Contextualized Encoders

short explaination of long paper

Introduction

Goals of this survey paper

This paper is organized as follows:

Publicizing negative results in this area is very important because we need lot of compute power, time to train these models and to ensure evaluation reproducibility. Also, probing studies need to focus not only on models and tasks but also on pretraining data.

Questions raised for users of contextulized encoders.

Token Prediction

WILL BE UPDATED SOON.