SPANBERT

• What is the name of the SpanBERT paper?

  SpanBERT: Improving Pre-training by Representing and Predicting Spans

  Mandar Joshi, FAIR

• How does SpanBERT modify BERT?
  • mask contiguous spans rather than random tokens
  • training span boundary representation to predict the entire content of the masked span
  • remove NSP
  • slight improvement on some tasks

  

• SpanBERT combines normal MLM loss and span boundary prediction loss
  • encourage boundary tokens to encode span information

  

• SpanBERT compares different span sampling schemes like geometric, noun phrases, named entities and whole words