Skimlit model improvement

[Rahul1758]

`Hey Daniel @mrdbourke,

I came across another research paper (https://arxiv.org/pdf/1909.04054.pdf) which achieves better results on PubMed using BERT. I read through the paper and want to implement the BERT+Transformer+CRF approach which gives an f1 score of 92.1%. I have a few doubts in Section 4 (Training and Implementation):

1. The way they have prepared the dataset is by joining sentences of abstract separated by ['SEP'] tag and then passing it to BERT. So I suppose each sample in dataset will be a string consisting of all sentences of an abstract joined together as mentioned above. Is my assumption correct?
2. I know how to use ['CLS'] output embedding of BERT for sentence classification. But how do I add a Transformed layer and CRF on this ['CLS'] token as per the paper?

Can you help me with this?
`

[mrdbourke]

Hey @Rahul1758,

It looks like they use a [SEP] token in between each sentence in an abstract rather than each actual abstract.

Then they use BERT to represent all the sentences in a document as one long sequence of words.

As for the model architecture, they share their code and data on GitHub: https://github.com/allenai/sequential_sentence_classification

I'd check that out for seeing how they added a transformer layer on top of the BERT representations.