728x90
→ Text Generation을 위한 자동 Evaluation Metric인 BERTScore 제시. Contextual Embedding으로 Reference Sentence와 Candidate Sentence간의 Token Similarity를 계산
기존 n-gram matching 방식
래퍼런스 문장과 후보 문장의 n그램을 비교. 정확한 위치와 n그램 길이에 제한적.
ex) BLEU, METEOR
- BERTScore
→ contextual embedding(BERT, ELMo) 은 word embedding 과 달리 문장에 따라 같은 단어여도 다른 vector representations를 생성.
- DATASET
WMT18 dataset. (en↔cs, en↔de, en↔et, en↔fi, en↔ru , en↔tr , en↔zh)
cs: 체코어, de: 독일어, et:에스토니안, fi: 핀란드어, ru:러시아어, tr:터키어, zh:중국어)
- RESULT
system level human judgement와의 피어슨 상관계수 (1에 가까울수록 상관 관계가 좋음)
Fbert가 Human judgement와 부합하는 결과를 보임
예시) 영어 문장 유사도 평가
| candidate | reference | f1 score |
| 28-year-old chef found dead in San Francisco mall | 28-Year-Old Chef Found Dead at San Francisco Mall | 0.9834 |
| A 28-year-old chef who recently moved to San Francisco was found dead in the staircase of a local shopping center. | A 28-year-old chef who had recently moved to San Francisco was found dead in the stairwell of a local mall this week. | 0.9782 |
| The victim's brother said he cannot imagine anyone who would want to harm him,"Finally, it went uphill again at him." | But the victim's brother says he can't think of anyone who would want to hurt him, saying, "Things were finally going well for him." | 0.9162 |
| The corpse, found Wednesday morning in the Westfield Mall, was identified as the 28-year-old Frank Galicia from San Francisco, the Justice Department said in San Francisco. | The body found at the Westfield Mall Wednesday morning was identified as 28-year-old San Francisco resident Frank Galicia, the San Francisco Medical Examiner's Office said. | 0.9589 |
| The San Francisco Police Department said the death was classified as murder and the investigation is on the running. | The San Francisco Police Department said the death was ruled a homicide and an investigation is ongoing. | 0.9675 |
| The victim's brother, Louis Galicia, told the ABS broadcaster KGO in San Francisco that Frank, who formerly worked as a cook in Boston, had his dream job as a cook at the Sons & Daughters restaurant in San Francisco six months ago. | The victim's brother, Louis Galicia, told ABC station KGO in San Francisco that Frank, previously a line cook in Boston, had landed his dream job as line chef at San Francisco's Sons & Daughters restaurant six months ago. | 0.9680 |
| A spokesman for the Sons & Daughters said they were "shocked and destroyed on the ground" over his death. | A spokesperson for Sons & Daughters said they were "shocked and devastated" by his death. | 0.9602 |
| "We are a small team that works like a close family and we are going to miss him painfully," said the spokesman. | "We are a small team that operates like a close knit family and he will be dearly missed," the spokesperson said. | 0.9663 |
| Our thoughts and condolences are in this difficult time at Franks's family and friends. | Our thoughts and condolences are with Frank's family and friends at this difficult time. | 0.9438 |
논문 : https://arxiv.org/pdf/1904.09675.pdf
깃헙 : https://github.com/Tiiiger/bert_score (영어) https://github.com/lovit/KoBERTScore (한국어)