【英文】AI 全景报告 2020（178页）

47% of these implementations are based on PyTorch vs. 18% for TensorFlow. PyTorch offers greater flflexibility  and a dynamic computational graph that makes experimentation easier. JAX is a Google framework that is more  math friendly and favored for work outside of convolutional models and transformers.  Empirical scaling laws of neural language models show smooth power-law relationships, which means that as   model performance increases, the model size and amount of computation has to increase more rapidly.

PolyAI, a London-based conversational AI company, open-sourced their ConveRT model (a pre-trained contextual   re-ranker based on transformers). Their model outperforms Google’s BERT model in conversational applications,   especially in low data regimes, suggesting BERT is far from a silver bullet for all NLP tasks.GPT-3, T5, BART are driving a drastic improvement in the performance of transformer models for text-to-text   tasks like translation, summarization, text generation, text to code.