There is new king in the NLP models! The Chinchilla NLP model beats GPT-3 and Gopher by optimizing model size. The Chinchilla is trained less parameters and more tokens. Let’s familiarize with it! Introduction The GPT-3 was major breakthrough in 2020. It triggered wave of large NLP models. I imagined…

The GPT-3 introduced new features called Insert and Edit. The features improve contextual knowledge and allows using GPT-3 in new ways. — The GPT-3: Edit The Edit is a feature, which instructs the way we want the GPT-3 to change our text. It allows quick modification of texts using written instruction. The new Edit-endpoint requires only one mandatory parameter called: Instruction. It defines the way the input-parameter needs to change. …

OpenAI trained Video PreTraining (VPT) model with 70k hours to play Minecraft. Scaled version may include +1M hours of video. — Introduction Video Pretraining (VPT) model learned to play Minecraft at human-level. The model crafts diamond tools. The game environment is: complex, open-ended and randomly generated. It uses the native human interface: keyboard, mouse and 20Hz frame rate. Such environment is more difficult for Reinforcement Learning (RL) agents.

The Video PreTraining (VPT) learns from unlabeled video to make sequential decisions. — Introduction The researchers created an agent capable of playing Minecraft at human-level. The pretraining required 70k hours of video. The VideoPretraining (VPT) technique is here! The technique is generic for any domain. It is not only limited to playing Minecraft. …

Google’s LIMoE scales efficiently with State-of-the-Art performance in image classification. — Introduction LIMoE is a multimodal image image classifier. It is Sparsely-activated. 5.6B parameters, 675M parameters per token Sparsely-activated Mixture of Experts (MoE) model Multimodal contrastive learning model. Transformer-based model The model offers State-of-the-Art level results, while scaling compute in efficient manner.

New neural theorem proving techniques lead towards AGI. — Introduction Mathematics is a challenge to AI research due to (a) infinite action space & (b) lack of self-play. Yet, the AGI implies the ability for mathematical reasoning. Lean, Metamath, and Equations are environments used for proving formal mathematical theorems. Such tools offer high-level tactics to apply to the problem…

Google Brains releases text-to-image generator with State-of-the-Art performance. — Introduction Imagen is the new State-of-the-Art text in text-to-image generation. It is better than GLIDE or DALL·E 2. It achieves FID score of 7.21 in zero-shot learning with the COCO dataset. Imagen is 4.6B parameter text-encoder based on T5-XXL. The text encoder uses text only corpus, not image-text pairs. It…