Here’s what caught our eye this past week.

FeatUp

• Use a multiview reconstruction loss to learn to upscale image features, for greater accuracy in downstream tasks like segmentation and depth estimation.
• Paper

EnvGen

• Use an LLM to generate environments that help RL models learn better.
• Paper
• Tweet

RAFT

• Finetune the LLM to ignore irrelevant information retrieved during RAG.
• Paper

When Do We Not Need Larger Vision Models?

• Use multi-resolution image features for improved vision and multimodal performance.
• Paper
• Code
• Tweet

Logits of API-Protected LLMs Leak Proprietary Information

• Extract information about an LLM’s architecture using just the output vector (logits).
• Paper

Evolutionary Optimization of Model Merging Recipes

• Uses evolutionary algorithms to merge the weights and architectures of foundation models.
• Paper

SceneScript

• Given a video of an environment, SceneScript outputs the commands necessary to make a 3D representation of the environment.
• Paper
• Announcement

LLM2LLM

• Use a large LLM to generate synthetic data tailored to help a smaller LLM address its weak spots during training.
• Paper
• Code

RewardBench

• A new benchmark for evaluating reward models, in the context of Reinforcement Learning from Human Feedback (RLHF).
• Code
• Paper

Reverse Training to Nurse the Reversal Curse

• Train on reversed sentences to fix the Reversal Curse.
• Paper

Vid2Robot

• A new method for generating robotic actions directly from videos of tasks.
• Project page
• Paper

DROID

• A new robot manipulation dataset with 350 hours of interaction data.
• Project page

MindEye2

• Reconstruct what a person is seeing with 1 hour of fMRI data.
• Paper
• Project page

PERL

• Use LoRA to train the reward model and policy model in RLHF.
• Paper

Grok-1

• xAI open-sourced Grok-1, a 314B parameter model.
• Announcement
• Code

Inflection AI CEO leaving for Microsoft

• A co-founder of Inflection AI is leaving for Microsoft.
• Inflection AI raised $1.3 billion last June.
• Announcement

Stable Video 3D:

• Stability AI releases Stable Video 3D, a new image-to-3D model.
• Announcement

Runway + Musixmatch

• Partnership to allow artists to make lyric music videos using Runway.
• Announcement

Suno v3

• Startup releases a new version of their AI music generator. It’s safe to say musicians are here to stay, for now.
• Announcement + sample

Stay up to date

Interested in future weekly updates? Stay up to date by joining our Slack Community!