Programming Throwdown
Patrick Wheeler and Jason Gauci
188: World Models
Patrick Wheeler and Jason Gauci
Intro topic: Running
News/Links:
- Flow matching versus diffusion models
- OpenCV 5
- Claude fable beats pokemon with no harness
- Can the stockmarket swallow Anthropic, SpaceX and OpenAI?
Book of the Show
- Patrick
- Strength of the Few - James Islington
- Jason
- Descender - Jeff Lemire
Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h
Tool of the Show
- Patrick
- No Man’s Sky
- Jason
- Paperlib https://paperlib.app/en/
- Paperlib https://paperlib.app/en/
Topic: World Models
- Making decisions with AI
- Action-Value (called a Q model): What is the long-term value of making a decision at a position
- Policy: What action should I take (must be a distribution)
- Value (called a V model): What is the value of a position (depends on policy)
- Advantage/Disadvantage: difference in value given two policies
- When advantage is +, do that more.
- Model-Free
- Look at the current situation and suggest an action
- Run that action in the real world and measure the effect
- Use that measurement to suggest better actions next time
- Model-Based
- Observe rollouts (sequences of situations) and learn the dynamics
- Choose an action, use your dynamics model to measure the consequence
- Potentially do MPC (try many actions and choose the best)
- World Models
- Observe many many rollouts and learn a full forward model (how to create the input in the future)
- Train a policy & value inside the world model
- Deploy the policy and fine-tune based on the real world