Chapter Plans

[natolambert]

Here is a rough outline of what I would like to see in the book, and who will be writing it.

Introductions & History

1. Introduction
2. Economics, Psychology, Philosophy of preference, etc.: VNM Theory, Bradley Terry, Impossibility theorems, social choice, etc
3. Optimal Control, Deep RL, ML etc.
4. RLHF for LLM lit (pre chatgpt stuff), maybe summarize instrugpt

Links:

• https://arxiv.org/abs/2310.13595

Problem Specification

1. Definitions, basic stuff, math
2. Preference data collection
3. Preference model training
4. KL constraints and other penalties

Policy Optimization

1. IFT / SFT / Chat Templates
2. Rejection Sampling / Best of N
3. PPO, REINFORCE, Policy Gradient
4. DPO (Eric, Archit, Rafael)
5. Other variants (short)

Advanced (optional)

1. CAI
2. Synthetic vs human data
3. Evaluation

Open Questions (TBD / optional)

1. Reward model over-optimization

[SalmanMohammadi]

Hey @natolambert. This is super awesome to see, and something which is sorely needed.

I'm wondering if you'd be open to contributors/collaborators? I have some examples of my writing in my blog, linked in my GitHub page. I've also authored a significant portion of torchtune's RLHF functionality.

[natolambert]

@SalmanMohammadi mostly looking for people to help with typos / formatting issues / other minor things, which is mostly service work (I'll send books to the top people), but adding other writers is too complex.