Fine tuning "thinking"/"reasoning" models

[GhostDog98]

Say I have the following sample code:

from trl import SFTConfig, SFTTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")

training_args = SFTConfig(output_dir="sample")
trainer = SFTTrainer(
    args=training_args,
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
    train_dataset=dataset,
)
trainer.train()

This would seemingly present an issue, since the tags that it uses would be put into it's response, where I may not want them to be.
What is the recommended method of removing these? I could not find an option in SFTTrainer itself, but I am probably missing something...

For example, for the prompt "Complete this sentence: Chocolate is "
The model gives me an entire spiel:

<think>
Okay, so I need to complete the sentence "Chocolate is __." I'm not entirely sure what goes in the blank. Let me think about this. Chocolate is a common item people use a lot, so maybe it's something related to its uses or significance.

I remember that chocolate is often considered a sweet treat, but I also recall that it's used in various other contexts. Maybe it's not just about being sweet but also about something else. I think about how people might use chocolate in recipes. Like, maybe in baking or in recipes for cookies or candies. Or perhaps it's something about its properties, like being a natural ingredient or something.

Wait, I think chocolate is also called "sugar" in some contexts, especially in mathematical problems or when discussing things like integrals. But that might not be the right direction.

Another thought: chocolate is a common ingredient in many snacks and candies. Maybe the blank is about its versatility. Or perhaps it's about something else like its taste or texture.

I remember reading that chocolate is a versatile ingredient because it can be used for many things like making chocolate chip cookies, making chocolate bars, or even in desserts. So maybe the sentence should talk about its versatility.

Alternatively, I could think about the different types of chocolate—like dark, milk, white, etc.—but that doesn't seem directly related to the blank.

Wait, I think I've heard that chocolate is a sweetener. So maybe the sentence is about its use as a sweetener. But that seems too simple.

Alternatively, perhaps the blank is about its role in baking or in cooking. For example, chocolate is often used in baking to make things like cookies or cakes. So maybe the sentence is about how chocolate is used in baking.

But I'm not sure if that's the correct answer. Let me think again. Maybe the blank is about something more specific.

I also think about the fact that chocolate is often associated with being warm and rich, but that's not a sentence structure I can use here.

Another angle: in some contexts, chocolate is called "sugar." But that might not be the intended answer.

Wait, maybe the blank is about something else, like the shape or the texture. But I'm not sure.

I think I'll go with the idea that chocolate is a versatile ingredient because of its many uses. So the sentence would be "Chocolate is a versatile ingredient because it's used in many recipes, from cookies to candies." But I'm not sure if that's the most complete answer.

Alternatively, maybe the blank is about something like "sugar" as I thought before, but that seems too simple.

Wait, another thought: chocolate is a natural sweetener, so maybe the sentence is about that. But again, that's too simple.

I'm a bit stuck here. Maybe I should think about other words related to chocolate. For example, "milk chocolate," "white chocolate," "dark chocolate." But those are different uses.

Wait, I think I remember that chocolate is also called "sugar" in some contexts, especially in mathematical problems or when discussing things like integrals. But that might not be the right place to use it here.

Alternatively, maybe the blank is about something else like the taste or the texture. But I'm not sure.

I'm going to try to think of a more creative way to complete the sentence. Maybe something like "Chocolate is a versatile ingredient because of its many uses in baking and cooking."

But I'm not sure if that's the most appropriate answer. It might be too wordy.

Alternatively, maybe the blank is about something like "sugar," but I'm not confident.

Wait, perhaps the answer is simpler. Maybe it's about the fact that chocolate is sweet. So the sentence could be: "Chocolate is a sweet ingredient because of its versatility in being used in various baking and cooking recipes."

But again, I'm not sure if that's the best fit.

I think I'll have to make a choice. Since I can't think of anything else, I'll go with the idea that chocolate is a versatile ingredient because of its many uses in baking and cooking. So the sentence would be: "Chocolate is a versatile ingredient because it's used in many recipes, from cookies to candies."
</think>

Before outputting my desired result that I would like for fine tuning:
Chocolate is a versatile ingredient because it's used in many recipes, from cookies to candies.