feat: config file support

[xukai92]

tested by the code below:

from openai import OpenAI

from instructlab.sdg.default_flows import Flow, \
    SimpleFreeformSkillFlow, SimpleGroundedSkillFlow, SimpleKnowledgeFlow, \
    MMLUBenchFlow, SynthKnowledgeFlow, SynthSkillsFlow, SynthGroundedSkillsFlow

openai_api_key = "NO_API_KEY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)
model_family = "mixtral"
model_id = "Mixtral-8x7B-Instruct-v0.1"
num_instructions_to_generate = 10

# %%
flow_v0 = SimpleFreeformSkillFlow(client, model_family, model_id, num_instructions_to_generate).get_flow()
flow_v1 = Flow(client, None, None, num_instructions_to_generate).get_flow_from_file(
    "flows/simple_freeform_skill.yaml"
)
flow_v0 == flow_v1

# %%
flow_v0 = SimpleGroundedSkillFlow(client, model_family, model_id, num_instructions_to_generate).get_flow()
flow_v1 = Flow(client, None, None, num_instructions_to_generate).get_flow_from_file(
    "flows/simple_grounded_skill.yaml"
)
flow_v0 == flow_v1

# %%
flow_v0 = SimpleKnowledgeFlow(client, model_family, model_id, num_instructions_to_generate).get_flow()
flow_v1 = Flow(client, None, None, num_instructions_to_generate).get_flow_from_file(
    "flows/simple_knowledge.yaml"
)
flow_v0 == flow_v1

# %%
flow_v0 = MMLUBenchFlow(client, model_family, model_id, num_instructions_to_generate).get_flow()
flow_v1 = Flow(client, None, None, None).get_flow_from_file(
    "flows/mmlu_bench.yaml"
)
flow_v0 == flow_v1

# %%
flow_v0 = SynthKnowledgeFlow(client, model_family, model_id, num_instructions_to_generate).get_flow()
flow_v1 = Flow(client, None, None, None).get_flow_from_file(
    "flows/synth_knowledge.yaml"
)
flow_v0 == flow_v1

# %%
flow_v0 = SynthSkillsFlow(client, model_family, model_id, num_instructions_to_generate).get_flow()
flow_v1 = Flow(client, None, None, 10).get_flow_from_file(
    "flows/synth_skills.yaml"
)
flow_v0 == flow_v1

# %%
flow_v0 = SynthGroundedSkillsFlow(client, model_family, model_id, num_instructions_to_generate).get_flow()
flow_v1 = Flow(client, None, None, 10).get_flow_from_file(
    "flows/synth_grounded_skills.yaml"
)
flow_v0 == flow_v1

[xukai92]

is this correct? or we only want to do this for the first block in the flow?

[aakankshaduggal]

I see that you have removed the fields for teacher_model and "mixtral" could you please help me understand how it is interacting with the teacher model.

[xukai92]

"mixtral" is the model family and teacher_model is the model ID

• the model ID is defined in the YAML files as model_id
• the model family is derived from the model_id by the MODEL_FAMILY_MAP

[xukai92]

as of bd370b5, the CLI command below works

ilab data generate --endpoint-url http://localhost:3000/v1 --model mistralai/Mixtral-8x7B-Instruct-v0.1 --num-instructions 1 --api-key NO_API_KEY --model-family mixtral --pipeline full

[shivchander]

Tested this with @xukai92 - works as expected