Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(js): Adds doc for context. #1843

Merged
merged 1 commit into from
Feb 5, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 172 additions & 0 deletions docs/context.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
# Passing information through context

When working with LLMs, there are often different categories of information
being handled simultaneously:

- **Input:** Information that is directly relevant to guide the LLM's response
for a particular call such as the text that needs to be summarized.
- **Generation Context:** Information that is relevant to the LLM but isn't
specific to the call, such as the current time or a user's name.
- **Execution Context:** Information that is important to the code surrounding
the LLM call but not to the LLM itself, e.g. a user's current auth token.

Genkit provides a consistent `context` object that can propagate generation and
execution context throughout the process. This context is made available to all
actions including [flows](flows), [tools](tool-calling), and
[prompts](dotprompt).

Context is automatically propagated to all actions called within the scope of
execution - context passed to a flow will be made available to prompts executed
within the flow. Context passed to the `generate()` method will be available to
tools called within the generation loop.

## Why is context important?

As a best practice, you should provide the minimum amount of information to the
LLM that it needs to complete a task. This is important for multiple reasons:

- The less extraneous information the LLM has, the more likely it is to perform
well at its task.
- If an LLM needs to pass around information like user or account ids to tools,
it can potentially be tricked into leaking information.

Context gives you a side channel of information that can be used by any of your
code but doesn't necessarily have to be sent to the LLM. As an example, it can
allow you to restrict tool queries to the current user's available scope.

## Context structure

Context must be an object, but its properties are yours to decide. In some
situations Genkit will automatically populate context. For example, when using
[persistent sessions](chat) the `state` property is automatically added to
context.

One of the most common uses of context is to store information about the current
user. We recommend adding auth context in the following format:

```js
{
auth: {
uid: "...", // the user's unique identifier
token: {...}, // the decoded claims of a user's id token
rawToken: "...", // the user's raw encoded id token
// ...any other fields
}
}
```

The context object can store any information that you might need to know somewhere
else in the flow of execution.

## Use context in an action

To use context within an action, you can access the provided context helper
that is automatically supplied to your function definition:

* {Flow}

```ts
const summarizeHistory = ai.defineFlow({
name: 'summarizeMessages',
inputSchema: z.object({friendUid: z.string()}),
outputSchema: z.string();
}, async ({friendUid}, {context}) => {
if (!context.auth?.uid) throw new Error("Must supply auth context.");
const messages = await listMessagesBetween(friendUid, context.auth.uid);
const {text} = await ai.generate({
prompt:
`Summarize the content of these messages: ${JSON.stringify(messages)}`,
});
return text;
});
```

* {Tool}

```ts
const searchNotes = ai.defineTool({
name: 'searchNotes',
description: "search the current user's notes for info",
inputSchema: z.object({query: z.string()}),
outputSchmea: z.array(NoteSchema);
}, async ({query}, {context}) => {
if (!context.auth?.uid) throw new Error("Must be called by a signed-in user.");
return searchUserNotes(context.auth.uid, query);
});
```
* {Prompt file}

When using [Dotprompt templates](dotprompt), context is made available with the
`@` variable prefix. For example, a context object of
`{auth: {name: 'Michael'}}` could be accessed in the prompt template like so:

```none
---
input:
schema:
pirateStyle?: boolean
---

{{#if pirateStyle}}
Avast, {{@auth.name}}, how be ye today?
{{else}}
Hello, {{@auth.name}}, how are you today?
{{/if}}
```

## Provide context at runtime

To provide context to an action, you pass the context object as an option
when calling the action.

* {Flows}

```ts
const summarizeHistory = ai.defineFlow(/* ... */);

const summary = await summarizeHistory(friend.uid, {context: {auth: currentUser}});
```

* {Generation}

```ts
const {text} = await ai.generate({
prompt: "Find references to ocelots in my notes.",
// the context will propagate to tool calls
tools: [searchNotes],
context: {auth: currentUser},
});
```

* {Prompts}

```ts
const helloPrompt = ai.prompt('sayHello');
helloPrompt({pirateStyle: true}, {context: {auth: currentUser}});
```

## Context propagation and overrides

By default, when you provide context it is automatically propagated to all
actions called as a result of your original call. If your flow calls other
flows, or your generation calls tools, the same context will be provided.

If you wish to override context within an action, you can pass a different
context object to replace the existing one:

```ts
const otherFlow = ai.defineFlow(/* ... */);

const myFlow = ai.defineFlow({
// ...
}, (input, {context}) => {
// override the existing context completely
otherFlow({/*...*/}, {context: {newContext: true}});
// or selectively override
otherFlow({/*...*/}, {context: {...context, updatedContext: true}});
});
```

When context is replaced it propagates the same way. In the above example,
any actions that `otherFlow` called during its execution would inherit the
overridden context.
Loading