Citations when using a custom data source (eg. vector database)

[goncafab]

Hello,

I'm using a custom data source to retrieve data from a vector database (postgres with pgvector), however I need to display the citations used to generate the answer, which I'm not able to.

I already ran all the samples that mentioned citations, but they don't seem to be working on my end, for example, this the result of running the h.datasource-azureOpenAI sample, which highly differs from the example image in the sample folder:

Where could I inject the sources used to generate the answer? I don't need to have a custom action card, text at the bottom would be enough.

For reference, this is my render_data method, I'm using Python in my stack:

    async def render_data(self, _context: TurnContext, memory: Memory, tokenizer: Tokenizer, maxTokens: int):
        query = memory.get('temp.input')

        if not query:
            return Result('', 0, False)

        searchResults = self.searchClient.search(
            query,
            "similarity",
        )

        if not searchResults:
            return Result('', 0, False)

        usedTokens = 0
        doc = ''
        for result in searchResults:
            tokens = len(tokenizer.encode(result.page_content))

            if usedTokens + tokens > maxTokens:
                break

            doc += result.page_content
            usedTokens += tokens

        return Result(doc, usedTokens, usedTokens > maxTokens)

[mpriem]

+1 I have exactly the same requirements for a JS/TS solution where the LLM is behind a custom API. How can I reuse the code that AI library uses for citations?

[corinagum]

you may need to enable public preview on your teams (either at the user or administrator level) in order to see citations.

[mpriem]

I ended up writing my own converter class to map the output of our API to what msteams expects. It would be nice if the documentation provided some information about how the citations needed to be formatted. I now needed to reverse engineer it which took a long time.

something like this:

convertOutput(originalOutput, channelId) {
        let answerContent = originalOutput.answer;
        // Extract the answer content before the specified text and append the citations references
        if(channelId == 'msteams'){
            const citationRefs = originalOutput.context.map((_, index) => `[doc${index + 1}]`).join(' ');
            answerContent = originalOutput.answer.split("For more information refer to the source url below")[0].trim() + "\n\nSources:\n" + citationRefs;
        }
        else{
            const citationRefs = originalOutput.context.map((citation, index) => `[${index + 1}] ${citation.metadata.source}`).join('\n\n');
            answerContent = originalOutput.answer.split("For more information refer to the source url below")[0].trim() + "\n\nSources:\n\n" + citationRefs;
        }
        
        const newOutput = {
          status: 'success',
          input: originalOutput.input,
          message: {
            role: 'assistant',
            content: answerContent,
            end_turn: true,
            context: {
              citations: originalOutput.context.map((doc, index) => ({
                content: doc.page_content,
                title: doc.metadata.source,
                url: doc.metadata.source,
              })),
            },
          },
        };      
        return newOutput;
    }

[mpriem]

Thanks @corinagum. I did read that article, but couldn't make sense of it.

[corinagum]

Can you elaborate on what key pieces of information you needed? Your feedback isn't really helpful for us in terms of improving our documentation. What part didn't make sense? The code that handles the citations is indicated, as well as the interface for the citations format that's expected by the Teams client, so it's not clear to me why you needed to reverse-engineer it.

[mpriem]

Sorry about that. It seems I missed the interface part. I didn't find the formatting of the citations at the time I was looking into this and had to reverse engineer the formatting. Likely I was looking at the js files instead of the typescript files.

[goncafab]

@corinagum From my side I'm having a hard time understanding exactly where should I introduce the citations when using a custom data source, using the Python library.

[corinagum]

@goncafab You would need to extend the on_say definition, similar to how you would do it for TypeScript or C#.

https://github.com/microsoft/teams-ai/blob/main/python/samples/07.planners.a.langchain/src/bot.py#L80

Thanks for the question. I'll add a work item to add this to the Powered by AI documentation.