python/notebooks/09-groundedness-checking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f5c76c5f",
   "metadata": {},
   "source": [
    "# Groundedness Checking Plugins\n",
    "\n",
    "A well-known problem with large language models (LLMs) is that they make things up. These are sometimes called 'hallucinations' but a safer (and less anthropomorphic) term is 'ungrounded addition' - something in the text which cannot be firmly established. When attempting to establish whether or not something in an LLM response is 'true' we can either check for it in the supplied prompt (this is called 'narrow grounding') or use our general knowledge ('broad grounding'). Note that narrow grounding can lead to things being classified as 'true, but ungrounded.' For example \"I live in Switzerland\" is **not** _narrowly_ grounded in \"I live in Geneva\" even though it must be true (it **is** _broadly_ grounded).\n",
    "\n",
    "In this notebook we run a simple grounding pipeline, to see if a summary text has any ungrounded additions as compared to the original, and use this information to improve the summary text. This can be done in three stages:\n",
    "\n",
    "1. Make a list of the entities in the summary text\n",
    "1. Check to see if these entities appear in the original (grounding) text\n",
    "1. Remove the ungrounded entities from the summary text\n",
    "\n",
    "What is an 'entity' in this context? In its simplest form, it's a named object such as a person or place (so 'Dean' or 'Seattle'). However, the idea could be a _claim_ which relates concepts (such as 'Dean lives near Seattle'). In this notebook, we will keep to the simpler case of named objects.\n",
    "\n",
    "Let us first define our grounding text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23b26e2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "grounding_text = \"\"\"I am by birth a Genevese, and my family is one of the most distinguished of that republic.\n",
    "My ancestors had been for many years counsellors and syndics, and my father had filled several public situations\n",
    "with honour and reputation. He was respected by all who knew him for his integrity and indefatigable attention\n",
    "to public business. He passed his younger days perpetually occupied by the affairs of his country; a variety\n",
    "of circumstances had prevented his marrying early, nor was it until the decline of life that he became a husband\n",
    "and the father of a family.\n",
    "\n",
    "As the circumstances of his marriage illustrate his character, I cannot refrain from relating them. One of his\n",
    "most intimate friends was a merchant who, from a flourishing state, fell, through numerous mischances, into poverty.\n",
    "This man, whose name was Beaufort, was of a proud and unbending disposition and could not bear to live in poverty\n",
    "and oblivion in the same country where he had formerly been distinguished for his rank and magnificence. Having\n",
    "paid his debts, therefore, in the most honourable manner, he retreated with his daughter to the town of Lucerne,\n",
    "where he lived unknown and in wretchedness. My father loved Beaufort with the truest friendship and was deeply\n",
    "grieved by his retreat in these unfortunate circumstances. He bitterly deplored the false pride which led his friend\n",
    "to a conduct so little worthy of the affection that united them. He lost no time in endeavouring to seek him out,\n",
    "with the hope of persuading him to begin the world again through his credit and assistance.\n",
    "\n",
    "Beaufort had taken effectual measures to conceal himself, and it was ten months before my father discovered his\n",
    "abode. Overjoyed at this discovery, he hastened to the house, which was situated in a mean street near the Reuss.\n",
    "But when he entered, misery and despair alone welcomed him. Beaufort had saved but a very small sum of money from\n",
    "the wreck of his fortunes, but it was sufficient to provide him with sustenance for some months, and in the meantime\n",
    "he hoped to procure some respectable employment in a merchant's house. The interval was, consequently, spent in\n",
    "inaction; his grief only became more deep and rankling when he had leisure for reflection, and at length it took\n",
    "so fast hold of his mind that at the end of three months he lay on a bed of sickness, incapable of any exertion.\n",
    "\n",
    "His daughter attended him with the greatest tenderness, but she saw with despair that their little fund was\n",
    "rapidly decreasing and that there was no other prospect of support. But Caroline Beaufort possessed a mind of an\n",
    "uncommon mould, and her courage rose to support her in her adversity. She procured plain work; she plaited straw\n",
    "and by various means contrived to earn a pittance scarcely sufficient to support life.\n",
    "\n",
    "Several months passed in this manner. Her father grew worse; her time was more entirely occupied in attending him;\n",
    "her means of subsistence decreased; and in the tenth month her father died in her arms, leaving her an orphan and\n",
    "a beggar. This last blow overcame her, and she knelt by Beaufort's coffin weeping bitterly, when my father entered\n",
    "the chamber. He came like a protecting spirit to the poor girl, who committed herself to his care; and after the\n",
    "interment of his friend he conducted her to Geneva and placed her under the protection of a relation. Two years\n",
    "after this event Caroline became his wife.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73f4abc1",
   "metadata": {},
   "source": [
    "## Set up Semantic Kernel\n",
    "\n",
    "We prepare our kernel in the usual way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b22d324",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m pip install semantic-kernel==0.9.0.beta1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e13d3519",
   "metadata": {},
   "outputs": [],
   "source": [
    "import semantic_kernel as sk\n",
    "from semantic_kernel.connectors.ai.open_ai import (\n",
    "    AzureChatCompletion,\n",
    "    OpenAIChatCompletion,\n",
    ")\n",
    "\n",
    "kernel = sk.Kernel()\n",
    "\n",
    "useAzureOpenAI = False\n",
    "\n",
    "# Configure AI service used by the kernel\n",
    "if useAzureOpenAI:\n",
    "    deployment, api_key, endpoint = sk.azure_openai_settings_from_dot_env()\n",
    "    service_id = \"default\"\n",
    "    azure_chat_service = AzureChatCompletion(\n",
    "        service_id=service_id, deployment_name=\"turbo\", endpoint=endpoint, api_key=api_key\n",
    "    )  # set the deployment name to the value of your chat model\n",
    "    kernel.add_service(azure_chat_service)\n",
    "else:\n",
    "    api_key, org_id = sk.openai_settings_from_dot_env()\n",
    "    service_id = \"default\"\n",
    "    oai_chat_service = OpenAIChatCompletion(\n",
    "        service_id=service_id, ai_model_id=\"gpt-3.5-turbo\", api_key=api_key, org_id=org_id\n",
    "    )\n",
    "    kernel.add_service(oai_chat_service)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c65f786",
   "metadata": {},
   "source": [
    "## Import the Plugins\n",
    "\n",
    "We are going to be using the grounding plugin, to check its quality, and remove ungrounded additions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56ed7688",
   "metadata": {},
   "outputs": [],
   "source": [
    "# note: using plugins from the samples folder\n",
    "plugins_directory = \"../../samples/plugins\"\n",
    "\n",
    "groundingSemanticFunctions = kernel.import_plugin_from_prompt_directory(plugins_directory, \"GroundingPlugin\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d087993",
   "metadata": {},
   "source": [
    "We can also extract the individual semantic functions for our use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "738eb0ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "entity_extraction = groundingSemanticFunctions[\"ExtractEntities\"]\n",
    "reference_check = groundingSemanticFunctions[\"ReferenceCheckEntities\"]\n",
    "entity_excision = groundingSemanticFunctions[\"ExciseEntities\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5bda16e",
   "metadata": {},
   "source": [
    "## Calling Individual Semantic Functions\n",
    "\n",
    "We will start by calling the individual grounding functions in turn, to show their use. For this we need to create a same summary text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9a872f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_text = \"\"\"\n",
    "My father, a respected resident of Milan, was a close friend of a merchant named Beaufort who, after a series of\n",
    "misfortunes, moved to Zurich in poverty. My father was upset by his friend's troubles and sought him out,\n",
    "finding him in a mean street. Beaufort had saved a small sum of money, but it was not enough to support him and\n",
    "his daughter, Mary. Mary procured work to eke out a living, but after ten months her father died, leaving\n",
    "her a beggar. My father came to her aid and two years later they married when they visited Rome.\n",
    "\"\"\"\n",
    "\n",
    "summary_text = summary_text.replace(\"\\n\", \" \").replace(\"  \", \" \")\n",
    "print(summary_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a41eccb",
   "metadata": {},
   "source": [
    "Some things to note:\n",
    "\n",
    "- The implied residence of Geneva has been changed to Milan\n",
    "- Lucerne has been changed to Zurich\n",
    "- Caroline has been renamed as Mary\n",
    "- A reference to Rome has been added\n",
    "\n",
    "\n",
    "The grounding plugin has three stages:\n",
    "\n",
    "1. Extract entities from a summary text\n",
    "2. Perform a reference check against the grounding text\n",
    "3. Excise any entities which failed the reference check from the summary\n",
    "\n",
    "Now, let us start calling individual semantic functions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "071c05e4",
   "metadata": {},
   "source": [
    "### Extracting the Entities\n",
    "\n",
    "The first function we need is entity extraction. We are going to take our summary text, and get a list of entities found within it. For this we use `entity_extraction()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1d4b7ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "extraction_result = await kernel.invoke(\n",
    "    entity_extraction,\n",
    "    input=summary_text,\n",
    "    topic=\"people and places\",\n",
    "    example_entities=\"John, Jane, mother, brother, Paris, Rome\",\n",
    ")\n",
    "\n",
    "print(extraction_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b93c661f",
   "metadata": {},
   "source": [
    "So we have our list of entities in the summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "958ad1ff",
   "metadata": {},
   "source": [
    "### Performing the reference check\n",
    "\n",
    "We now use the grounding text to see if the entities we found are grounded. We start by adding the grounding text to our context:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "894e38d7",
   "metadata": {},
   "source": [
    "With this in place, we can run the reference checking function. This will use both the entity list in the input, and the `reference_context` in the context object itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d240d669",
   "metadata": {},
   "outputs": [],
   "source": [
    "grounding_result = await kernel.invoke(reference_check, input=extraction_result.value, reference_context=grounding_text)\n",
    "\n",
    "print(grounding_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a83c66f",
   "metadata": {},
   "source": [
    "So we now have a list of ungrounded entities (of course, this list may not be well grounded itself). Let us store this in the context:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35c1c329",
   "metadata": {},
   "source": [
    "### Excising the ungrounded entities\n",
    "\n",
    "Finally we can remove the ungrounded entities from the summary text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db82d97d",
   "metadata": {},
   "outputs": [],
   "source": [
    "excision_result = await kernel.invoke(entity_excision, input=summary_text, ungrounded_entities=grounding_result.value)\n",
    "\n",
    "print(excision_result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}