Add RAGTool

[mingshl]

Description

The RAGTool is added to support agent framework. It's tested to support the current two types of agent: flow and conversational(ReAct) agents.

Here is the sample request to register RAGTools:

1. using flow agent:

POST /_plugins/_ml/agents/_register
{
  "name": "Test_Agent_For_RagTool",
  "type": "flow",
  "description": "this is a test flow agent for RAG",
  "tools": [
  {
    "type": "RAGTool",
    "description": "A description of the tool",
    "parameters": {
      "embedding_model_id": "<please input the embedding model id>",
      "model_id": "<please input the llm model id>",
      "index": "my_test_data",
      "embedding_field": "embedding",
      "source_field": [
        "text"
      ],
      "input": "${parameters.question}",
      "prompt": "\n\nHuman:You are a professional data analysis. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\n Context:\n${parameters.output}\n\nHuman:${parameters.question}\n\nAssistant:"
    }
  }
]
}

expected response:

{"agent_id": "xxxxxxxxxxxxx"}

you can execute agent and ask questions:

POST /_plugins/_ml/agents/<your_agent_id_from_above_response>/_execute
{
  "parameters": {
    "question": "Hello?",
    "verbose": true
  }
}

2. use conversational(ReAct) agent:

POST /_plugins/_ml/agents/_register
{
  "name": "Test_Agent_For_ReAct_RAG",
  "type": "conversational",  
  "description": "this is a test ReAct agent for RAG",
  "llm": {
    "model_id": "<please input the llm model id>",
    "parameters": {
      "max_iteration": 5,
      "stop_when_no_tool_found": true,
      "response_filter": "$.completion"
    }
  },
  "memory": {
    "type": "conversation_index"
  },
  "tools": [
  {
    "type": "RAGTool",
    "description": "A description of the tool",
    "parameters": {
      "embedding_model_id": "<please input the embedding model id>",
      "model_id": "<please input the llm model id>",
      "index": "my_test_data",
      "embedding_field": "embedding",
      "source_field": [
        "text"
      ],
      "input": "${parameters.question}",
      "prompt": "\n\nHuman:You are a professional data analysist. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\n Context:\n${parameters.output}\n\nHuman:${parameters.question}\n\nAssistant:"
    }
  },
  {
    "type": "CatIndexTool",
    "name": "RetrieveIndexMetaTool",
    "description": "Use this tool to get OpenSearch index information: (health, status, index, uuid, primary count, replica count, docs.count, docs.deleted, store.size, primary.store.size)."
  }
],
  "app_type": "my app"
}

expected response:

{"agent_id": "xxxxxxxxxxxxx"}

you can execute agent using the agent_id from above response and ask questions:

POST /_plugins/_ml/agents/xxxxxxxxxxxxx/_execute
{
  "parameters": {
    "question": "How many indexes do I have in my cluster?",
    "verbose": true
  }
}

expected response:

{
  "inference_results": [
    {
      "output": [
        {
          "name": "memory_id",
          "result": "uuuuuuuuuuuuuu"
        },
        {
          "name": "response",
          "result": "Thought: To find out how many indexes are in the cluster, we can use the RetrieveIndexMetaTool to get a list of indexes."
        },
        {
          "name": "response",
          "result": """
Observation: health    status    index    uuid    pri    rep    docs.count    docs.deleted    store.size    pri.store.size
yellow    open   ...
"""
        },
        {
          "name": "response",
          "result": "Based on the index information provided earlier, there are 16 indexes in the cluster."
        },
        {
          "name": "response",
          "result": "There are 16 indexes in the cluster."
        }
      ]
    }
  ]
}

you can continue the conversation by using the same agent id and pass the memory id from the above response in the parameter:

POST /_plugins/_ml/agents/xxxxxxxxxxxxx/_execute
{
  "parameters": {
    "memory_id": "uuuuuuuuuuuuuu",
    "question": "within the 16 indexes in the cluster, how many indices have yellow status in health?",
    "verbose": true
  }
}

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dhrubo-os]

As we need to merge codes to main branch, let's start adding tests to the PR as well.

[mingshl]

As we need to merge codes to main branch, let's start adding tests to the PR as well.

We want to test the RAG tools in the feature branch first, will add tests when merging to main branch.

[codecov]

Codecov Report

Attention: 131 lines in your changes are missing coverage. Please review.

Comparison is base (44118ef) 68.95% compared to head (50365af) 68.25%.
Report is 3 commits behind head on feature/agent_framework_dev.

Files	Patch %	Lines
...n/java/org/opensearch/ml/engine/tools/RAGTool.java	0.00%	129 Missing ⚠️
...nsearch/ml/engine/tools/AbstractRetrieverTool.java	0.00%	0 Missing and 1 partial ⚠️
...a/org/opensearch/ml/engine/tools/VectorDBTool.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                        Coverage Diff                        @@
##             feature/agent_framework_dev    #1747      +/-   ##
=================================================================
- Coverage                          68.95%   68.25%   -0.70%     
  Complexity                          2611     2611              
=================================================================
  Files                                241      242       +1     
  Lines                              12880    13011     +131     
  Branches                            1291     1306      +15     
=================================================================
  Hits                                8881     8881              
- Misses                              3393     3523     +130     
- Partials                             606      607       +1     

Flag	Coverage Δ
ml-commons	68.25% <1.50%> (-0.70%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mingshl]

tried to reproduce the build failure locally, but it runs successfully.

  ./gradlew :opensearch-ml-plugin:test     
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.1.1
  OS Info               : Linux 5.4.261-181.360.amzn2int.x86_64 (amd64)
  JDK Version           : 11 (Amazon Corretto JDK)
  JAVA_HOME             : /local/home/.jdks/corretto-11.0.18
  Random Testing Seed   : 907D3A1C7EDD3492
  In FIPS 140 mode      : false
=======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1m 11s
22 actionable tasks: 1 executed, 21 up-to-date