Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix bug on node_id missing in local index #207

Merged
merged 1 commit into from
Jan 24, 2025
Merged

Fix bug on node_id missing in local index #207

merged 1 commit into from
Jan 24, 2025

Conversation

ansjcy
Copy link
Member

@ansjcy ansjcy commented Jan 24, 2025 •
edited
Loading

Description

We are attaching node id as we request for top n queries. This was a legacy / hacky method we use before having local indices support. Like other attributes, we should add node_id in the QueryInsightsListener as well.

Issues Resolved

#206

testing

  • run and get top n queries
{
  "top_queries" : [
    {
      "timestamp" : 1737679474829,
      "id" : "c86d4b2a-3946-42af-a2b5-95443f72925f",
      "search_type" : "query_then_fetch",
      "group_by" : "NONE",
      "total_shards" : 1,
      "node_id" : "6e-0X7wdSQWWx_04-GQwlQ",
      "phase_latency_map" : {
        "expand" : 0,
        "query" : 14,
        "fetch" : 0
      },
      "labels" : { },
      "task_resource_usages" : [
        {
          "action" : "indices:data/read/search[phase/query]",
          "taskId" : 299,
          "parentTaskId" : 298,
          "nodeId" : "6e-0X7wdSQWWx_04-GQwlQ",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 11078000,
            "memory_in_bytes" : 929520
          }
        },
        {
          "action" : "indices:data/read/search",
          "taskId" : 298,
          "parentTaskId" : -1,
          "nodeId" : "6e-0X7wdSQWWx_04-GQwlQ",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 99000,
            "memory_in_bytes" : 3720
          }
        }
      ],
      "source" : { },
      "indices" : [
        "top_queries-*"
      ],
      "measurements" : {
        "cpu" : {
          "number" : 11177000,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "latency" : {
          "number" : 16,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "memory" : {
          "number" : 933240,
          "count" : 1,
          "aggregationType" : "NONE"
        }
      }
    },
    {
      "timestamp" : 1737679421233,
      "id" : "6b6da258-1ddd-4642-ac50-74824b89db6a",
      "search_type" : "query_then_fetch",
      "group_by" : "NONE",
      "total_shards" : 1,
      "node_id" : "6e-0X7wdSQWWx_04-GQwlQ",
      "phase_latency_map" : {
        "expand" : 0,
        "query" : 8,
        "fetch" : 0
      },
      "labels" : { },
      "task_resource_usages" : [
        {
          "action" : "indices:data/read/search[phase/query]",
          "taskId" : 178,
          "parentTaskId" : 194,
          "nodeId" : "xGYdKsaBR8K-XXDBXBzcKg",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 1942000,
            "memory_in_bytes" : 33240
          }
        },
        {
          "action" : "indices:data/read/search",
          "taskId" : 194,
          "parentTaskId" : -1,
          "nodeId" : "6e-0X7wdSQWWx_04-GQwlQ",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 346000,
            "memory_in_bytes" : 3816
          }
        }
      ],
      "source" : {
        "size" : 20,
        "query" : {
          "term" : {
            "user.id" : {
              "value" : "cyji",
              "boost" : 1.0
            }
          }
        }
      },
      "indices" : [
        "my-index-0"
      ],
      "measurements" : {
        "cpu" : {
          "number" : 2288000,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "latency" : {
          "number" : 10,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "memory" : {
          "number" : 37056,
          "count" : 1,
          "aggregationType" : "NONE"
        }
      }
    },
    {
      "timestamp" : 1737679425513,
      "id" : "73995583-d59a-473a-8294-787e23cdbef9",
      "search_type" : "query_then_fetch",
      "group_by" : "NONE",
      "total_shards" : 1,
      "node_id" : "xGYdKsaBR8K-XXDBXBzcKg",
      "labels" : { },
      "phase_latency_map" : {
        "expand" : 0,
        "query" : 2,
        "fetch" : 0
      },
      "task_resource_usages" : [
        {
          "action" : "indices:data/read/search[phase/query]",
          "taskId" : 185,
          "parentTaskId" : 184,
          "nodeId" : "xGYdKsaBR8K-XXDBXBzcKg",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 1248000,
            "memory_in_bytes" : 33192
          }
        },
        {
          "action" : "indices:data/read/search",
          "taskId" : 184,
          "parentTaskId" : -1,
          "nodeId" : "xGYdKsaBR8K-XXDBXBzcKg",
          "taskResourceUsage" : {
            "cpu_time_in_nanos" : 281000,
            "memory_in_bytes" : 3816
          }
        }
      ],
      "source" : {
        "query" : {
          "term" : {
            "user.id" : {
              "value" : "cyji",
              "boost" : 1.0
            }
          }
        }
      },
      "indices" : [
        "my-index-0"
      ],
      "measurements" : {
        "latency" : {
          "number" : 4,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "cpu" : {
          "number" : 1529000,
          "count" : 1,
          "aggregationType" : "NONE"
        },
        "memory" : {
          "number" : 37008,
          "count" : 1,
          "aggregationType" : "NONE"
        }
      }
    }
  ]
}
  • wait for it to export, check local index
{
  "took" : 382,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "top_queries-2025.01.24-62114",
        "_id" : "MrHElZQBV-FvImauErJO",
        "_score" : 1.0,
        "_source" : {
          "timestamp" : 1737679425513,
          "id" : "73995583-d59a-473a-8294-787e23cdbef9",
          "search_type" : "query_then_fetch",
          "source" : {
            "query" : {
              "term" : {
                "user.id" : {
                  "value" : "cyji",
                  "boost" : 1.0
                }
              }
            }
          },
          "labels" : { },
          "group_by" : "NONE",
          "node_id" : "xGYdKsaBR8K-XXDBXBzcKg",
          "indices" : [
            "my-index-0"
          ],
          "total_shards" : 1,
          "phase_latency_map" : {
            "expand" : 0,
            "query" : 2,
            "fetch" : 0
          },
          "task_resource_usages" : [
            {
              "action" : "indices:data/read/search[phase/query]",
              "taskId" : 185,
              "parentTaskId" : 184,
              "nodeId" : "xGYdKsaBR8K-XXDBXBzcKg",
              "taskResourceUsage" : {
                "cpu_time_in_nanos" : 1248000,
                "memory_in_bytes" : 33192
              }
            },
            {
              "action" : "indices:data/read/search",
              "taskId" : 184,
              "parentTaskId" : -1,
              "nodeId" : "xGYdKsaBR8K-XXDBXBzcKg",
              "taskResourceUsage" : {
                "cpu_time_in_nanos" : 281000,
                "memory_in_bytes" : 3816
              }
            }
          ],
          "measurements" : {
            "memory" : {
              "number" : 37008,
              "count" : 1,
              "aggregationType" : "NONE"
            },
            "cpu" : {
              "number" : 1529000,
              "count" : 1,
              "aggregationType" : "NONE"
            },
            "latency" : {
              "number" : 4,
              "count" : 1,
              "aggregationType" : "NONE"
            }
          }
        }
      },
      {
        "_index" : "top_queries-2025.01.24-62114",
        "_id" : "RNDElZQB7SjLSe0aEcSf",
        "_score" : 1.0,
        "_source" : {
          "timestamp" : 1737679421233,
          "id" : "6b6da258-1ddd-4642-ac50-74824b89db6a",
          "search_type" : "query_then_fetch",
          "group_by" : "NONE",
          "total_shards" : 1,
          "node_id" : "6e-0X7wdSQWWx_04-GQwlQ",
          "phase_latency_map" : {
            "expand" : 0,
            "query" : 8,
            "fetch" : 0
          },
          "labels" : { },
          "task_resource_usages" : [
            {
              "action" : "indices:data/read/search[phase/query]",
              "taskId" : 178,
              "parentTaskId" : 194,
              "nodeId" : "xGYdKsaBR8K-XXDBXBzcKg",
              "taskResourceUsage" : {
                "cpu_time_in_nanos" : 1942000,
                "memory_in_bytes" : 33240
              }
            },
            {
              "action" : "indices:data/read/search",
              "taskId" : 194,
              "parentTaskId" : -1,
              "nodeId" : "6e-0X7wdSQWWx_04-GQwlQ",
              "taskResourceUsage" : {
                "cpu_time_in_nanos" : 346000,
                "memory_in_bytes" : 3816
              }
            }
          ],
          "source" : {
            "size" : 20,
            "query" : {
              "term" : {
                "user.id" : {
                  "value" : "cyji",
                  "boost" : 1.0
                }
              }
            }
          },
          "indices" : [
            "my-index-0"
          ],
          "measurements" : {
            "cpu" : {
              "number" : 2288000,
              "count" : 1,
              "aggregationType" : "NONE"
            },
            "latency" : {
              "number" : 10,
              "count" : 1,
              "aggregationType" : "NONE"
            },
            "memory" : {
              "number" : 37056,
              "count" : 1,
              "aggregationType" : "NONE"
            }
          }
        }
      },
      {
        "_index" : "top_queries-2025.01.24-62114",
        "_id" : "RdDElZQB7SjLSe0a-8TH",
        "_score" : 1.0,
        "_source" : {
          "timestamp" : 1737679474829,
          "id" : "c86d4b2a-3946-42af-a2b5-95443f72925f",
          "search_type" : "query_then_fetch",
          "group_by" : "NONE",
          "total_shards" : 1,
          "node_id" : "6e-0X7wdSQWWx_04-GQwlQ",
          "phase_latency_map" : {
            "expand" : 0,
            "query" : 14,
            "fetch" : 0
          },
          "labels" : { },
          "task_resource_usages" : [
            {
              "action" : "indices:data/read/search[phase/query]",
              "taskId" : 299,
              "parentTaskId" : 298,
              "nodeId" : "6e-0X7wdSQWWx_04-GQwlQ",
              "taskResourceUsage" : {
                "cpu_time_in_nanos" : 11078000,
                "memory_in_bytes" : 929520
              }
            },
            {
              "action" : "indices:data/read/search",
              "taskId" : 298,
              "parentTaskId" : -1,
              "nodeId" : "6e-0X7wdSQWWx_04-GQwlQ",
              "taskResourceUsage" : {
                "cpu_time_in_nanos" : 99000,
                "memory_in_bytes" : 3720
              }
            }
          ],
          "source" : { },
          "indices" : [
            "top_queries-*"
          ],
          "measurements" : {
            "cpu" : {
              "number" : 11177000,
              "count" : 1,
              "aggregationType" : "NONE"
            },
            "latency" : {
              "number" : 16,
              "count" : 1,
              "aggregationType" : "NONE"
            },
            "memory" : {
              "number" : 933240,
              "count" : 1,
              "aggregationType" : "NONE"
            }
          }
        }
      }
    ]
  }
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ansjcy ansjcy force-pushed the fix-node-id-missing branch from fc12420 to 8d9a3fe Compare January 24, 2025 00:48
@deshsidd
Copy link
Collaborator

LGTM

@ansjcy ansjcy merged commit bd6debd into opensearch-project:main Jan 24, 2025
12 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jan 24, 2025
@github-actions
Fix bug on node_id missing in local index (#207)
7c934ab 
Signed-off-by: Chenyang Ji <[email protected]>
(cherry picked from commit bd6debd)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Jan 24, 2025
Merged
ansjcy pushed a commit that referenced this pull request Jan 24, 2025
@opensearch-trigger-bot @github-actions
Fix bug on node_id missing in local index (#207) (#208)
cce34be 
(cherry picked from commit bd6debd)

Signed-off-by: Chenyang Ji <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deshsidd deshsidd deshsidd approved these changes

@jainankitk jainankitk Awaiting requested review from jainankitk jainankitk is a code owner

@dzane17 dzane17 Awaiting requested review from dzane17 dzane17 is a code owner

Assignees
No one assigned
Labels
backport 2.x
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ansjcy @deshsidd