Fix sort on get trace API and flaky test of conversation

[Hailong-am]

Description

1. Fix sort on get trace Api, in the case two traces with create time are "create_time": "2023-12-05T05:14:42.823172Z" and "create_time": "2023-12-05T05:14:42.823833Z" they are equally same date due to mapping field of create_time is date not date_nanos, either we change type to date_nanos or add one more sort field.

2. Add refresh policy to update interaction api so that get conversation api will able to fetch latest data.

3. fix flaky test for conversation

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (fcbfc7e) 68.94% compared to head (82d62c4) 68.92%.

Files	Patch %	Lines
...g/opensearch/ml/engine/memory/MLMemoryManager.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                        Coverage Diff                        @@
##             feature/agent_framework_dev    #1733      +/-   ##
=================================================================
- Coverage                          68.94%   68.92%   -0.03%     
+ Complexity                          2612     2611       -1     
=================================================================
  Files                                241      241              
  Lines                              12875    12876       +1     
  Branches                            1291     1291              
=================================================================
- Hits                                8877     8875       -2     
- Misses                              3392     3394       +2     
- Partials                             606      607       +1     

Flag	Coverage Δ
ml-commons	68.92% <0.00%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Hailong-am]

@HenryL27 @Zhangxunmt please help to review

[Zhangxunmt]

lgtm. This sorting may need to apply to "get Interactions" as well, but let's not overkill for now.

[Hailong-am]

lgtm. This sorting may need to apply to "get Interactions" as well.

@Zhangxunmt there is no other sort key than create_time for interactions, do you mean we update the mappings of create_time filed to date_nanos?

[Zhangxunmt]

@Hailong-am , I mean the problem you described ideally applies to all docs that are sorted based on "date" field type, which includes conversation and interactions. However this race condition may be extremely rare for conversation/interaction because they are human-generated, so what you have changed in using the trace_number should be just enough.
Also we can always pick up another second key to sort, like the "updated_time" in conversation and "input" in interaction.