Add support for MySQL Date/Time/TimeStamp/DateTime date type transformation #5190
Conversation
…mation to OpenSearch Signed-off-by: Dinu John <[email protected]>
Signed-off-by: Dinu John <[email protected]>
Signed-off-by: Dinu John <[email protected]>
dinujoh
requested review from
sb2k16,
chenqi0805,
engechas,
graytaylor0,
kkondaka,
KarstenSchnitter,
dlvenable and
oeyh
as code owners
Signed-off-by: Dinu John <[email protected]>
Signed-off-by: Dinu John <[email protected]>
| final Long dateEpoch = parseDateTimeStrAsEpoch(dateStr); | ||
| if (dateEpoch != null) return dateEpoch; |
There was a problem hiding this comment.
Is the return value here epoch in millis or in days, since "Date from binlog is long value representing days since epoch"?
There was a problem hiding this comment.
It's Unix time (number of seconds that have elapsed since 00:00:00 (UTC), Thursday, 1 January 1970, not counting leap seconds)
There was a problem hiding this comment.
In that case, we need to convert it to epoch millis to be consistent with the conversion below for export data?
There was a problem hiding this comment.
There was a problem hiding this comment.
Makes sense. Thanks for clarifying!
| final Long dateTimeEpoch = parseDateTimeStrAsEpoch(dateTimeStr); | ||
| if (dateTimeEpoch != null) return dateTimeEpoch; |
There was a problem hiding this comment.
Similarly here, is the return value epoch in millis or epoch in micros?
|
|
||
| private static Stream<Arguments> provideDateTestCases() throws ParseException { | ||
| return Stream.of( | ||
| Arguments.of("2023-12-25", getEpochMillisFromDateStr("2023-12-25")), |
There was a problem hiding this comment.
We may want to add a test case with a string of long value as input.
| } | ||
|
|
||
| private static Stream<Arguments> provideTimeTestCases() { | ||
| return Stream.of( |
There was a problem hiding this comment.
Similarly, we may want to add a test case with a string of long value as input.
Signed-off-by: Dinu John <[email protected]>
Signed-off-by: Dinu John <[email protected]>
…mation (opensearch-project#5190) * Add support for MySQL Date/Time/TimeStamp/DateTime date type transformation to OpenSearch Signed-off-by: Dinu John <[email protected]> * Configure BinlogClient to deserialize data and time as long Signed-off-by: Dinu John <[email protected]> * Add documentation Signed-off-by: Dinu John <[email protected]> * Update documentation Signed-off-by: Dinu John <[email protected]> * Update documentation Signed-off-by: Dinu John <[email protected]> * Update unit test and documentation Signed-off-by: Dinu John <[email protected]> * Update documentation Signed-off-by: Dinu John <[email protected]> --------- Signed-off-by: Dinu John <[email protected]> Signed-off-by: Kondaka <[email protected]>
Description
Add support for MySQL Date/Time/TimeStamp/DateTime date type transformation between binlog and S3 export formats to OpenSearch.
MySQL binlog represents temporal types as follows:
S3 export formats:
Note: Fractional seconds are optional for DATETIME
Issues Resolved
Contributes to #4561
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.