Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]Csv report generation had missing nested Fields #502

Merged
merged 11 commits into from
Jan 28, 2025
Merged

[BUG]Csv report generation had missing nested Fields #502

Changes from 2 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
47b6b0d
csv report generation added nested fields
sumukhswamy Jan 18, 2025
9880e7c
csv report generation added nested fields
sumukhswamy Jan 18, 2025
472e338
addressed comments for handling multi level nested fields
sumukhswamy Jan 21, 2025
380ea4d
linter changes
sumukhswamy Jan 21, 2025
d800b85
linter changes
sumukhswamy Jan 24, 2025
bf612d0
linter changes
sumukhswamy Jan 24, 2025
10f563b
linter changes
sumukhswamy Jan 24, 2025
92af2c8
flatten hits recursive function and modified nested test
sumukhswamy Jan 27, 2025
0b43116
removed unnecessary loops
sumukhswamy Jan 27, 2025
0c498c2
added test for deeply nested conditions
sumukhswamy Jan 28, 2025
b750dda
linter changes
sumukhswamy Jan 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions server/routes/utils/dataReportHelpers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,20 @@
} from '../../../../../src/plugins/data/common';
import { ExcelBuilder } from './excelBuilder';

export var metaData = {

Check failure on line 18 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected var, use let or const instead
saved_search_id: <string>null,

Check failure on line 19 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Use 'as string' instead of '<string>'
report_format: <string>null,

Check failure on line 20 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Use 'as string' instead of '<string>'
start: <string>null,

Check failure on line 21 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Use 'as string' instead of '<string>'
end: <string>null,

Check failure on line 22 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Use 'as string' instead of '<string>'
fields: <string>null,

Check failure on line 23 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Use 'as string' instead of '<string>'
type: <string>null,

Check failure on line 24 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Use 'as string' instead of '<string>'
timeFieldName: <string>null,

Check failure on line 25 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Use 'as string' instead of '<string>'
sorting: <string>null,

Check failure on line 26 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Use 'as string' instead of '<string>'
fields_exist: <boolean>false,

Check failure on line 27 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Use 'as boolean' instead of '<boolean>'
selectedFields: <any>[],

Check warning on line 28 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
paternName: <string>null,
searchSourceJSON: <any>[],

Check warning on line 30 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
dateFields: <any>[],

Check warning on line 31 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
};

// Get the selected columns by the user.
Expand All @@ -51,7 +51,7 @@
// Build the OpenSearch query from the meta data
// is_count is set to 1 if we building the count query but 0 if we building the fetch data query
export const buildRequestBody = (
report: any,

Check warning on line 54 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
allowLeadingWildcards: boolean,
is_count: number
) => {
Expand Down Expand Up @@ -128,13 +128,13 @@

// Fetch the data from OpenSearch
export const getOpenSearchData = (
arrayHits: any,

Check warning on line 131 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
report: { _source: any },

Check warning on line 132 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
params: { excel: any; limit: number },

Check warning on line 133 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
dateFormat: string,
timezone: string
) => {
let hits: any = [];

Check warning on line 137 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
for (let valueRes of arrayHits) {
for (let data of valueRes.hits) {
const fields = data.fields;
Expand Down Expand Up @@ -224,7 +224,7 @@

// Convert the data to Csv format
export const convertToCSV = async (dataset, csvSeparator) => {
let convertedData: any = [];

Check warning on line 227 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
const options = {
delimiter: { field: csvSeparator, eol: '\n' },
emptyFieldValue: ' ',
Expand Down Expand Up @@ -252,7 +252,7 @@
return result;
}

function flattenObject(obj = {}, parentKey = '', result: any = {}) {

Check warning on line 255 in server/routes/utils/dataReportHelpers.ts

View workflow job for this annotation

GitHub Actions / Lint 

Unexpected any. Specify a different type
for (const [key, value] of Object.entries(obj)) {
const newKey = parentKey ? `${parentKey}.${key}` : key;

Expand Down Expand Up @@ -307,6 +307,30 @@
Object.keys(data)
.filter((sourceKey) => sourceKey.startsWith(key + '.'))
.forEach((sourceKey) => (result[sourceKey] = data[sourceKey]));
Object.keys(data).forEach((key) => {
const value = data[key];

if (Array.isArray(value)) {
const flattenedValues = {};

value.forEach((item) => {
Object.keys(item).forEach((subKey) => {
const newKey = `${key}.${subKey}`;
if (!flattenedValues[newKey]) {
flattenedValues[newKey] = [];
}
flattenedValues[newKey].push(item[subKey]);
});
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems this only does it one level, does it work if the data is more nested like

{
  "a": {
    "b": {
      "c": [
        {
          "d": [{ "e": 1 }, { "e": 2 }]
        },
        {
          "d": [{ "e": 3 }, { "e": 4 }]
        }
      ]
    }
  }
}

and could you add some unit tests?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have updated it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't really see any logic change in the new commits?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding more comments here:

  1. This may be still only working with one level of array nesting
  2. Best way to check is by adding multi-level nested array of objects
  3. Nit - Object.keys(data) is looped twice can just keep it one.


// Add flattened values to the result object, ensuring no duplicates
Object.keys(flattenedValues).forEach((newKey) => {
result[newKey] = flattenedValues[newKey];
});
} else {
result[key] = value;
}
});
}
});
return result;
Expand Down
Loading