Excessively large generated report files

[Maarc]

Describe the bug
Since you updated to .NET 5, the generated report files (JSON & HTML) became very very large.

To Reproduce
Steps to reproduce the behavior:

1. Retrieve https://github.com/OrchardCMS/OrchardCore as a test application (git clone)
2. Analyze the repo dotnet ApplicationInspector.CLI.dll analyze -s "/folder/to/OrchardCore" -v Info -x high -c high,medium -k lib,.vs,.git,.idea
3. You will obtain one 611MB-large HTML file and one 564MB large JSON file

Expected behavior
With AppInspector 1.2.89 (version I keep using) the same files had a size of around 3 MB.

Desktop

• OS: macOS BigSur 11.3.1 (20E241)
• Version: App Inspector 1.3.17 dotnet_core distribution (leveraging dotnet 5.0.203)

[gfs]

Thanks for your report.

One note: Starting with 1.3 the -k option uses glob patterns so it's likely not excluding what you expect. For a folder use

**/foldername/** 

as we only enumerate files not folders. See https://en.m.wikipedia.org/wiki/Glob_(programming) for info on Glob patterns.

It shouldn't be outputting both an html and a JSON from a single run, which I will investigate.

Thank you for providing the source your using. I will try a run against the sample project you provided as well.

[gfs]

I ran ApplicationInspector 1.3.3 (latest, 17 is a beta sorry for the confusion. I've taken that version down.) against your project and I don't see anything in the output that appears to have incorrectly matched.

I also checked the code and it was intentional that the .json is produced alongside the .html, for the default run arguments so that you could "link" to it from the HTML. I'm opting to remove that so it will only produce the HTML report by default, as each may be large.

I think the difference in behavior you are seeing is that by default 1.2.89 would only return 1 (ish) matches for a single tag. Later versions include all matches which is why the report is larger.

One way to get a smaller report is - if you don't need the exact locations that the matches occurred - you can now use the -t option to only receive the tags that matched (this is also much faster to run). This is similar to the old behavior but significantly faster.

[gfs]

#379 will close this to cover the JSON and HTML dual report issue.

The rest of the things you reported appear to be working as expected.

[gfs]

I've also added an argument that works will with -t to not gather extra metadata about individual files with -n.

Running dotnet .\bin\Debug\net5.0\ApplicationInspector.CLI.dll analyze -s C:\Users\gstocco\Documents\GitHub\OrchardCore -k **/lib/**,**/.vs/**,**/.git/**,**/.idea/** -f json -o out.json -t the output is less than 3mb.

Adding -n the size drops to 8kb.

If you do want the individual matches. you can also reduce the size of the output file by reducing the amount of context lines gathered for each match. i.e. set --num-lines-context 0

[gfs]

1.3.4 was just released and fixes a bug with the -t option.

[Maarc]

Thank you for your quick answer and fix. I will check out the latest version soon.

[Maarc]

Hi @gfs !

Thanks again for your support.

I just conducted a few tests with the latest 1.4.2 release.

(A) Using -t

Indeed, using the -t option tremendously shrunk the file, back to less than 3 MBs. However, the leveraged features are no more highlighted on the the "Application features" page:

and no tag is listed into the "Report Summary -> Tags List View" page:

To see the features used, I would need to open every feature group and look manually:

Thus, the -t option is not really usable for me yet.

(B) Using -t and --context-lines

When I add -n, the generated file is 3MB smaller, but the "Report Summary -> Project info -> Analyzed files" diagram is broken:

The file's size is around 15MB, which is great.

(C) Using --context-lines

I ended upgrading to the version 1.4.2 and using the --context-lines 0 option. The generated file is around 17MB, which is still a lot, but better than the >600MB I had before.

---

Should I file issues for (A) and (C)?

[gfs]

(A) Using -t

Indeed, using the -t option tremendously shrunk the file, back to less than 3 MBs. However, the leveraged features are no more highlighted on the the "Application features" page:

This is a bug.

and no tag is listed into the "Report Summary -> Tags List View" page:

This is also a bug. I have a fix for this. #383.

(B) Using -t and --context-lines

When I add -n, the generated file is 3MB smaller, but the "Report Summary -> Project info -> Analyzed files" diagram is broken:

That's expected. Its no longer gathering that data so there's nothing to show.

(C) Using --context-lines

I ended upgrading to the version 1.4.2 and using the --context-lines 0 option. The generated file is around 17MB, which is still a lot, but better than the >600MB I had before.

Should I file issues for (A) and (C)?

C doesn't sound like a bug.

[Maarc]

Thank you for your answer!

Sorry, I meant (A) and (B), not (A) and (C).

[gfs]

A will be fixed when I can address #384.

B will probably not be addressed until the redo of the interface in #2.