Swagger Aggregator crashes after getting the documentation from the crud service

[flower-of-the-bridges]

Hi

Currently we are using crud-service:6.0.1 and swagger-aggregator:3.4.8 on a project.

When the swagger aggregator gets the documentation from the path /documentation/json exposed by the crud-service, the swagger-aggregator microservice crashes with the following message:

{"level":30,"time":1665570530631,"pid":7,"hostname":"swagger-aggregator-669b9cd676-2kpdq","msg":"Server listening at http://0.0.0.0:3000"}
{"level":30,"time":1665570544506,"pid":7,"hostname":"swagger-aggregator-669b9cd676-2kpdq","reqId":"e16225a56927eb1ae44d5f7353ab07b3","msg":"requested openapi3 documentation"}
{"level":20,"time":1665570545229,"pid":7,"hostname":"swagger-aggregator-669b9cd676-2kpdq","reqId":"e16225a56927eb1ae44d5f7353ab07b3","msg":"Applying includePath filter on service swagger paths"}
{"level":20,"time":1665570545229,"pid":7,"hostname":"swagger-aggregator-669b9cd676-2kpdq","reqId":"e16225a56927eb1ae44d5f7353ab07b3","msg":"Applying excludePath filter on service swagger paths"}
{"level":20,"time":1665570545230,"pid":7,"hostname":"swagger-aggregator-669b9cd676-2kpdq","reqId":"e16225a56927eb1ae44d5f7353ab07b3","msg":"Transforming service swagger paths"}
{"level":30,"time":1665570545232,"pid":7,"hostname":"swagger-aggregator-669b9cd676-2kpdq","reqId":"fb2b1eb6ca6da635193b13b179289961","msg":"requested subswagger list"}
{"level":30,"time":1665570545320,"pid":7,"hostname":"swagger-aggregator-669b9cd676-2kpdq","reqId":"fb2b1eb6ca6da635193b13b179289961","http":{"request":{"method":"GET","userAgent":{"original":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"}},"response":{"statusCode":200,"body":{"bytes":2}}},"url":{"path":"/openapi/v3/subswaggers/?includeAll=false","params":{}},"host":{"hostname":"example.com","forwardedHost":"example.com","ip":"0.0.0.0"},"responseTime":88.32592511177063,"msg":"request completed"}

<--- Last few GCs --->

[7:0x7f72852433a0]    77395 ms: Mark-sweep 251.1 (258.1) -> 250.6 (258.9) MB, 4600.4 / 0.0 ms  (average mu = 0.239, current mu = 0.061) allocation failure scavenge might not succeed
[7:0x7f72852433a0]    82606 ms: Mark-sweep 251.7 (258.9) -> 251.0 (259.1) MB, 5013.5 / 0.0 ms  (average mu = 0.137, current mu = 0.038) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

The crud-service exposes the documentation for 15 single views, more or less.

Even if I increase the limits of the microservice, the API Portal gets stuck in the loading page and the swagger-aggregator crashes anyway.

Any ideas on how to solve the issue?
Thanks

[fredmaggiowski]

Hey, I think the openapi from the service is a bit too big for the memory limit you have set, try adding more memory and maybe some cpu too to speed up the processing

[flower-of-the-bridges]

Hi

Thanks for replying! Currently at the moment I have resolved the issue by assigning the following resources to the microservice:

resources:
  limits:
    cpu: 150m
    memory: 650Mi
  requests:
   cpu: 50m
   memory: 150Mi

Is this a normal assignment of resources for the swagger aggregator? For my experience based on other projects where the API Portal is deployed, It's too much.

Also, I have to retrieve documentations only from one source(the crud-service), so I don't think the problem is relied on the project resources but on how the swagger aggregator computes the Swagger doc based on the openapi3 standard.

[fredmaggiowski]

Since the service aggregates all the APIs from your project the resource consumption of the services depends on the size of the oas spec size of each of your services; so big configurations require more ram and cpu for processing

[flower-of-the-bridges]

I understand that, but in this case I'm only aggregating APIs from the CRUD Service.

In one project, I have 12 collections with few fields and the swagger-aggregator works just fine with 30Mi.
However, in another one, with only 7 collections but with a more complex data model, the swagger-aggregator returns the APIs only with 650Mi, otherwise the application goes on Out Of Memory.

Can't the swagger-aggregator return the documentation in chunks, without keeping all the aggregated APIs in memory? I think this could be also a good workaround to make the API Portal more "user friendly" against this kind of issues: at the moment the API Portal shows only the information Something went wrong to the user, without letting him know what exactly happened and how to solve the issue.

[fredmaggiowski]

I think the whole concept of the swagger aggregator should be revised; this could be a good proposal. I'd think about a feature request / improvement to be moved forward on the service itself

[davidebianchi]

Yes, the swagger aggregator workflow should be improved.
An interesting possibility to improve the workflow for services running in kubernetes could be something like watching the deployment resources of namespaces and cache the oas returned by the services and aggregate it. When deployment is updated, invalid the cache and recreate the final oas.