[BUG] Extractor replacing encoded characters in policies

[ochadwick-westminster]

Release version

v4.10.2

Describe the bug

Whilst extracing another, unrelated API, we have encountered an issue where 2 pre-existing API's that are included in the extractor config. These APIs both contain inbound policies in the policy.xml that were, for example:

When the extract now runs, the encoded & characters are getting set back to ampersands. Nothing has been changed on these APIs and the version of APIOps we are using, 4.10.2, has not been changed.

The API in APIM contains the encoded characters, and APIOps has successfully extracted these APIs mulitple times previously without incident. I have tired running APIOps locally on my machine and the problem seems to be in the extract process, as the files extracted contain the problem before any further pipeline steps are applied.

I have also tried running locally with the latest release but encounter the same problem.

The policy when viewed in APIM looks like:

However when viewed in the repo after extraction, it shows:

Expected behavior

The policy content is preserved exactly as defined in APIM.

Actual behavior

The encoded characters are being decoded when extracted.

Reproduction Steps

I'm unclear what steps to add as this seems to have started happening with no changes on either the APIs in question or the APIOps version being used.

[github-actions]

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

[RaguSoftEng]

We also facing the same issue.

[itssKrish]

Hi, I'm also facing the same. I tried switching between versions v4.10.2 and v5.1.2 but no luck. This creates a major issue when we re-deploy the extracted file back to APIM portal (as a backup). I hope this will get resolved as soon as possible!

[ochadwick-westminster]

We were able to work around my original example by re-writing the apim policy to add query parameter policies to include them rather than adding them onto the end of the url rewrite (which is likely the better way to do it in general anyway) but it did mean updating pre-existing APIs that have worked on the extract with older versions and could mainfest elsewhere so be good to have it resolved.

[vandriot]

Hi,
I'm also facing the issue with version v5.0.1.

[Buffelstoot]

Hi,

We're having the same issue with version v5.1.4.
Please advise!

[waelkdouh]

Did you try the latest v6? Also did you try the suggestion by @ochadwick-westminster ?

[vandriot]

Hi,

I no longer have the issue with v5.0.1 or latest v6.
I saw that there was an APIM release 2 weeks ago (https://github.com/Azure/API-Management/releases/tag/release-service-2024-06) and there is a fix which might have resolved our issue:

• We fixed a bug causing XML elements within Liquid templates in policy documents to be needlessly decoded

All my APIM instances are updated with this release so I can't be 100% sure but I no longer have the issue when running the extractor using Azure DevOps pipeline. I also tried to retrieve the policy using the REST API endpoint and I don't have the issue either.

If someone else who didn't get the APIM update can try to call this REST API endpoint and check if you have the encoding issue?

https://management.azure.com/subscriptions/xxxxx/resourceGroups/xxxxx/providers/Microsoft.ApiManagement/service/xxxxx/apis/xxxxx/operations/xxxxx/policies/policy?api-version=2022-08-01&format=rawxml

When retrieving the policy using the REST API, if you don't have the & in the response body, it means that the issue will be fixed when the update will be applied to your APIM instance.

You can check the Activity logs of your APIM to check if you got the update or not yet.

[itssKrish]

Hi all, I checked the same with version "v6.0.1-rc1" and I can confirm this issue still persists.
Thanks!

[waelkdouh]

Hi all, I checked the same with version "v6.0.1-rc1" and I can confirm this issue still persists.

Thanks!

Please take a look at @vandriot response.

[waelkdouh]

Please remember to close the issue if you error has been resolved

[itssKrish]

@waelkdouh - I checked this again as mentioned by @vandriot and I don't have this issue anymore. Thanks for the help!!!

@ochadwick-westminster - maybe you can check this at your end and close this one. Thanks.