Very slow deployments to Azure from hosted agent

[gabbsmo]

Hi! I have been trying to get assistance with performance degradation in deployments from Azure Pipelines to Azure services, e.g. App Service. DevOps support forum is unable to assist due to the larger scope, Azure QA automatically deletes my post. Then I was referred here by AzureSupport on Twitter. Are you able to assist me in some way?

Issue description:
We have been using Azure DevOps pipelines since before it was called Azure DevOps. We have been running pretty much the same deployment process since then. Deploy to App Service staging slot, run Entity Framework database migrations and finally swap slots.

Before, the whole flow was completed in about 6 minutes and was very reliable. We even deployed mid day in good confidence.

Today it takes 30 minutes or more. Running the EF migrations (ef6.exe) will take 1 minute, even if there are no changes to apply! Occasionally App Service deployment grinds to a halt, transferring 1 file every 5 minutes or so. Because of this, we are very careful with deployments these days, despite having a high availability App Service configuration.

We are using the hosted Windows Latest agents and have verified that older Windows images have the same issue.

For comparison, deployments from my own local machine takes around 5 minutes. It seems that the network I/O between the hosted pipelines and Azure is very slow.

I do not think there is any issue with the agent itself since CPU bound tasks, like builds, runs as expected.

I am sorry for not reporting this earlier. There seem to have been a major degradation in performance and reliability over the years.