SiteExtension: Update site extension upgrade to perform incremental deployment

[ashelley]

Hello,

The process for updating a node.js based site extension is extremely slow.

I have hypothesized that this is because the whole site extension directory is deleted before a new version is installed.

The actual reproduction steps for this problem are hard to explain/setup and hard to measure so I have created a simulation to show the effects of this problem.

Here is a typical package.json with some amount of dependencies

package.json

{
  "name": "test",
  "version": "0.0.1",
  "description": "example package.json",
  "main": "index.js",
  "author": "",
  "dependencies": {
    "body-parser": "^1.12.2",
    "express": "^4.12.3",
    "express-session": "^1.11.1",
    "fs-extra": "^0.18.0",
    "lodash": "^3.6.0",
    "moment": "^2.10.2",
    "morgan": "^1.5.2",
    "multer": "^0.1.8",
    "passport": "^0.2.1",
    "request": "^2.55.0"
  },
  "engines": {
    "node": "0.10.x"
  }
}

Here is a power shell command to simulate what would happen during a site extension upgrade
install-perf-test.ps1

$sw = [Diagnostics.Stopwatch]::StartNew()
$perf = Measure-Command {rm -r node_modules 2>&1 > $null}
Write-Output "delete took $perf"
$perf = Measure-Command {npm install --production 2>&1 > $null}
Write-Output "npm took $perf"
$sw.Stop()
$total = $sw.Elapsed.TotalSeconds
Write-Output "total time $total"

• first this script deletes the node_modules folder (to simulate d:\home\siteextensions\testextension being deleted before being upgraded)
• then this script runs npm install

1. You will notice that the first time the site extension is installed performance is slow but okay
2. Second time you run it (simulating an upgrade) you will notice that the delete operation takes a long time causing the upgrade process to be very slow

Here is a screen shot of the simulation:

Expected results:

Ideally the deployment/upgrade of a node base site extension would as fast as deploying a website. For example using the above package.json as the basis for an app this is how long it takes:

$ time git push azure master
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 312 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Updating branch 'master'.
remote: Updating submodules.
remote: Preparing deployment for commit id '33666ae906'.
remote: Generating deployment script.
remote: Running deployment command...
remote: Handling node.js deployment.
remote: KuduSync.NET from: 'D:\home\site\repository' to: 'D:\home\site\wwwroot'
remote: Copying file: 'app.js'
remote: Looking for app.js/server.js under site root.
remote: Using start-up script app.js
remote: Generated web.config.
remote: Node.js versions available on the platform are: 0.6.17, 0.6.20, 0.8.2, 0.8.19, 0.8.26, 0.8.27, 0.8.28, 0.10.5, 0.10.18, 0.10.21, 0.10.24, 0.10.26, 0.10.
28, 0.10.29, 0.10.31, 0.10.32, 0.12.0.
remote: Selected node.js version 0.10.32. Use package.json file to choose a different version.
remote: Updating iisnode.yml at D:\home\site\wwwroot\iisnode.yml
remote: npm WARN package.json [email protected] No repository field.
remote: npm WARN package.json [email protected] No README data
remote: npm WARN package.json [email protected] No repository field.
remote: Finished successfully.
remote: Deployment successful.

real    0m23.863s
user    0m0.000s
sys     0m0.000s

In this simulated scenario it was 23 seconds vs 73 seconds to upgrade a website vs a site extension. In reality to upgrade an actual node based site extension it is taking 2-4 minutes (with no binary dependencies).

Maybe in the future there is some how a better way to deploy a site extension than by deleting the whole directory and installing from scratch or as an alternative use a faster delete method internally to mask the problem temporarily.

[ashelley]

Please also note that this may be solved in the future via compressed packages (hopefully)

nodejs/node#1278
nodejs/NG#5

[shrimpy]

@ashelley

RE:
I have hypothesized that this is because the whole site extension directory is deleted before a new version is installed.

[shrimpy] yes, i can confirm you hypothesizes. we do delete the folder before new version is installed

The idea behind site extension installation is that, we expect content can be run right after we copy them to site extension folder.

in your case, your app perform "node install" before running the site extension, that is why it is slow for you.

You should pack all the dependencies together into your package.

[ashelley]

@shrimpy thank you for the confirmation.

RE: You should pack all the dependencies together into your package.

I don't believe this will solve the problem. The very slow delete operation (lots of individual files) will have to happen even if I pack all my dependencies into the package.

This root of the problem is with how node packages work in combination with the networked filesystem that azure websites use. I just wanted to report that because the upgrade process is destructive that it takes a long time to complete.

Unless azure websites internally have a faster way to bulk delete a directory I don't think this problem is solvable but I just wanted to make note of it anyway just in case someone might have a good way to solve this.

I have contemplated making my site extension install itself outside of the site extension folder ie d:\home\myinstalledextension so that I can write my own upgrade routine to avoid the bulk delete however I'm not sure whether this is a good idea or not since there will be no way from the kudu UI to delete the extension once it's installed other than manually deleting the folder. However I don't feel as this would be friendly to people's website instance.

[shrimpy]

@ashelley
RE:
The very slow delete operation (lots of individual files) will have to happen even if I pack all my dependencies into the package.

[shrimpy] this is true. deletion would also consume large amount of time if there are lots of files.
what you contemplate to do seems like a good idea. but instead of you doing it, maybe kudu should implement that logic. when upgrade to a new version, instead of clear the folder, we install new package into a sibling folder e.g {extension_name}-new. once installation finished, swap the two folder and remove the one that contains old bits

@davidebbo what do you think?

[davidebbo]

Swapping between two folders could add a fair bit of complexity in the system. And it also wouldn't save us from having to recreate all the files even for a minor upgrade. Maybe a better approach is to support incremental deployment, as we do for the site itself. We could even entertain the idea of git based site extensions, where upgrading just means pulling and checking out a different commit.

[ashelley]

originally (before I realized the whole folder was getting deleted on upgrade) I had the logic in my install.cmd to do this:

1. copy package.json from site extenstion to d:\local\temp\hellotml
2. npm install --prefix d:\local\temp\hellohtml
3. kudusync -f d:\local\temp\hellohtml\node_modules -t d:\home\SiteExtensions\HelloHtml\node_modules

However, obviously because of the previous delete this did not help. But i tried :(

[shrimpy]

@davidebbo
yes, "incremental deployment" is better. And won`t be hard to implement.
When perform upgrade, compare the content from new package with old package, and only apply the "differences".

@ashelley
if we have "incremental deployment", will solve the slowness of deletion issue. And i would still suggest you to pack all the dependencies into the package. Then things should work much more faster.

[ashelley]

@shrimpy yes an incremental deployment would be great and would solve the issue.