loading and compressing files

[dazinator]

I use bicep to create some vm's and then I also use it to run RunCommand's on those VM's.

Run commands seem fairly limited in that you can only send a single script file to execute or a single command.

In my case I am using a run command that executes a script. The script accepts a single file as an argument.

I am able to send the file as an argument to the run command by using the loadFileAsBase64() command in Bicep in order to load the file data and construct the argument as a base64 string. The run command script accepts the arguement, base 64 decodes it to get the original file data. This works well.

But now I need the ability to send multiple files to this script. The script should just take the file/files it's given and loop through them and do stuff with them. So how to transmit multiple files to the run command efficiently?

I considered mediating these files via a third party - i.e like a shared disk, or azure blob storage. But this comes at additional complexity and resource cost. I prefer to keep this simple.

So my thought is to define a custom json object like a dictionary object with the keys being the file paths, and the values being the base64 encoded file data

{
"foo.config": loadFileAsBase64('/foo.config'),
"bar.config": loadFileAsBase64('/bar.config')
}

I haven't tried this yet but I suspect sending a json string as an argument to a run command may casue issues the other side where the argument is not passed correctly to the bash script. In this case I may have to take an approach where I load each file in plain text into the dictionary and then base64 encode the entire json dictionary object and send that as the parameter. I want to avoid base64 encoding each file, and then base64 encoding the json envelope because that's double base64 encoding on the file data! This can remain a side issue I am weary about for now.

So this "envelope" would be sent as an argument to the RunCommand and received by my bash script that is run on the server. In that bash script I can use jq and decode or whatever to restore the original files and do stuff with them.

This approach works, but it has me questioning if there is room for improvement especially when it comes to efficient use of bandwidth.

So I am asking whether the following may be worth considering.. Add a new function to bicep that is able to load (multiple?) files in some sort of compressed wrapper / envelope.

For example in my case, I am converting each file (or the entire json envelope containing each file in plain text) to base 64 for transmission because in bicep there is no way to load and compress the file data first, prior to the base64 encoding operation. Likewise I don't want to have to "pre-compress" these files prior to bicep deployment each time. The files themselves are things like config files in yml or json, and are maintained within the project, so having to compress them as a build step each time a change is made is not great.

Ideally, there would be some sort of bicep function to load file data as base 64, but also the with the option or compressing the file data before the base 64 encoding operation.

Not sure if it makes sense to have the ability to load multiple files at once with compression and get back some sort of envelope object?

var filesData = loadFilesAsBase64([ './file1.config', 'file2.config' ], Compression.Gzip)

Or just one file at a time?

var compressedFile = loadFileAsBase64( './file1.config', Compression.Gzip)

However it should be possible for the calle to determine what the original file size wasand perhaps other metadata abotu the file, so returning just the raw base64 data does not seem sufficient - ideally it would be an object with the data but also additional metadata.

This proposal is premised on the assumption that there is a worthwhile bandwidth saving to be had in some situations - enough to make it useful / warranted.

[brwilkinson]

Can I ask what you are trying to configure ?

• You may want to consider the custom script extension as an alternative.

We just had a similar question on how to do similar

• Create a custom VM image then install software #10264

There are multiple way to achieve this:

1. Using Azure Automation DSC Extension (which is still fully supported, however also considered legacy i.e. not good linux support)

   • https://learn.microsoft.com/en-us/azure/automation/automation-dsc-onboarding

2. Using Azure Automanage Machine Configuration (which is new and was previously called Guest Configuration).

   • https://learn.microsoft.com/en-us/azure/governance/machine-configuration/overview

3. Using a Custom Script extension (I wouldn't recommend this it doesn't scale very well).

   • https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-windows

4. Using a template approach with your imaging, called: Azure VM Image Builder

   • https://learn.microsoft.com/en-us/azure/virtual-machines/image-builder-overview?tabs=azure-powershell

5. If you just need applications installed, consider VM application feature

   • https://learn.microsoft.com/en-us/azure/virtual-machines/vm-applications-how-to?tabs=powershell

They all have Bicep support, once you decide on which suits, then you can dig into some templates on that technique.

[dazinator]

Can I ask what you are trying to configure ?

'The VM is a manager node of a docker swarm cluster. I am doing a docker stack deploy to deploy a stack. This command accepts a docker-compose.yml file which defines the containers / services to deploy, but also if that compose file references docker config files,; those files should also be present.

I am using this run command to deploy a stack that contains azure devops agent right now which is pretty simple - just has a compose.yml file and no additional config files required
However I'd like to generalise this run command so it is capable of deploying any swarm stack which means allowing not just to transmit the compose file but also any config files.

• You may want to consider the custom script extension as an alternative.

I use that already when initialising the vm's to install docker and other things. However as its not very extensible or modular - I decided against using it in certain cases like this one.

• Create a custom VM image then install software10264

I've seen this but image building is a brand new skill for me that would require significant investment. I am also not sure this is a very flexible process where as using a reusable run command to deploy a workload to any target ubuntu VM is very composable.

P

There are multiple way to achieve this:

1. Using Azure Automation DSC Extension (which is still fully supported, however also considered legacy i.e. not good linux support)

I need robustness on Linux so not good linux support is not good for me.

2. Using Azure Automanage Machine Configuration (which is new and was previously called Guest Configuration).

Thanks will take a look to see what this is.

• https://learn.microsoft.com/en-us/azure/governance/machine-configuration/overview

3. Using a Custom Script extension (I wouldn't recommend this it doesn't scale very well).

Already dismissed, single entry point not very composable.

4. Using a template approach with your imaging, called: Azure VM Image Builder

As above, not appealing.

5. If you just need applications installed, consider VM application feature

This doesn't look like it's suitable for deploying docker workloads?

[brwilkinson]

The latest way to configure machines in Azure is "Machine Config"

Check out this Module for commands and specifically the DSC modules.

• https://techcommunity.microsoft.com/t5/azure-governance-and-management/now-open-source-nxtools-managing-linux-iac-just-got-simpler/ba-p/3781203

This way you build a set of standard config, then apply the one you want. This can replace the custom script extension plus do all config that you need.

[dazinator]

Thanks that looks promising!

[brwilkinson]

Here are the docs

• https://learn.microsoft.com/en-us/azure/governance/machine-configuration/overview