Backends and volumes

[SpheMakh]

Backends

I think how a Cab is executed should be specified by Step.backend which is inherited from the parent if not specified. When running a Cab natively, backend should be set to native

recipe:
  backend: native # Recipe steps use a native runner unless stated otherwise or overridden on command-line 
  steps:
    step1:
       ...
    step2:
      backend: docker # use docker for this step

For cabs that do not have an image set can check if the command is natively available, raise a warning (" Container image not set, will attempt to run foo-bar natively") and attempt execution.

Volumes

When running a container backend, we mount all recipe.dirs as well go through File, Directory and MS types in step.inputs and step.outputs and mount directories not in recipe.dirs. Inputs not designated as writable should be mounted as read-only unless a directory hosts multiple files that are both writable and read-only.

@o-smirnov thoughts?

[o-smirnov]

Sounds good. All agreed on volumes, this is exactly as I remember we discussed it on the farm.

For backends, let's recap then:

• there's a default backend setting specified via opts.backend or the --backend option. A button-pushing astronomer will just set it once and for all in ~/.config/stimela.conf and never touch it again

• recipe.backend is optional, and overrides the default backend for all its children

• step.backend is optional, and overrides the default backend for a step

• the latter two can be changed from the command-line using the x.y.z=value syntax

I'm a little unsettled about what the most intuitive fallback-to-native-if-image-is-not-set behaviour should be, but let's go with a warning for now and see how it feels.

Which reminds me: we could also contemplate a cab definition that consists of a Python file and a function name.

[SpheMakh]

I'm a little unsettled about what the most intuitive fallback-to-native-if-image-is-not-set behaviour should be, but let's go with a warning for now and see how it feels.

This could be a pain if you running a long reduction and it doesn't crash and inform you that you are missing an image. So maybe better to crash and provide option to ignore

[o-smirnov]

Hehe, wsclean-style: "if you wanted a source list, you should have enabled fit-spectral-pol. About 36 hours ago. Sucker!"

Note though that recipes and steps have a prevalidate() method now which gets called up front before anything is run. So if there's missing images and stuff, that's where it should be checked.

[o-smirnov]

I've had a happy thought. My current recipe is full of custom cab definitions as per the below. It occurs to me that, in container mode, all of these (and many more) could be served by a single Stimela image containing KERN-7 plus the standard software packages. Same image, just different command fields.

On top of that, we'll have the backend possibility at recipe or cab level to select native versus container mode. Straightforward and flexible as fsck!

cabs:
  stack_freq_cube:
    command: fitstool.py
    inputs:
      images:
        dtype: List[File]
        required: true
        policies:
          positional: true
          repeat: repeat
    outputs:
      cube:
        dtype: File
        policies:
          positional: true
          format: '--stack={0}:FREQ'

  pybdsm:
    command: ./run_pybdsm.py
    parameter_passing: yaml
    inputs:
      image:
        dtype: File
    outputs:
      gaul:
        dtype: File

  tigger_convert:
    command: tigger-convert
    inputs:
      src:
        dtype: File
        required: true
        policies:
          positional: true
      force:
        dtype: bool
        default: true
    outputs:
      dest:
        dtype: File
        policies:
          positional: true

  crystalball:
    command: crystalball
    policies:
      prefix: '--'
      replace: {'_': '-'}
    inputs:
      ms:
        dtype: MS
        writable: true
        required: true
        policies:
          positional: true
      output_column:
        dtype: str
        required: true
      sky_model:
        dtype: File
        required: true
      within:
        dtype: File
      num_sources:
        dtype: int

  breizorro:
    command: breizorro
    policies:
      replace: {'_': '-'}
    inputs:
      restored_image:
        dtype: File
        required: true
      threshold:
        dtype: float
        default: 6.5
    outputs:
      mask:
        dtype: File
        nom_de_guerre: outfile
        required: true