Input/output folders

[o-smirnov]

There's a few wrinkles that could use straightening out for R2:

• msdir is something of an atavism, since you're introducing I+O parameters (mixed or both? I prefer both as it seems more precise use of English) which is more general.

• no possibility of multiple input/output dirs (think the Caracal use case, when we split an ms from raw data dir to msdir -- one day someone is going to ask for something silly like applying caltables OTF. Seems restrictive that we can't read stuff from multiple places at once.)

• the :msfile syntax to change a file's destination feels kludgy

• for "casual" use, as e.g. when running a single cab with stimela exec, input=output=. ought to be the default?

Proposal:

• Multiple directories can be defined and mounted inside the container. The default is to just have the two standard ones, dirs.input and dirs.output.

• File/directory parameters without a leading slash are implicitly taken relative to dirs.input for input parameters, and dirs.output for I/O and output ones (which is more or less what happens now).

• For file/directory parameters with a leading slash, add their resident directory to the list of mounts (r/o or r/w as appropriate)

• Support standard substitutions e.g. "{dirs.output}/input-file.txt" for twisted cases where you want an input to come from the output directory (kind of what :msfile does right now...)

• Should input=output=. be the default for "casual" mode?

[o-smirnov]

P.S. This is motivated by trying to simplify the user experience. If, say, you're running a wsclean cab, it would be nice if the behaviour was similar to the one outside Stimela. That is, you give it a path to the MS and to a mask -- and you shouldn't care if they live in different directories -- and you expect the output images under . unless you've explicitly specified otherwise, right?

[SpheMakh]

I agree, except for setting input=output by default. The input folder(s) should be read-only, we can give the user rope to hang themselves by allowing it to be writable but that should not be the default.

[o-smirnov]

How about a default of separate dirs for recipes, and "." for executing a single cab? If you're using it to run one single standalone program, my expectation as a user would be that it would work just like the non-containerized version, i.e. with the CWD.

[SpheMakh]

sounds good

[o-smirnov]

To put a bit more meat on the proposal:

1. Let us have a recipe.dir section, with the default being

recipe:
  dir:
    input: input    # or perhaps '.'?
    output: output  # or perhaps '.'? 

2. The author of the recipe is free to declare additional dirs inside this section, if they're making a recipe that works with multiple dirs (think Caracal's msdir, raw_msdir, input and output dirs for example)

3. Any recipe step can declare its own dir section to override things within that step (if not specified, it is inherited from the parent recipe)

4. Parameter settings specified in the recipe can use constructs such as {dir.input}/filename.txt, {dir.ms}/foo.ms.

5. Paths that do not start with / or ./ are implicitly taken to be relative to dir.input or dir.output, according to whether the parameter is of an input kind, or of an output/both kind.

6. Paths that do start with / or ./ are taken to be absolute, or relative to the cwd, respectively. The backend shall mount all required directories inside the container (which means that additional dirs, over and above the content of dir, may need to be mounted -- but this is straightforward to work out from the set of given cab parameters.)

Actually, if we implement (6), then the whole dir thing becomes just a convenience variable i.e. a way to enforce a naming convention.

[SpheMakh]

Do you mind switching dir to dirs since dir is already taken by python. dirs is also more grammatically correct

[o-smirnov]

True, and more consistent with "inputs" and "outputs" and "vars".

(As for "already taken", is that why you renamed type to dtype? Shouldn't matter: x.type does not clash with the built-in symbol.)

[SpheMakh]

(As for "already taken", is that why you renamed type to dtype? Shouldn't matter: x.type does not clash with the built-in symbol.)

Yes, x.type does not clash, I just try to stay away from using these variable names. Also why I was switching from input to indir