Determining distributive details

[o-smirnov]

Following on from #8, how do we determine where to run stuff in a distribution scenario?

One obvious idea is recipe-level variables (or inputs) to provide runner-specific options. For example:

my_recipe:
  for_loop:
    var: ms
    over: ms_list
    scatter: -1

  assign:
    opts:
      runner:
        slurm:  # slurm-specific options
          some_slurm_option: foo
        kube:   # k8s-specific options
          some_kube_option: bar

  assign_based_on:
    ms@index:   # when looping over 'ms', 'ms@index' is an integer loop enumerator that's maintained automatically
      0:
        opts.runner.slurm.target_node: node1
      1:
        opts.runner.slurm.target_node: node2

Also, remember _include and _use. The above YaML could live in e.g. custom-recipe-options.yml, which you can _include into your top-level recipe. That way your (custom, local) distribution config lives entirely separately from your (generic) recipe.

@landmanbester @SpheMakh thoughts?

[o-smirnov]

In terms of internal implementation -- stimela could just pass the contents of opts.runner.runner_type as **kwargs to the appropriate runner.