Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: shell: refactor output plugin and enable per-node/task output files #6539

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

WIP: shell: refactor output plugin and enable per-node/task output files #6539

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
7d5e88a
libsubprocess: workaround potential deadlock in pre-exec hook
grondo Dec 29, 2024
468b22d
shell: allow custom task, shell_rank passed to mustache_render()
grondo Dec 19, 2024
b841a91
shell: add rank/task specific mustache render functions to api
grondo Dec 19, 2024
bf9f2fe
shell: output: drop unused `term` output implementation
grondo Dec 19, 2024
92e1e2c
shell: output: refactor output plugin
grondo Dec 20, 2024
fb2a0c0
shell: output: split kvs eventlog ops into separate file
grondo Dec 23, 2024
7fc0afe
shell: output: split config out of main output plugin
grondo Dec 24, 2024
b2a701f
shell: output: support per-task output
grondo Dec 25, 2024
cbf0343
shell: output: write shell.output data to correct task
grondo Jan 4, 2025
abf0268
shell: output: move service specific code to output/service.c
grondo Dec 27, 2024
65ba2ed
shell: output: move log output handling to output/log.[ch]
grondo Jan 4, 2025
1d618c4
shell: output: move output.c to output subdir
grondo Jan 4, 2025
ffd3df8
shell: output: update top-level plugin comment
grondo Jan 6, 2025
4f5f5ca
doc: update documentation for output mustache templates
grondo Jan 5, 2025
00a337f
testsuite: add tests for per-node/task output redirection
grondo Jan 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions doc/man1/common/job-mustache-templates.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,6 @@ these include:
Task-specific tags are prefixed with *task.*, with the special case of
*{{taskid}}*, which is an alias for *{{task.id}}*.

.. note::
The :option:`--output` and :option:`--error` options do not support
task-specific tags at this time.

*{{task.id}}*, *{{task.rank}}*, or *{{taskid}}*
Expands to the global rank for the current task.
Expand Down
8 changes: 5 additions & 3 deletions doc/man1/common/job-standard-io.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
.. note::
Paths specified in the options :option:`--output`, :option:`--error`,
and :option:`--input` will be opened relative to the working directory
on the first node of the job allocation, not on the node from which
the job is submitted.
of the job on the nodes on which the job is running, not on the node
from which the job is submitted.

.. option:: --input=FILENAME|RANKS

Expand All @@ -13,7 +13,9 @@
.. option:: --output=TEMPLATE

Specify the filename *TEMPLATE* for stdout redirection, bypassing
the KVS. *TEMPLATE* may be a mustache template.
the KVS. *TEMPLATE* may be a mustache template. If the mustache template
contains node or task-specific tags, then a different output file will
be opened per node or task, respectively.
See `MUSTACHE TEMPLATES`_ below for more information.

.. option:: --error=TEMPLATE
Expand Down
5 changes: 5 additions & 0 deletions src/common/libsubprocess/fork.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@

#include "src/common/libczmqcontainers/czmq_containers.h"
#include "src/common/libutil/fdwalk.h"
#include "src/common/libutil/fdutils.h"
#include "src/common/libutil/llog.h"

#include "subprocess_private.h"
Expand Down Expand Up @@ -164,9 +165,13 @@ static int local_child (flux_subprocess_t *p)
idset_destroy (ids);

if (p->hooks.pre_exec) {
fd_set_nonblocking (STDERR_FILENO);
fd_set_nonblocking (STDOUT_FILENO);
p->in_hook = true;
(*p->hooks.pre_exec) (p, p->hooks.pre_exec_arg);
p->in_hook = false;
fd_set_blocking (STDERR_FILENO);
fd_set_blocking (STDOUT_FILENO);
}

if (!(p->flags & FLUX_SUBPROCESS_FLAGS_NO_SETPGRP)
Expand Down
17 changes: 16 additions & 1 deletion src/shell/Makefile.am
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,22 @@ flux_shell_SOURCES = \
input/service.c \
input/file.c \
input/kvs.c \
output.c \
output/filehash.h \
output/filehash.c \
output/client.h \
output/client.c \
output/kvs.h \
output/kvs.c \
output/conf.h \
output/conf.c \
output/task.h \
output/task.c \
output/service.h \
output/service.c \
output/log.h \
output/log.c \
output/output.h \
output/output.c \
svc.c \
svc.h \
kill.c \
Expand Down
Loading
Loading