-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft: Create experiment, workflow, and tasks documentation from configuration, as well as metadata #1884
base: master
Are you sure you want to change the base?
Conversation
…on, as well as metadata
In GitLab by @kinow on Jan 19, 2024, 12:19 The current code is a draft and only produces:
This is valid Markdown, that could be used to generate a web page, PDF, docx, etc.. The contents of the ini:
FILE: 1.sh
RUNNING: once
TITLE: Initialize the experiment
doc: This task runs locally, the expid is %EXPID% Pasting the content below so GitLab can render the markdown (expanding the Workflow JobsThese are the jobs in the workflow “a001 INIInitialize the experiment SIMASIMPOSTMetadata |
In GitLab by @kinow on Jan 19, 2024, 12:20 /cc @mcastril , @GinkaVanThielen , @ainagaya , @Lerriola , @franra9 , @sebastianbeyer , @fdoblasreyes wrote a quick merge request just now to demonstrate the idea of automated documentation generation ☝️ |
In GitLab by @kinow on Jan 19, 2024, 12:20 Commented on autosubmit/docgen/init.py line 1 Note to self, update to 2024 |
In GitLab by @kinow on Jan 19, 2024, 12:22 And this doesn't need to replace any other documentation planned for models; this can be used to complement any documentation/wiki/deliverable/etc., but automating the parts that contain variables from the workflow. |
In GitLab by @kinow on Jan 19, 2024, 12:36 mentioned in issue digital-twins/de_340/project_management#579 |
In GitLab by @mcastril on Jan 19, 2024, 19:09 This is wonderful Bruno. Can you clarify one doubt?
Wouldn't it be |
In GitLab by @kinow on Jan 22, 2024, 10:01 Yup! That's the initial idea. But we can work on it and decide what would be the best way for that. I think with a few more hours (2...3) I can get it to work and produce the final markdown. With a working prototype we can test in the climatedt workflow or another example workflow, and then discuss what needs changing. |
In GitLab by @ainagaya on Jan 24, 2024, 13:07 mentioned in issue digital-twins/de_340/workflow#174 |
In GitLab by @kinow on Jan 19, 2024, 12:16
Closes #939
Creating a merge request to demonstrate the idea of the automated documentation of experiments. This is not super new, as you can achieve almost the same with Cylc and ecFlow. In Cylc you have DOC and TITLE too. In ecFlow they use MANUAL.
I am extending the scope of the issue (created 11 months ago) to add something that was discussed more recently, about the documentation for models, metadata, ...
The idea is documented in the
docgen/__init__.py
docstrings. Pasting it here:The gist is that the maintainers of the experiment would be able to select configuration values from the workflow that must be exported as workflow metadata (in the
METADATA
YAML object/entry above).The new command in Autosubmit will take care to go over the experiment configuration, resolve variables like
%EXPID%
,%RUN.IFS.RESOLUTION.ETC%
, and produce the metadata list, with the current values used in the experiment.I think this is key here, to have the documentation generated with the values used in the workflow -- i.e. the workflow run configuration is the single source of truth, to avoid having a PDF or Wiki page manually written that might be out of sync with what's actually happening in the workflow.