You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
process: Would need Collect to be finished, and would need a new API endpoint (e.g. /reload/), which would delete all collection files and derived collections (and most notes), and then add new files from disk. Otherwise, would need to do same as for Collect (cancel and start new).
pelican: Can skip implementation, as its current usage for coverage will be replaced with Cardinal. If we reintroduce Pelican later (e.g. via Allow JOB_TASKS_PLAN to be configurable per publication #304 to run more checks on priority datasets), then we can look into its retry logic. See old docs below.
We can start with the easily retryable tasks (exporter, flattener) and create a follow-up issue for Process.
This issue is mostly relevant for flattener, which seems to fail sometimes, but we don't want to have to re-run the entire job.
Add admin action
Update the docs that refer to this issue (#354) and surrounding text.
The old docs were:
Pelican
Delete the dataset, using Pelican backend's ``remove`` `command <https://pelican-backend.readthedocs.io/en/latest/tasks/datasets.html#remove>`__.
Change the status of the Pelican task and subsequent tasks to ``PLANNED``, then change the status of the job to ``RUNNING``.
Exporter
Publish a message from the :ref:`Django shell<django-shell>`, using the compiled collection in Kingfisher Process:
.. code-block:: bash from exporter.util import publish publish({"job_id": 123, "collection_id": 456}, "exporter_init")
Flattener
Delete the ``.csv.tar.gz.lock`` files in the job's directory within the ``EXPORTER_DIR`` :ref:`directory<env-exporter-flattener>`.
Publish a message from the :ref:`Django shell<django-shell>`:
.. code-block:: bash from exporter.util import publish publish({"job_id": 123}, "flattener_init")
The text was updated successfully, but these errors were encountered:
Maybe another way to do it is to set the task back to PLANNED and to have PLANNED always perform retry logic (assuming such logic is safe).
Comment I had written about Process:
# For this task to be retryable, it needs to finish (in case any messages are queued), and then call a new API
# endpoint in Kingfisher Process (e.g. /collections/<pk>/reload/) that deletes the collection's collection files,
# derived collections and (most) collection notes and re-adds the files from the filesystem.
And code for TaskManager:
@property@abstractmethoddefretryable(self) ->bool:
""" Whether the task can be retried. """@abstractmethoddefretry(self) ->None:
""" Restart the task. """
/reload/)
, which would delete all collection files and derived collections (and most notes), and then add new files from disk. Otherwise, would need to do same as for Collect (cancel and start new).We can start with the easily retryable tasks (exporter, flattener) and create a follow-up issue for Process.
This issue is mostly relevant for flattener, which seems to fail sometimes, but we don't want to have to re-run the entire job.
Add admin action#354
) and surrounding text.The old docs were:
The text was updated successfully, but these errors were encountered: