Jobs return standardized and flexible output

[njbrake]

What's changing

Tl;dr this design de-couples the presentation (frontend) layer from the data management/storage layer (backend), as well as the execution layer (job). Although the frontend still has hardcoded references to several keys, they are now organized in such a way that the frontend can implement more dynamic techniques and management of keys when they're ready.

In order to make it easier to add an manage jobs as they are added, this PR creates a loose interface between job result (in the result.json file) and the backend.

The job can create a schema as they please as long as the main structure of the file they save has only the keys "artifacts", "parameters", and "metrics". This allows for easy storage in the tracking interface (e.g. mlflow) and easy access for the frontend.

The design focuses on keeping the jobs as a flexible entity that doesn't require knowledge about how the frontend works. The design also allows for new visualizations of existing data, which means that the frontend is able to make decisions about how to display the data, and it isn't tied to the backend.

If this PR is related to an issue or closes one, please link it here.

Refs #678

How to test it

The unit tests handle testing of the SDK and the backend, but this should be tested by running make local-up and then using the Web UI to upload a dataset, create an experiment, and view all the results.

Additional notes for reviewers

Anything you'd like to add to help the reviewer understand the changes you're proposing.

I already...

• Tested the changes in a working environment to ensure they work as expected
• Added some tests for any new functionality
• Updated the documentation (both comments in code and product documentation under /docs)
• Checked if a (backend) DB migration step was required and included it if required

[njbrake]

@khaledosman adding you to this review for the frontend perspective. I ran the UI to ensure that the changes I made to the Vue code don't break anything I could see. I think that these changes should enable you to simplify how you load and organize the data you get back from the backend.

[javiermtorres]

LGTM. Only a couple of issues:

• Add another layer between artifacts and the columns (admittedly waivable if we assume this specific jobs will contain only these columns and will only accept one input, but would make it more generic, like having several in/out params)
• Try to shape the result object fields (params, metrics, etc) into a more "appropriate" type that can be more flexible and at the same time more restrictive. Since I don't have a firm proposal yet :-/ we can leave it for next PR, but just let me know your thoughts on this.

[njbrake]

Thanks for the review, Javier! You made some great points 💯

[njbrake]

LGTM. Only a couple of issues:

* Add another layer between `artifacts` and the columns (admittedly waivable if we assume this specific jobs will contain only these columns and will only accept one input, but would make it more generic, like having several in/out params)

Agree that this needs to be improved. I created #835 to track this, because implementing this change will require refactors in a handful of places and I'd like to prevent this PR from getting too unwieldy.

* Try to shape the result object fields (params, metrics, etc) into a more "appropriate" type that can be more flexible and at the same time more restrictive. Since I don't have a firm proposal yet :-/ we can leave it for next PR, but just let me know your thoughts on this.

This is a good comment and I think in a way it will be evergreen: there's this push and pull between making the output type standardized, opinionated, but also flexible. I liked your suggestion of turning the fields into at least Pydantic BaseModels and initializing them as empty dicts if the fields aren't provided; I implemented that change. I think that this will require more refinement down the road, but at least now this gets us into a world where we have three high level buckets to put things in for the SDK and frontend to access: parameters, metrics, and artifacts.

[njbrake]

• Try to shape the result object fields (params, metrics, etc) into a more "appropriate" type that can be more flexible and at the same time more restrictive. Since I don't have a firm proposal yet :-/ we can leave it for next PR, but just let me know your thoughts on this.

This is a good comment and I think in a way it will be evergreen: there's this push and pull between making the output type standardized, opinionated, but also flexible. I liked your suggestion of turning the fields into at least Pydantic BaseModels and initializing them as empty dicts if the fields aren't provided; I implemented that change. I think that this will require more refinement down the road, but at least now this gets us into a world where we have three high level buckets to put things in for the SDK and frontend to access: parameters, metrics, and artifacts.

Update: I had to undo the suggestion of changing the JobResultOutput fields into Pydantic BaseModels: pydantic models require you to know all of the keys that you want the class to hold, so it can't function as an arbitrary dict. :(

[javiermtorres]

Now I remember this issue about the use of Base model :( Thanks for trying out anyway!