Comparison of versions

[MarieS-WiMLDS]

We had the feedback from several users, and it's in the natural flow of data scientists, that they want to compare several versions of items. Indeed, once the data scientist has experimented several things (models, pipelines, hyperparameters...), they want to compare the various results to take the one that best fit their needs. It is illustrated by this issue for instance: #1038.

Let's take a step back: what do the data scientists most usually need, and what will bring the most value?
What they usually want to do is:

• compare metrics (= floats)
• compare plots, such as roc curves. This one is more tricky because they can be seaborn, matplotlib, plotly objects, or even svg or anything like this.

What they want to compare is actually what we provide in our reporters. This has the additional advantage that we know the context. Therefore it seems logical to start to build a comparison (/evolution) system based on our objects, at least at first.

How do users want to have this comparison provided?
There are two uses cases, which correspond to 2 steps of the investigation:

1. to start, they want to grasp trends, and be able to select the best option (whenever this makes sense)
2. when necessary, they want to dig further to evaluate according to several parameters.

Today, how far are we from it? I think that our biggest challenge to make something useful is to be able to identify experiments. Should we ask the user to label them? If so, how? How can we identify that we deal with the same entities (most importantly: the dataset)?

exemple of design

API

from skore import CrossValidationReporter
from sklearn import plenty of models, make_classification

project.create("test", overwrite = True)

X, y = make_classification()
models_to_test = [DecisionTreeClassifier(), LogisticRegression(), GradienBoostingClassifier()]

cv_reporter_baseline1 = CrossValidationReporter(baseline_model, X, y)
cv_reporter_baseline2 = CrossValidationReporter(baseline_model_2, X, y)

for model in models_to_test: 
  cv_reporter_obj = CrossValidationReporter(model, X, y)
  project.put("cv_reporter_id", cv_reporter_obj)

comp_v = skore.compare_versions("cv_reporter_id", project = project)
comp_items = skore.compare(cv_reporter_obj, cv_reporter_baseline, cv_reporter_baseline2)

Front end

[sylvaincom]

@Vincent-Maladiere's take in #889