[v4.0] feat: add new tasks for image classification v2

[anhappdev]

Closes #719

This is how the app looks with 2 new tasks:

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[freedomtan]

version name scheme: should we call it Classification V2 (after model name) or V3.1 (align with the app version)? Let's discuss this in main mobile working group.

[freedomtan]

we can move the two mobilenet edgetpu models to the end of the list later

[freedomtan]

Let's call it V2 (not V2.0).

[anhappdev]

Screenshot updated in the first comment.

[anhappdev]

The accuracy test of image_classification_offline_v2 failed on Pixel 6 with Pixel backend because the accuracy is 0.
But image_classification_v2 on Pixel 6 passed the test.
Other backends (TFLite, Core ML) also passed the test.

@freedomtan Do you have an idea what can be the reason for this?

[freedomtan]

The accuracy test of image_classification_offline_v2 failed on Pixel 6 with Pixel backend because the accuracy is 0. But image_classification_v2 on Pixel 6 passed the test. Other backends (TFLite, Core ML) also passed the test.

@freedomtan Do you have an idea what can be the reason for this?

Yes, I ran into the same problem on Pixel 7 before. It seems you have to reduce the batch size to a much smaller one. I guess it’s a bug or limitation of EdgeTPU.

[anhappdev]

Yes, I ran into the same problem on Pixel 7 before. It seems you have to reduce the batch size to a much smaller one. I guess it’s a bug or limitation of EdgeTPU.

Thank you. Reducing the batch size worked.

[freedomtan]

let's rebase this one and create an 4.0 branch.

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

[freedomtan]

Let's rebase this to current master branch and make the model path local (local://) and without checksum, so that it's easier to test new models (we may get more than one model to test).

[anhappdev]

@freedomtan

Let's rebase this to current master branch

Done

and make the model path local (local://) and without checksum, so that it's easier to test new models (we may get more than one model to test).

Each vendor should add a new benchmark_setting to its setting file like this, and can set its own path to local for testing.

benchmark_setting {
  benchmark_id: "image_classification_v2"
  framework: "TFLite"
  delegate_choice: {
    delegate_name: "CPU"
    accelerator_name: "cpu"
    accelerator_desc: "CPU"
    model_path: "https://github.com/mlcommons/mobile_models/raw/anh/add-MNEdgeTPU-V2-L/v3_1/tflite/mobilenet-edgetpu-v2-L_fp32.tflite"
    model_checksum: "4b7ce0a82f8eaa542404d2ab2ee4e575"
  }
}

benchmark_setting {
  benchmark_id: "image_classification_offline_v2"
  framework: "TFLite"
  delegate_choice: {
    delegate_name: "CPU"
    accelerator_name: "cpu"
    accelerator_desc: "CPU"
    batch_size: 2
    model_path: "https://github.com/mlcommons/mobile_models/raw/anh/add-MNEdgeTPU-V2-L/v3_1/tflite/mobilenet-edgetpu-v2-L_fp32.tflite"
    model_checksum: "4b7ce0a82f8eaa542404d2ab2ee4e575"
  }
}

Or should I remove the checksum and set the path to local for tflite_settings_android.pbtxt?

[freedomtan]

@freedomtan

Let's rebase this to current master branch

Done

we need max_duration for new benchmark items.

diff --git a/flutter/assets/tasks.pbtxt b/flutter/assets/tasks.pbtxt
index 5a113ca..d7cf0be 100644
--- a/flutter/assets/tasks.pbtxt
+++ b/flutter/assets/tasks.pbtxt
@@ -221,6 +221,7 @@ task {
   name: "Image Classification v2"
   min_query_count: 1024
   min_duration: 60
+  max_duration: 600
   max_throughput: 1000
   max_accuracy: 1.0
   scenario: "SingleStream"
@@ -257,6 +258,7 @@ task {
   name: "Image Classification v2 (Offline)"
   min_query_count: 24576
   min_duration: 0
+  max_duration: 600
   max_throughput: 2000
   max_accuracy: 1.0
   scenario: "Offline"

[anhappdev]

we need max_duration for new benchmark items.

Done. max_duration does not work in Offline scenario, so I set it to zero.

[freedomtan]

@mohitmundhragithub and @AhmedTElthakeb for some new models, we need different preprocessing, I have a hack based on this one, 1d9e1e9

[mohitmundhragithub]

@mohitmundhragithub and @AhmedTElthakeb for some new models, we need different preprocessing, I have a hack based on this one, 1d9e1e9

this is for which model?
I think MobileNetEdgeTPUv2 model was working fine... no?

[freedomtan]

@freedomtan to send PRs for different input sizes for new models (with different preprocessing). 224x224, 256x256, 384x384

[anhappdev]

Should we merge this to submission-v4.0 branch first? And then after the submission, we merge submission-v4.0 into master.

[freedomtan]

Should we merge this to submission-v4.0 branch first? And then after the submission, we merge submission-v4.0 into master.

I am fine with either way. Let's discuss it in the meeting.

[freedomtan]

1. let's merge this to submission-v4.0 first, and back to master when we have good qualcomm backend for the classification V2.
2. change order of test as @mohitmundhragithub suggested.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud