[FEATURE] enhance model_uploader workflow to support MIT-licensed models from huggingface #388

zhichao-aws · 2024-04-26T05:04:45Z

Description

Code changes to enhance model upload workflow to support MIT licensed models from huggingface

Issues Resolved

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws · 2024-04-26T08:58:10Z

The workflow works well in my repo GH actions (I commented out the manual approve part)
They traced model and upload to S3 and create PR in my repo
link:
https://github.com/zhichao-aws/opensearch-py-ml/actions/runs/8843390731
https://github.com/zhichao-aws/opensearch-py-ml/actions/runs/8843323304

autocut update PR:
zhichao-aws#7
zhichao-aws#8

zhichao-aws · 2024-04-26T08:59:28Z

With this workflow, to upload BGE-small models, we trigger it with these settings, and the last input MIT license url should be : https://github.com/FlagOpen/FlagEmbedding/raw/master/LICENSE

zhichao-aws · 2024-04-26T09:00:19Z

With this workflow, to upload BGE-small models, we trigger it with these settings, and the last input MIT license url should be : https://github.com/FlagOpen/FlagEmbedding/raw/master/LICENSE

To upload BGE-base, change the model id to BAAI/bge-base-en-v1.5

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws · 2024-04-26T13:08:01Z

One screenshot for the upload workflow at my GH repo

zhichao-aws · 2024-05-07T06:36:17Z

This PR is ready for review.

dhrubo-os · 2024-05-13T01:44:40Z

.github/workflows/model_uploader.yml

@@ -49,7 +57,10 @@ on:
        options:
        - "NO"
        - "YES"
-
+      MIT_license_url:
+        description: "(Optional) MIT license url of the huggingface MIT model."


Does every model have different MIT licenses?

There should be a standard statements for MIT licenses, but the copyright header is customized according to the author info

dhrubo-os · 2024-05-13T01:46:56Z

.github/workflows/model_uploader.yml

-        echo "verified=:white_check_mark: — It is verified that this model is licensed under Apache 2.0" >> $GITHUB_OUTPUT
-        echo "unverified=- [ ]  :warning: The license cannot be verified. Please confirm by yourself that the model is licensed under Apache 2.0  :warning:" >> $GITHUB_OUTPUT
+        echo "verified_apache=:white_check_mark: — It is verified that this model is licensed under Apache 2.0" >> $GITHUB_OUTPUT
+        echo "verified_mit=:white_check_mark: — It is verified that this model is licensed under MIT, and we have provided copyright statements" >> $GITHUB_OUTPUT


For apache 2.0 licensed models, will this make any issue?

I've done a regression test using Apache-2 models and it still works well https://github.com/zhichao-aws/opensearch-py-ml/actions/runs/8843322311/job/24283396611

For apache 2.0, I'm seeing:

- Model License Apache-2.0 - Model Version: 1.0.0 - Tracing Format: BOTH - Embedding Dimension: N/A - Pooling Mode: N/A - Model Description: N/A - MIT License Url: N/A

I was thinking may be we can make this bit more dynamic? Like in MIT License Url, MIT could be picked up from the Model License? So that later if we use any other different licenses, it won't show only MIT?

dhrubo-os · 2024-05-13T02:44:34Z

utils/model_uploader/third_party_statements.py

+
+    # find the copyright statements from origin MIT license. It looks like: Copyright (c) {year} {authorname}
+    copyright_statements = re.findall("Copyright.*\n", license_text)[0].strip()
+    huggingface_url = "https://huggingface.co/" + model_id


We are giving model source from the UI. Can this model source be dynamic?

Yes, the model_id and mit_license_url should be provided at the UI chart

I meant about huggingface.co--> Currently we have model source field in the UI. Can we leverage that?

dhrubo-os · 2024-05-13T02:46:02Z

utils/model_uploader/model_autotracing.py

@@ -428,12 +449,15 @@ def main(
    :type pooling_mode: string
    :param model_description: Model description input
    :type model_description: string
+    :param third_party_copyrights_statements: Statements text for non Apache-2.0 licensed third party model. Should be put in the final artifact.


are we sure that for any kind non Apahe-2.0 licenses we need this copy right statements? Or this is only for MIT licenses?

Now we only know we need third party statements for MIT models, and know the rule to generate third-party file applies for BGE models. We still need to confirm the generating rule applies for other MIT licensed models, to confirm we need third-party file and how to generate third-party file for other non Apahe-2.0 licenses

dhrubo-os · 2024-05-13T02:48:32Z

.github/workflows/model_uploader.yml

+              echo "missing MIT license url"
+              exit 1
+          fi
+          if [[ "${{ github.event.inputs.model_source }}" == "" ]]


what if we provide some other source?

Then the third party statements can be wrong. So we need to check whether they are matched before approve the release

What I tried to mean is let's say github.event.inputs.model_source == "XYZ" then this condition will pass, right?

opensearch_py_ml/ml_models/sentencetransformermodel.py

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws · 2024-05-13T03:45:33Z

@dhrubo-os Any clue why the IT fails? It seems not related to the change, is it a flaky test?

.ci/run-repository.sh

.github/workflows/model_uploader.yml

dhrubo-os · 2024-05-13T17:36:04Z

opensearch_py_ml/ml_models/sentencetransformermodel.py

@@ -38,6 +38,7 @@
 )

 LICENSE_URL = "https://github.com/opensearch-project/opensearch-py-ml/raw/main/LICENSE"
+THIRD_PARTY_FILE_NAME = "THIRD-PARTY"


Could you please check with the open source engineer that if naming the attribution file as THIRD-PARTY looks good to him or not?

The name THIRD_PARTY is provided by the open source engineer

dhrubo-os · 2024-05-13T17:39:51Z

opensearch_py_ml/ml_models/sentencetransformermodel.py

+        if add_apache_license == True and third_party_copyrights_statements is not None:
+            assert (
+                False
+            ), "When the model is from third party under non Apache-2.0 license, we can not add Apache-2.0 license for it."


My understanding is, we will still distribute the artifacts under apache 2.0 license but for MIT licenses models, we also need to add extra attribution file to deliberately mention about the contributors name. Am I missing anything?

dhrubo-os · 2024-05-13T17:44:32Z

@dhrubo-os Any clue why the IT fails? It seems not related to the change, is it a flaky test?

Yeah, I think that was a flaky test.

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws · 2024-07-03T05:43:54Z

close it for the issue of BGE training data. Feel free to re-open it when we need to upload other MIT models

zhichao-aws added 20 commits April 23, 2024 17:26

add fields and check

e70d699

Signed-off-by: zhichao-aws <[email protected]>

add code for MIT licensed models

dcea54e

Signed-off-by: zhichao-aws <[email protected]>

license line; remove manual approve

607336c

Signed-off-by: zhichao-aws <[email protected]>

rename to third_party_statements_text

7adfbf3

Signed-off-by: zhichao-aws <[email protected]>

change to copyright statements

7d18378

Signed-off-by: zhichao-aws <[email protected]>

change to copyright statements

a5c6627

Signed-off-by: zhichao-aws <[email protected]>

add fi

46b99d7

Signed-off-by: zhichao-aws <[email protected]>

add quote to string var

efbc533

Signed-off-by: zhichao-aws <[email protected]>

fix quote

3e9871d

Signed-off-by: zhichao-aws <[email protected]>

fix quote

20d4fc8

Signed-off-by: zhichao-aws <[email protected]>

log failure state

337ffa9

Signed-off-by: zhichao-aws <[email protected]>

use url

a007301

Signed-off-by: zhichao-aws <[email protected]>

lint

e997594

Signed-off-by: zhichao-aws <[email protected]>

improve error log

08540b2

Signed-off-by: zhichao-aws <[email protected]>

lint

98794e0

Signed-off-by: zhichao-aws <[email protected]>

fix

70f0713

Signed-off-by: zhichao-aws <[email protected]>

add test

fe0471a

Signed-off-by: zhichao-aws <[email protected]>

lint

db49002

Signed-off-by: zhichao-aws <[email protected]>

changelog

4200890

Signed-off-by: zhichao-aws <[email protected]>

finalize

34a18d6

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws marked this pull request as ready for review April 26, 2024 09:00

zhichao-aws requested review from dhrubo-os, greaa-aws, ylwu-amzn, b4sjoo, jngz-es and rbhavna as code owners April 26, 2024 09:00

lint

5f6e714

Signed-off-by: zhichao-aws <[email protected]>

dhrubo-os reviewed May 13, 2024

View reviewed changes

change third_party to static var

5da6ae9

Signed-off-by: zhichao-aws <[email protected]>

dhrubo-os reviewed May 13, 2024

View reviewed changes

.ci/run-repository.sh Outdated Show resolved Hide resolved

dhrubo-os reviewed May 13, 2024

View reviewed changes

.github/workflows/model_uploader.yml Outdated Show resolved Hide resolved

dhrubo-os reviewed May 13, 2024

View reviewed changes

change mit_license to addtional license

4f6af15

Signed-off-by: zhichao-aws <[email protected]>

zhichao-aws closed this Jul 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] enhance model_uploader workflow to support MIT-licensed models from huggingface #388

[FEATURE] enhance model_uploader workflow to support MIT-licensed models from huggingface #388

zhichao-aws commented Apr 26, 2024

zhichao-aws commented Apr 26, 2024 •

edited

Loading

zhichao-aws commented Apr 26, 2024 •

edited

Loading

zhichao-aws commented Apr 26, 2024 •

edited

Loading

zhichao-aws commented Apr 26, 2024

zhichao-aws commented May 7, 2024

dhrubo-os May 13, 2024

zhichao-aws May 13, 2024

dhrubo-os May 13, 2024

zhichao-aws May 13, 2024

dhrubo-os May 13, 2024

dhrubo-os May 13, 2024

zhichao-aws May 13, 2024

dhrubo-os May 13, 2024

dhrubo-os May 13, 2024

zhichao-aws May 13, 2024 •

edited

Loading

dhrubo-os May 13, 2024

zhichao-aws May 13, 2024

dhrubo-os May 13, 2024

zhichao-aws commented May 13, 2024

dhrubo-os May 13, 2024

zhichao-aws May 14, 2024

dhrubo-os May 13, 2024 •

edited

Loading

dhrubo-os commented May 13, 2024

zhichao-aws commented Jul 3, 2024

[FEATURE] enhance model_uploader workflow to support MIT-licensed models from huggingface #388

[FEATURE] enhance model_uploader workflow to support MIT-licensed models from huggingface #388

Conversation

zhichao-aws commented Apr 26, 2024

Description

Issues Resolved

Check List

zhichao-aws commented Apr 26, 2024 • edited Loading

zhichao-aws commented Apr 26, 2024 • edited Loading

zhichao-aws commented Apr 26, 2024 • edited Loading

zhichao-aws commented Apr 26, 2024

zhichao-aws commented May 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhichao-aws May 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhichao-aws commented May 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhrubo-os May 13, 2024 • edited Loading

Choose a reason for hiding this comment

dhrubo-os commented May 13, 2024

zhichao-aws commented Jul 3, 2024

zhichao-aws commented Apr 26, 2024 •

edited

Loading

zhichao-aws commented Apr 26, 2024 •

edited

Loading

zhichao-aws commented Apr 26, 2024 •

edited

Loading

zhichao-aws May 13, 2024 •

edited

Loading

dhrubo-os May 13, 2024 •

edited

Loading