Red teaming blogpost #849

nazneenrajani · 2023-02-23T03:03:01Z

No description provided.

natolambert

Seems like there is duplicated assets red-teaming.png and thumbnail.png. I would follow this in the huggingface/blog readme:

This folder will contain your thumbnail only. The folder number is mostly for (rough) ordering purposes, so it's no big deal if two concurrent articles use the same number.

For the rest of your files, create a mirrored folder in the HuggingFace Documentation Images [repo](https://huggingface.co/datasets/huggingface/documentation-images/tree/main/blog). This is to reduce bloat in the GitHub base repo when cloning and pulling.

Also, let's move to the new blog post format (so we don't break anything / the post is formatted weird. I can help with this once you go through the suggestions.

Ex.

---
title: "Illustrating Reinforcement Learning from Human Feedback (RLHF)" 
thumbnail: /blog/assets/120_rlhf/thumbnail.png
authors:
- user: natolambert
- user: LouisCastricato
  guest: true
- user: lvwerra
- user: Dahoas
  guest: true
---

# Illustrating Reinforcement Learning from Human Feedback (RLHF)

<!-- {blog_metadata} -->
<!-- {authors} -->

Text starts here....

natolambert · 2023-02-23T20:56:36Z

_blog.yml

+  title: "Red-Teaming Large Language Models"
+  author: nazneen
+  thumbnail: /blog/assets/red-teaming/thumbnail.png
+  date: February 22, 2023


Maybe update to when we want to post it? (in case other blogs are posted after this date, for sorting / be on blog front page). In this vein, I'd move it to the bottom of _blog.yml

natolambert · 2023-02-23T20:58:49Z

red-teaming.md

+        </div>
+    </a>
+    </div>
+Large language models (LLMs) trained on an enormous amount of text data are very good at generating realistic text. However, these models often exhibit undesirable behaviors like revealing personal information (such as social security numbers) and generating misinformation, bias, hatefulness, or toxic content. For example, GPT3 is known to be sexist (see below) and [biased against Muslims](https://dl.acm.org/doi/abs/10.1145/3461702.3462624),


Suggested change

Large language models (LLMs) trained on an enormous amount of text data are very good at generating realistic text. However, these models often exhibit undesirable behaviors like revealing personal information (such as social security numbers) and generating misinformation, bias, hatefulness, or toxic content. For example, GPT3 is known to be sexist (see below) and [biased against Muslims](https://dl.acm.org/doi/abs/10.1145/3461702.3462624),

Large language models (LLMs) trained on an enormous amount of text data are very good at generating realistic text. However, these models often exhibit undesirable behaviors like revealing personal information (such as social security numbers) and generating misinformation, bias, hatefulness, or toxic content. For example, earlier versions of GPT3 were known to be sexist (see below) and [biased against Muslims](https://dl.acm.org/doi/abs/10.1145/3461702.3462624),

Is the current GPT3 version still showing this behavior? If not, I would say a version, like above.

red-teaming.md

natolambert · 2023-02-23T21:19:10Z

red-teaming.md

+
+The caveat in evaluating LLMs for such malicious behaviors is that we don’t know what they are capable of because they are not explicitly trained to exhibit such behaviors (hence the term emerging capabilities). The only way is to actually simulate scenarios and evaluate for the model would behave. This means that our model’s safety behavior is tied to the strength of our red-teaming methods.
+
+**Open source datasets for Red-teaming:**


Any of these on the hub / can we try to port before posting??

Yup they are on the hub.

red-teaming.md

natolambert

added some more suggestions (like the formatting fix for the header)

natolambert · 2023-02-24T03:14:16Z

red-teaming.md

+**Open source datasets for Red-teaming:**
+
+1. Meta’s [Bot Adversarial Dialog dataset](https://aclanthology.org/2021.naacl-main.235.pdf)
+2. Anthropic’s [red-teaming attempts](https://github.com/anthropics/hh-rlhf/tree/master/red-team-attempts)


Suggested change

2. Anthropic’s [red-teaming attempts](https://github.com/anthropics/hh-rlhf/tree/master/red-team-attempts)

2. Anthropic’s [red-teaming attempts](https://huggingface.co/datasets/Anthropic/hh-rlhf/tree/main/red-team-attempts)

natolambert · 2023-02-24T03:15:05Z

red-teaming.md

+
+1. Meta’s [Bot Adversarial Dialog dataset](https://aclanthology.org/2021.naacl-main.235.pdf)
+2. Anthropic’s [red-teaming attempts](https://github.com/anthropics/hh-rlhf/tree/master/red-team-attempts)
+3. AI2’s [RealToxicityPrompts](https://arxiv.org/pdf/2009.11462.pdf) 


Suggested change

3. AI2’s [RealToxicityPrompts](https://arxiv.org/pdf/2009.11462.pdf)

3. Allen Institute for AI’s [RealToxicityPrompts](https://huggingface.co/datasets/allenai/real-toxicity-prompts)

natolambert · 2023-02-24T03:18:25Z

red-teaming.md

+---
+title: "Red-Teaming Large Language Models" 
+thumbnail: /blog/assets/red-teaming/thumbnail.png
+---
+
+# Red-Teaming Large Language Models
+
+<div class="blog-metadata">
+    <small>Published February 22, 2023.</small>
+    <a target="_blank" class="btn no-underline text-sm mb-5 font-sans" href="https://github.com/huggingface/blog/blob/main/red-teaming.md">
+        Update on GitHub
+    </a>
+</div>
+<div class="author-card">
+    <a href="/nazneen"> 
+        <img class="avatar avatar-user" src="https://avatars.githubusercontent.com/u/3278583?v=4?w=200&h=200&f=face" title="Gravatar">
+        <div class="bfc">
+            <code>Nazneen</code>
+            <span class="fullname">Nazneen Rajani</span>
+        </div>
+    </a>
+    </div>


Suggested change

---

title: "Red-Teaming Large Language Models"

thumbnail: /blog/assets/red-teaming/thumbnail.png

---

# Red-Teaming Large Language Models

<div class="blog-metadata">

<small>Published February 22, 2023.</small>

<a target="_blank" class="btn no-underline text-sm mb-5 font-sans" href="https://github.com/huggingface/blog/blob/main/red-teaming.md">

Update on GitHub

</a>

</div>

<div class="author-card">

<a href="/nazneen">

<img class="avatar avatar-user" src="https://avatars.githubusercontent.com/u/3278583?v=4?w=200&h=200&f=face" title="Gravatar">

<div class="bfc">

<code>Nazneen</code>

<span class="fullname">Nazneen Rajani</span>

</div>

</a>

</div>

---

title: "Red-Teaming Large Language Models"

thumbnail: /blog/assets/red-teaming/thumbnail.png

authors:

- user: nazneen

- user: natolambert

---

# Red-Teaming Large Language Models

This should update to the modern formatting.

osanseviero · 2023-02-24T05:39:35Z

red-teaming.md

+    </div>
+Large language models (LLMs) trained on an enormous amount of text data are very good at generating realistic text. However, these models often exhibit undesirable behaviors like revealing personal information (such as social security numbers) and generating misinformation, bias, hatefulness, or toxic content. For example, GPT3 is known to be sexist (see below) and [biased against Muslims](https://dl.acm.org/doi/abs/10.1145/3461702.3462624),
+
+![GPT3](assets/red-teaming/gpt3.png)


Except for the thumbnail, we try to have new assets in https://huggingface.co/datasets/huggingface/documentation-images/tree/main/blog now. That helps keep the git repo smaller

lewtun

Really well written and informative blog post @nazneenrajani 🚀 !

I've left a few minor comments, but otherwise this looks good to publish :)

lewtun · 2023-02-24T13:19:38Z

red-teaming.md

+thumbnail: /blog/assets/red-teaming/thumbnail.png
+authors:
+- user: nazneen
+- user: HuggingFaceH4


woah do Org authors work????

red-teaming.md

lewtun · 2023-02-24T13:22:26Z

red-teaming.md

+  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/red-teaming/gedi.png"/>
+</p>
+
+**Red-teaming** *is a form of evaluation that elicits model vulnerabilities that might lead to undesirable behaviors.* Jailbreaking is another term for red-teaming wherein the LLM is manipulated to break away from its guardrails. [Microsoft’s Chatbot Tay](https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/) launched in 2016 and the more recent [Bing's Chatbot Sydney](https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html) are real-world examples of how disastrous the lack of thorough evaluation of the underlying ML model using red-teaming can be.


nit:

Suggested change

**Red-teaming** *is a form of evaluation that elicits model vulnerabilities that might lead to undesirable behaviors.* Jailbreaking is another term for red-teaming wherein the LLM is manipulated to break away from its guardrails. [Microsoft’s Chatbot Tay](https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/) launched in 2016 and the more recent [Bing's Chatbot Sydney](https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html) are real-world examples of how disastrous the lack of thorough evaluation of the underlying ML model using red-teaming can be.

**Red-teaming** *is a form of evaluation that elicits model vulnerabilities that might lead to undesirable behaviors.* Jailbreaking is another term for red-teaming wherein the LLM is manipulated to break away from its guardrails. [Microsoft’s Chatbot Tay](https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/) launched in 2016 and the more recent [Bing's Chatbot Sydney](https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html) are real-world examples of how disastrous the lack of thorough evaluation of the underlying LLM using red-teaming can be.

Also, do you know who invented the term "red teaming" for LLMs? Perhaps we can mention them early in the blog post with a reference to their paper?

If you look at scholar, there is a deep history of Red Teaming. We can try and find the first LLM paper.

https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=red+teaming+machine+learning&btnG=

lewtun · 2023-02-24T13:33:57Z

red-teaming.md

+
+**Red-teaming** *is a form of evaluation that elicits model vulnerabilities that might lead to undesirable behaviors.* Jailbreaking is another term for red-teaming wherein the LLM is manipulated to break away from its guardrails. [Microsoft’s Chatbot Tay](https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/) launched in 2016 and the more recent [Bing's Chatbot Sydney](https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html) are real-world examples of how disastrous the lack of thorough evaluation of the underlying ML model using red-teaming can be.
+
+The goal of red-teaming language models is to craft a prompt that would trigger the model to generate offensive text. Red-teaming shares some similarities and differences with the more well-known form of evaluation in ML called *adversarial attacks*. The similarity is that both red-teaming and adversarial attacks share the same goal of “attacking” or “fooling” the model to generate offensive content. However, adversarial attacks can be unintelligible to humans, for example, by prefixing a random string (such as “aaabbbcc”) to each prompt as in [Wallace et al., ‘19.](https://aclanthology.org/D19-1221.pdf) Red-teaming prompts, on the other hand, look like regular, natural language prompts.


I wouldn't call the strings in Wallace et al "random". Perhaps use an explicit example (maybe the screenshot from their paper?)

I also suggest using the arxiv link since it's got more details than the published version

Suggested change

The goal of red-teaming language models is to craft a prompt that would trigger the model to generate offensive text. Red-teaming shares some similarities and differences with the more well-known form of evaluation in ML called *adversarial attacks*. The similarity is that both red-teaming and adversarial attacks share the same goal of “attacking” or “fooling” the model to generate offensive content. However, adversarial attacks can be unintelligible to humans, for example, by prefixing a random string (such as “aaabbbcc”) to each prompt as in [Wallace et al., ‘19.](https://aclanthology.org/D19-1221.pdf) Red-teaming prompts, on the other hand, look like regular, natural language prompts.

The goal of red-teaming language models is to craft a prompt that would trigger the model to generate offensive text. Red-teaming shares some similarities and differences with the more well-known form of evaluation in ML called *adversarial attacks*. The similarity is that both red-teaming and adversarial attacks share the same goal of “attacking” or “fooling” the model to generate offensive content. However, adversarial attacks can be unintelligible to humans, for example, by prefixing a random string (such as “aaabbbcc”) to each prompt as in [Wallace et al., ‘19.](https://arxiv.org/abs/1908.07125) Red-teaming prompts, on the other hand, look like regular, natural language prompts.

Actually, on second thought - would prompt injection attacks count as red teaming? If yes, maybe that's more compelling than the offensive references above? See e.g. https://simonwillison.net/2022/Sep/12/prompt-injection/

lewtun · 2023-02-24T13:45:21Z

red-teaming.md

+
+**Open source datasets for Red-teaming:**
+
+1. Meta’s [Bot Adversarial Dialog dataset](https://aclanthology.org/2021.naacl-main.235.pdf)


This doesn't seem to be the right link to the dataset - can we point to one on hf.co?

lewtun · 2023-02-24T13:50:43Z

red-teaming.md

+
+**Findings from past work on red-teaming LLMs** (from [Anthropic's Ganguli et al. 2022](https://arxiv.org/abs/2209.07858) and [Perez et al. 2022](https://arxiv.org/abs/2202.03286))
+
+1. Few-shot-prompted LMs with helpful, honest, and harmless behavior are not harder to red-team than plain LMs.


I would love to see some explicit examples for each of these bullet points (maybe from their paper?)

lewtun · 2023-02-24T13:51:43Z

red-teaming.md

+3. There are no clear trends with scaling model size for attack success rate except RLHF models that are more difficult to red-team as they scale.
+4. Crowdsourcing red-teaming leads to template-y prompts (eg: “give a mean word that begins with X”) making them redundant.
+
+**Future directions:**


Maybe we can add a reference to Anthropic's helpful/harmless and Constitutional AI papers for bleeding edge insights into making this stuff work at scale? https://arxiv.org/abs/2204.05862

Co-authored-by: lewtun <[email protected]>

yjernite

I proposed some edits of offensive/harmless in a few places; I tried to make sure to keep the intention of the original sentence while stressing the role of the deployment context. Let me know what you think!

red-teaming.md

Co-authored-by: Yacine Jernite <[email protected]>

thomwolf

nice blog post @nazneenrajani!

thomwolf · 2023-02-24T22:05:27Z

red-teaming.md

+---
+
+# Red-Teaming Large Language Models
+


Maybe add a quick note warning:

Warning note: this article is about red-teaming and as such contains examples of model generation that may be offensive or upsetting

thomwolf · 2023-02-24T22:07:46Z

red-teaming.md

+
+Red-teaming can reveal model limitations that can cause upsetting user experiences or enable harm by aiding violence or other unlawful activity for a user with malicious intentions. The outputs from red-teaming (just like adversarial attacks) are generally used to train the model to be less likely to cause harm or steer it away from undesirable outputs.
+
+Since red-teaming requires creative thinking of possible model failures, it is a problem with a large search space making it resource intensive. A workaround would be to augment the LLM with a classifier trained to predict whether a given prompt contains topics or phrases that can possibly lead to offensive generations and if the classifier predicts the prompt would lead to a potentially offensive text, generate a canned response. Such a strategy would err on the side of caution. But that would be very restrictive and cause the model to be frequently evasive. So, there is tension between the model being *helpful* (by following instructions) and being *harmless* (or at least less likely to enable harm). This is where red-teaming can be very useful.


Not sure about "This is where red-teaming can be very useful" maybe more that while "red-teaming" is about surfacing problems, solving them in a way that don't render the model useless is not an easy task either and maybe point to some work on pushing this pareto surface like the Constitutional AI paper

thomwolf · 2023-02-24T22:13:30Z

red-teaming.md

+1. Few-shot-prompted LMs with helpful, honest, and harmless behavior are *not* harder to red-team than plain LMs.
+2. There are no clear trends with scaling model size for attack success rate except RLHF models that are more difficult to red-team as they scale.
+3. Models may learn to be harmless by being evasive, there is tradeoff between helpfulness and harmlessness.
+4. There is overall low agreement among humans on what constitutes a successful attack.
+5. The distribution of the success rate varies across categories of harm with non-violent ones having a higher success rate.
+6. Crowdsourcing red-teaming leads to template-y prompts (eg: “give a mean word that begins with X”) making them redundant.


thomwolf · 2023-02-24T22:18:09Z

red-teaming.md

+4. Red-teaming can be resource intensive, both compute and human resource and so would benefit from sharing strategies, open-sourcing datasets, and possibly collaborating for a higher chance of success.
+
+These limitations and future directions make it clear that red-teaming is an under-explored and crucial component of the modern LLM workflow.
+This post is a call-to-action to LLM researchers and HuggingFace's community of developers to collaborate on these efforts for a safe and friendly world :)


Reach out to us (@nazneenrajani @natolambert @lewtun @TristanThrush @yjernite @thomwolf) if you're interested in joining such a collaboration.

nazneenrajani added 8 commits January 23, 2023 19:56

dialog agents blog

af809c4

dialog agents blog

034c90a

dialog agents blog

1a5d397

fix author style

e726aa5

fix author style

1c4b73a

fix author style

357fe6e

fix author style

c309141

red teaming blog

f1a5768

nazneenrajani requested review from natolambert and lewtun February 23, 2023 03:03

nazneenrajani and others added 3 commits February 23, 2023 09:21

Delete dialog-agents.md

eab33ca

remove old files

dc8ecce

Merge branch 'main' into red-teaming

c14a236

natolambert reviewed Feb 23, 2023

View reviewed changes

natolambert approved these changes Feb 24, 2023

View reviewed changes

osanseviero reviewed Feb 24, 2023

View reviewed changes

nazneenrajani added 2 commits February 23, 2023 22:34

edits

f71ec5e

edits

ca34509

nazneenrajani force-pushed the red-teaming branch from ad8a9c7 to ca34509 Compare February 24, 2023 06:44

nazneenrajani added 4 commits February 23, 2023 23:15

edits

1c42aa7

edits

26723fa

edits

b1ecaca

edits

5506753

lewtun approved these changes Feb 24, 2023

View reviewed changes

Update red-teaming.md

0c7daeb

Co-authored-by: lewtun <[email protected]>

yjernite reviewed Feb 24, 2023

View reviewed changes

red-teaming.md Outdated Show resolved Hide resolved

red-teaming.md Outdated Show resolved Hide resolved

red-teaming.md Outdated Show resolved Hide resolved

red-teaming.md Outdated Show resolved Hide resolved

red-teaming.md Outdated Show resolved Hide resolved

nazneenrajani and others added 4 commits February 24, 2023 09:35

Update red-teaming.md

17aadd6

Co-authored-by: Yacine Jernite <[email protected]>

Update red-teaming.md

0b3aab3

Co-authored-by: Yacine Jernite <[email protected]>

Update red-teaming.md

84deadf

Co-authored-by: Yacine Jernite <[email protected]>

Update red-teaming.md

da6e6ff

Co-authored-by: Yacine Jernite <[email protected]>

nazneenrajani and others added 3 commits February 24, 2023 09:35

Update red-teaming.md

513d5ee

Co-authored-by: Yacine Jernite <[email protected]>

edits

4d32ad4

edits

ee4c858

nazneenrajani merged commit 3f1431d into main Feb 24, 2023

nazneenrajani deleted the red-teaming branch February 24, 2023 18:46

thomwolf reviewed Feb 24, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Red teaming blogpost #849

Red teaming blogpost #849

nazneenrajani commented Feb 23, 2023

natolambert left a comment

natolambert Feb 23, 2023

natolambert Feb 23, 2023

natolambert Feb 23, 2023

nazneenrajani Feb 23, 2023

natolambert left a comment

natolambert Feb 24, 2023

natolambert Feb 24, 2023

natolambert Feb 24, 2023

natolambert Feb 24, 2023

osanseviero Feb 24, 2023

nazneenrajani Feb 24, 2023

lewtun left a comment

lewtun Feb 24, 2023

natolambert Feb 24, 2023 •

edited

Loading

lewtun Feb 24, 2023

lewtun Feb 24, 2023

natolambert Feb 24, 2023

lewtun Feb 24, 2023

lewtun Feb 24, 2023

lewtun Feb 24, 2023

lewtun Feb 24, 2023

lewtun Feb 24, 2023

yjernite left a comment •

edited

Loading

thomwolf left a comment

thomwolf Feb 24, 2023

thomwolf Feb 24, 2023

thomwolf Feb 24, 2023

thomwolf Feb 24, 2023 •

edited

Loading


		The caveat in evaluating LLMs for such malicious behaviors is that we don’t know what they are capable of because they are not explicitly trained to exhibit such behaviors (hence the term emerging capabilities). The only way is to actually simulate scenarios and evaluate for the model would behave. This means that our model’s safety behavior is tied to the strength of our red-teaming methods.

		Open source datasets for Red-teaming:

	2. Anthropic’s [red-teaming attempts](https://github.com/anthropics/hh-rlhf/tree/master/red-team-attempts)
	2. Anthropic’s [red-teaming attempts](https://huggingface.co/datasets/Anthropic/hh-rlhf/tree/main/red-team-attempts)

	3. AI2’s [RealToxicityPrompts](https://arxiv.org/pdf/2009.11462.pdf)
	3. Allen Institute for AI’s [RealToxicityPrompts](https://huggingface.co/datasets/allenai/real-toxicity-prompts)


		Red-teaming is a form of evaluation that elicits model vulnerabilities that might lead to undesirable behaviors. Jailbreaking is another term for red-teaming wherein the LLM is manipulated to break away from its guardrails. [Microsoft’s Chatbot Tay](https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/) launched in 2016 and the more recent [Bing's Chatbot Sydney](https://www.nytimes.com/2023/02/16/technology/bing-chatbot-transcript.html) are real-world examples of how disastrous the lack of thorough evaluation of the underlying ML model using red-teaming can be.

		The goal of red-teaming language models is to craft a prompt that would trigger the model to generate offensive text. Red-teaming shares some similarities and differences with the more well-known form of evaluation in ML called adversarial attacks. The similarity is that both red-teaming and adversarial attacks share the same goal of “attacking” or “fooling” the model to generate offensive content. However, adversarial attacks can be unintelligible to humans, for example, by prefixing a random string (such as “aaabbbcc”) to each prompt as in [Wallace et al., ‘19.](https://aclanthology.org/D19-1221.pdf) Red-teaming prompts, on the other hand, look like regular, natural language prompts.


		Open source datasets for Red-teaming:

		1. Meta’s [Bot Adversarial Dialog dataset](https://aclanthology.org/2021.naacl-main.235.pdf)


		Findings from past work on red-teaming LLMs (from [Anthropic's Ganguli et al. 2022](https://arxiv.org/abs/2209.07858) and [Perez et al. 2022](https://arxiv.org/abs/2202.03286))

		1. Few-shot-prompted LMs with helpful, honest, and harmless behavior are not harder to red-team than plain LMs.


		Red-teaming can reveal model limitations that can cause upsetting user experiences or enable harm by aiding violence or other unlawful activity for a user with malicious intentions. The outputs from red-teaming (just like adversarial attacks) are generally used to train the model to be less likely to cause harm or steer it away from undesirable outputs.

		Since red-teaming requires creative thinking of possible model failures, it is a problem with a large search space making it resource intensive. A workaround would be to augment the LLM with a classifier trained to predict whether a given prompt contains topics or phrases that can possibly lead to offensive generations and if the classifier predicts the prompt would lead to a potentially offensive text, generate a canned response. Such a strategy would err on the side of caution. But that would be very restrictive and cause the model to be frequently evasive. So, there is tension between the model being helpful (by following instructions) and being harmless (or at least less likely to enable harm). This is where red-teaming can be very useful.

Red teaming blogpost #849

Red teaming blogpost #849

Conversation

nazneenrajani commented Feb 23, 2023

natolambert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natolambert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewtun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natolambert Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yjernite left a comment • edited Loading

Choose a reason for hiding this comment

thomwolf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomwolf Feb 24, 2023 • edited Loading

Choose a reason for hiding this comment

natolambert Feb 24, 2023 •

edited

Loading

yjernite left a comment •

edited

Loading

thomwolf Feb 24, 2023 •

edited

Loading