[Bug]: When using add() to insert embeddings and ids, I found that the maximum number is only 100. Why?

[13hgj14]

What happened?

start_time = time.time()
client = chromadb.PersistentClient(path="./chroma")
# client = chromadb.Client()
end_time = time.time()
print(end_time - start_time)

start_time = time.time()
collection = client.get_or_create_collection(name="test_collection", 
                                            metadata={"hnsw:space": "cosine",
                                                      "hnsw:search_ef": 100})
start_time = time.time()
results = collection.get()
end_time = time.time()
print(end_time - start_time)
print(results)

embedding = np.random.randn(1000, 768).astype(np.float32).tolist()
num = 0
keys = [str(i) for i in range(num, num + embedding.__len__())]
metadatas = [{"keys":v} for v in keys]
print(keys[:10])
print(embedding.__len__())
print(metadatas.__len__())

dict_ = {
    "id1": {"source": "doc1"},
    "id2": {"source": "doc2"},
    "id3": {"source": "doc3"}
}

try:
    collection.add(
        ids=keys,           
        embeddings=embedding,
        # metadatas=list(dict_.values()),  
        metadatas=metadatas,  
    )
   print(“success”)
except Exception as e:
   print("defeat")

start_time = time.time()
results = collection.get()
end_time = time.time()
print(end_time - start_time)
print(results)

Versions

Chroma 0.6.1, Windows

Relevant log output

[tazarov]

@13hgj14, what is the exact error you get? There are cases where your sqlite3 system library in such a way that limits the number of inserts that Chroma can do.

Can you also try:

docker run -d --rm  -p 8000:8000 --name chromadb -v ./chroma:/chroma/chroma -e IS_PERSISTENT=TRUE chromadb/chroma:0.6.1

Then use the same code above but with HttpClient instead.