Skip to content

Releases: IBM/unitxt

Unitxt 1.15.9

01 Dec 13:00
@yoavkatz yoavkatz
1.15.9
22d1fff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Unitxt 1.15.9

Main changes

image

What's Changed

Full Changelog: 1.15.8...1.15.9

Contributors

bnayahu, dafnapension, and 2 other contributors
Assets 2
Loading

Unitxt 1.15.8

26 Nov 15:05
@OfirArviv OfirArviv
1.15.8
70e9908
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Unitxt 1.15.8

Main changes

Added support for RITS Inference Engine

Inference Engines

Assets

Bug Fixes

Full Changelog: 1.15.7...1.15.8

Contributors

martinscooper, OfirArviv, and lilacheden
Assets 2
Loading

1.15.7

22 Nov 18:06
@yoavkatz yoavkatz
1.15.7
db339b6
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
1.15.7

Assets

Usability

Bug fixes

Documentation

Inference Engine

  • Tests and minor changes Changes to GenAI, WML and HF inference engines by @pawelknes in #1290

Full Changelog: 1.15.6...1.15.7

Contributors

elronbandel, dafnapension, and 3 other contributors
Assets 2
Loading

Unitxt 1.15.6 - Chat Inference

19 Nov 21:44
@elronbandel elronbandel
1.15.6
d5a43fc
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Unitxt 1.15.6 - Chat Inference

Main changes

  • Added support for generating output in ChatAPI format (user/assistant turns) and for inference engines to process ChatAPI input.
    See details in blog.

  • Improved catalog browsing experience with cleaner formatting of catalog assets, and clickable hyper links between catalog assets and between catalog assets and code. See for example.

New Features

Inference Engines that support ChatApi interface

Improved multi model support

New Asserts

Performance

Usuability

Documentation

CI/CD

New Contributors

Full Changelog: 1.14.1...1.15.6

Contributors

perlitz, csrajmohan, and 8 other contributors
Assets 2
Loading

Unitxt 1.14.1 - Faster Unitxt 🚀

27 Oct 11:49
@elronbandel elronbandel
1.14.1
3adcc54
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Unitxt 1.14.1 - Faster Unitxt 🚀

Important Change: Unitxt is Faster!

To improve Unitxt’s performance, we've made several optimizations:

  1. Operator Acceleration: Many operators have been sped up by removing unnecessary deep copying in their code, enhancing runtime efficiency.

  2. Caching Hugging Face Datasets: We added the option to cache Hugging Face datasets in loaders, which can prevent redundant loading operations. To enable this, you can either:

    • Set it globally in code: 
      import unitxt

unitxt.settings.disable_hf_datasets_cache = False
    • Use the settings context: 
      with settings.context(disable_hf_datasets_cache=False):
    # your code
    • Or set the environment variable: 
      export UNITXT_DISABLE_HF_DATASETS_CACHE=False

  3. Eager Execution Mode: Running Unitxt without streaming, which can be faster in certain scenarios. Enable eager execution using the environment variable or directly in code:

    unitxt.settings.use_eager_execution = True
# or
with settings.context(use_eager_execution=True):
    # your code

  4. Partial Stream Loading: This feature lets you load only the necessary data instances, avoiding full dataset loads when not required. Here's an example:

    from unitxt import load_dataset

dataset = load_dataset(
    card="cards.doc_vqa.lmms_eval",
    template="templates.qa.with_context.title",
    format="formats.models.llava_interleave",
    loader_limit=300,
    streaming=True,
)
print(next(iter(dataset["test"][0])))  # Loads only the first instance

    Complete Example: Combining the optimizations above can lead to near 1000x faster dataset loading:

    from unitxt import load_dataset, settings

with settings.context(
    disable_hf_datasets_cache=False,
    use_eager_execution=True,
):
    dataset = load_dataset(
        card="cards.doc_vqa.lmms_eval",
        template="templates.qa.with_context.title",
        format="formats.models.llava_interleave",
        loader_limit=300,
        streaming=True,
    )
    print(next(iter(dataset["test"][0])))  # Loads only the first instance

  5. Execution Speed Tracking: A GitHub action has been added to monitor Unitxt’s execution speed in new pull requests, helping ensure that optimizations are maintained.

Summary

This release is focused on accelerating performance in Unitxt by introducing several key optimizations. Operator efficiency has been enhanced by removing deep copies, making operations faster. Users can now enable dataset caching for Hugging Face datasets to prevent redundant loading, configured directly in code or through environment variables. An optional eager execution mode has been added, bypassing streaming to increase speed in certain scenarios. Additionally, partial stream loading allows selective instance loading, reducing memory usage and improving response times. To maintain these improvements, a new GitHub action now monitors Unitxt’s execution speed in pull requests, ensuring consistent performance across updates.

All Changes

New Contributors

Full Changelog: 1.13.1...1.14.1

Contributors

marukaz, perlitz, and 8 other contributors
Assets 2
Loading

Unitxt 1.14.0 - Faster Unitxt

20 Oct 11:55
@elronbandel elronbandel
1.14.0
3d027e9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Unitxt 1.14.0 - Faster Unitxt

What's Changed

New Contributors

Full Changelog: 1.13.0...1.14.0

Contributors

marukaz, perlitz, and 10 other contributors
Assets 2
Loading

Unitxt 1.13.1

30 Sep 07:05
@elronbandel elronbandel
1.13.1
d1219c9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Unitxt 1.13.1 
Update version to 1.13.1 (#1244)
Assets 2
Loading

Unitxt 1.13.0

25 Sep 06:37
@elronbandel elronbandel
1.13.0
80b284f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Unitxt 1.13.0

Unitxt 1.13.0 - Multi Modality and Types

New type handling capabilities

The most significant change in this release is the introduction of type serializers to unitxt.
Type serializers in charge of taking a specific type of data structure such as Table, or Dialog and serialize it to textual representation.
Now you can define tasks in unitxt that have complex types such as Table or Dialog and define serializers that handle their transformation to text.

This allows to control the representation of different types from the recipe api:

from unitxt import load_dataset
from unitxt.struct_data_operators import SerializeTableAsMarkdown

serializer = SerializeTableAsMarkdown(shuffle_rows=True, seed=0)
dataset = load_dataset(card="cards.wikitq", template_card_index=0, serializer=serializer)

And if you want to serialize this table differently you can change any of the many available table serializers.

Defining New Type

If you wish to define a new type with custom serializers you can do so by using python typing library:

from typing import Any, List, TypedDict

class Table(TypedDict):
    header: List[str]
    rows: List[List[Any]]

Once your type is ready you should register it to unitxt type handling within the code you are running:

from unitxt.type_utils import register_type

register_type(Table)

Now your type can be used anywhere across unitxt (e.g in task definition or serializers).

Defining a Serializer For a Type

If you want to define a serializer for your custom type or any typing type combination you can do so by:

class MySerizlizer(SingleTypeSerializer):
    serialized_type = Table
    def serialize(self, value: Table, instance: Dict[str, Any]) -> str:
        # your code to turn value of type Table to string

Multi-Modality

You now can process Image-Text to Text or Image-Audio to Text datasets in unitxt.
For example if you want to load the doc-vqa dataset you can do so by:

from unitxt import load_dataset

dataset = load_dataset(
    card="cards.doc_vqa.en",
    template="templates.qa.with_context.title",
    format="formats.models.llava_interleave",
    loader_limit=20,
)

Since we have data augmentation mechanisms it is just natural to use it for images. For example if you want your images in grey scale:

dataset = load_dataset(
    card="cards.doc_vqa.en",
    template="templates.qa.with_context.title",
    format="formats.models.llava_interleave",
    loader_limit=20,
    augmentor="augmentors.image.grey_scale", # <= Just like the text augmenters!
)

Then if you want to get the scores of a model on this dataset you can use:

from unitxt.inference import HFLlavaInferenceEngine
from unitxt.text_utils import print_dict
from unitxt import evaluate

inference_model = HFLlavaInferenceEngine(
    model_name="llava-hf/llava-interleave-qwen-0.5b-hf", max_new_tokens=32
)

test_dataset = dataset["test"].select(range(5))

predictions = inference_model.infer(test_dataset)
evaluated_dataset = evaluate(predictions=predictions, data=test_dataset)

print_dict(
    evaluated_dataset[0],
    keys_to_print=["source", "media", "references", "processed_prediction", "score"],
)

Multi modality support in unitxt is building upon the type handling introduced in the previous section with two new types: Image and Audio.

What's Changed

New Contributors

Full Changelog: 1.12.4...1.13.0

Contributors

yifanmai, eladven, and 10 other contributors
Assets 2
Loading

1.12.4

28 Aug 13:17
@eladven eladven
1.12.4
1a97cce
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
1.12.4

Main changes

Additions to catalog

New Features

  • Add LLM as judge ensemble metrics, and add LLMaaJ ensemble example by @pvn25 in #1081
  • Refactor RenameFields operator to Rename. The old operator is still supported but raises a deprecation warning by @elronbandel in #1123

Bug Fixes

Documentation changes

  • Update llm_as_judge.py --- copy edits (grammar, consistency, clarity) by @welisheva22 in #1164
  • Update formats.py --- copy edits (grammar, consistency, clarity) by @welisheva22 in #1163
  • Update loaders.py --- copy edits (grammar, consistency, clarity) by @welisheva22 in #1162
  • Update card.py - minor documentation changes by @welisheva22 in #1161
  • Update adding_dataset.rst - a few more minor documentation changes by @welisheva22 in #1160
  • Update artifact.py --- documentation edits (grammar, consistency, cla… by @welisheva22 in #1159
  • Update glossary.rst --- copy edits (grammar, consistency, clarity) by @welisheva22 in #1155
  • Update helm.rst --- copy edits (grammar, consistency, clarity) by @welisheva22 in #1154
  • Update operators.py --- copy edits (grammar, consistency, clarity) - take 2 by @welisheva22 in #1158
  • Docfix: Fix typo in Installation doc by @yifanmai in #1181

New Contributors

Contributors

yifanmai, csrajmohan, and 6 other contributors
Assets 2
Loading

1.12.3

15 Aug 23:01
@eladven eladven
1.12.3
8fd91be
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
1.12.3

Main changes

Non backward compatible changes in catalog

  • change rag metrics name convention (e.g. "metrics.rag.mrr" -> "metrics.rag.context_correctness.mrr",) - catalog non backward compatible change by @assaftibm in #1104
  • Update summarization task and templates to support multiple reference summaries - by @yoavkatz in #1126
  • Fix belebele due to new convention by @elronbandel in #1145

Additions to catalog

  • Add DeepSeek-Coder format and system prompt by @oktie in #1105
  • Add a metric to calculate the ratio of references included in the prediction by @marukaz in #1091
  • adding RAG bge metrics by @assaftibm

New Features

  • Add option to run multiple templates and or num_demos in single dataset recipe. Now it is possible to give a list of templates or num_demos. Unitxt will randomly sample from the templates and for each instance assign a random template from the list. by @elronbandel in #1110
  • A warning is now generated when a metric generate a score with the same name as that of another metric and overwrites it @dafnapension in #1124
  • MetricPipeline fields postpreprocess_steps has been renamed to postprocess_steps. The old field (postpreprocess_steps) still exists for backward compatible but depricated. by @dafnapension in #1117
  • Decrease runtime of demo examples
  • Add tests for RAG metrics by @matanor
  • Adding dedicated Unitxt warning and error classes to link online documentation by @yoavkatz in
  • The code now uses a central controllable deepcopy function by @elronbandel in #1120

Bug Fixes

  • Create a dedicated nltk a mixin, for downloading all versions of punkt which needed by metrics code. by @elronbandel in #1151
  • For bulk instance metrics, Replace mean function with nanmean to support aggregation in case of nan scores. by @elronbandel in #1150
  • Fix helm test by @elronbandel in #1109
  • Fix bug with RAG metrics: Fix use of minilm model by @assaftibm in #1115
  • Fix data classification of WML model to include 'public' classification by @yoavkatz in #1118
  • Fix WMLInferenceEngine by @pawelknes in #1122
  • Fix belebele HF path due to new convention by @elronbandel in #1145

Documentation changes

New Contributors

Contributors

oktie, marukaz, and 9 other contributors
Assets 2
Loading