Remove datashaper strip code (#1574)

* Remove most old pipeline running code and update tests * Restore update functionality * Remove most pipeline config references * Move workflows up * Simplify base_text_units naming * Simplify verb test setup * Simplify table read/write * Remove some pipeline_config refs * Remove misc unused utils/functions * Semver * Update blog posts (#1571) * Semver * Fix filename typo * Remove runtime_storage and snapshot * Remove unused exporter * Improve workflow_name reuse * Move derive_from_rows from DataShaper * Migrate callbacks/progress from DataShaper * Remove unused profiling * Remove DataShaper agg helper in create_base_text_units * Remove VerbParallelizationError * Remove datashaper dep and docs references * Chore/gleanings any encoding (#1569) * Make claims and entities independent of encoding * Semver * Change semver release type * Basic search implementation (#1563) * basic search implementation * basic streaming functionality * format check * check fix * release change * Chore/gleanings any encoding (#1569) * Make claims and entities independent of encoding * Semver * Change semver release type --------- Co-authored-by: Alonso Guevara <[email protected]> * Remove config input models (#1570) * Remove config input models * remove unit tests related to config input models * add semversioner change * Merge branch 'main' into config-remove-input-models * Bump ruff from 0.8.4 to 0.8.5 (#1579) * Bump ruff from 0.8.4 to 0.8.5 Bumps [ruff](https://github.com/astral-sh/ruff) from 0.8.4 to 0.8.5. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.8.4...0.8.5) --- updated-dependencies: - dependency-name: ruff dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Fix ruff * Semver * Another ruff --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alonso Guevara <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Alonso Guevara <[email protected]> Co-authored-by: gaudyb <[email protected]> Co-authored-by: Derek Worthen <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
microsoft · Jan 3, 2025 · 3f27d3d · 3f27d3d
1 parent 5b978e7
commit 3f27d3d
Show file tree

Hide file tree

Showing 151 changed files with 1,883 additions and 4,593 deletions.
diff --git a/.semversioner/next-release/minor-20241231213627966329.json b/.semversioner/next-release/minor-20241231213627966329.json
@@ -0,0 +1,4 @@
+{
+  "type": "minor",
+  "description": "Remove old pipeline runner."
+}
diff --git a/dictionary.txt b/dictionary.txt
@@ -148,10 +148,6 @@ codebases
 # Microsoft
 MSRC
 
-# Broken Upstream
-# TODO FIX IN DATASHAPER
-Arrary
-
 # Prompt Inputs
 ABILA
 Abila

diff --git a/docs/examples_notebooks/index_migration.ipynb b/docs/examples_notebooks/index_migration.ipynb
@@ -206,9 +206,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from datashaper import NoopVerbCallbacks\n",
-    "\n",
     "from graphrag.cache.factory import create_cache\n",
+    "from graphrag.callbacks.noop_verb_callbacks import NoopVerbCallbacks\n",
     "from graphrag.index.flows.generate_text_embeddings import generate_text_embeddings\n",
     "\n",
     "# We only need to re-run the embeddings workflow, to ensure that embeddings for all required search fields are in place\n",

diff --git a/docs/index/architecture.md b/docs/index/architecture.md
@@ -8,33 +8,9 @@ In order to support the GraphRAG system, the outputs of the indexing engine (in
 This model is designed to be an abstraction over the underlying data storage technology, and to provide a common interface for the GraphRAG system to interact with.
 In normal use-cases the outputs of the GraphRAG Indexer would be loaded into a database system, and the GraphRAG's Query Engine would interact with the database using the knowledge model data-store types.
 
-### DataShaper Workflows
-
-GraphRAG's Indexing Pipeline is built on top of our open-source library, [DataShaper](https://github.com/microsoft/datashaper).
-DataShaper is a data processing library that allows users to declaratively express data pipelines, schemas, and related assets using well-defined schemas.
-DataShaper has implementations in JavaScript and Python, and is designed to be extensible to other languages.
-
-One of the core resource types within DataShaper is a [Workflow](https://github.com/microsoft/datashaper/blob/main/javascript/schema/src/workflow/WorkflowSchema.ts).
-Workflows are expressed as sequences of steps, which we call [verbs](https://github.com/microsoft/datashaper/blob/main/javascript/schema/src/workflow/verbs.ts).
-Each step has a verb name and a configuration object.
-In DataShaper, these verbs model relational concepts such as SELECT, DROP, JOIN, etc.. Each verb transforms an input data table, and that table is passed down the pipeline.
-
-```mermaid
----
-title: Sample Workflow
----
-flowchart LR
-    input[Input Table] --> select[SELECT] --> join[JOIN] --> binarize[BINARIZE] --> output[Output Table]
-```
-
-### LLM-based Workflow Steps
-
-GraphRAG's Indexing Pipeline implements a handful of custom verbs on top of the standard, relational verbs that our DataShaper library provides. These verbs give us the ability to augment text documents with rich, structured data using the power of LLMs such as GPT-4. We utilize these verbs in our standard workflow to extract entities, relationships, claims, community structures, and community reports and summaries. This behavior is customizable and can be extended to support many kinds of AI-based data enrichment and extraction tasks.
-
-### Workflow Graphs
+### Workflows
 
 Because of the complexity of our data indexing tasks, we needed to be able to express our data pipeline as series of multiple, interdependent workflows.
-In the GraphRAG Indexing Pipeline, each workflow may define dependencies on other workflows, effectively forming a directed acyclic graph (DAG) of workflows, which is then used to schedule processing.
 
 ```mermaid
 ---
@@ -55,7 +31,7 @@ stateDiagram-v2
 The primary unit of communication between workflows, and between workflow steps is an instance of `pandas.DataFrame`.
 Although side-effects are possible, our goal is to be _data-centric_ and _table-centric_ in our approach to data processing.
 This allows us to easily reason about our data, and to leverage the power of dataframe-based ecosystems.
-Our underlying dataframe technology may change over time, but our primary goal is to support the DataShaper workflow schema while retaining single-machine ease of use and developer ergonomics.
+Our underlying dataframe technology may change over time, but our primary goal is to support the workflow schema while retaining single-machine ease of use and developer ergonomics.
 
 ### LLM Caching
 

diff --git a/examples/README.md b/examples/README.md
diff --git a/examples/__init__.py b/examples/__init__.py
diff --git a/examples/custom_input/__init__.py b/examples/custom_input/__init__.py
diff --git a/examples/custom_input/pipeline.yml b/examples/custom_input/pipeline.yml
diff --git a/examples/custom_input/run.py b/examples/custom_input/run.py
diff --git a/examples/single_verb/__init__.py b/examples/single_verb/__init__.py
diff --git a/examples/single_verb/input/data.csv b/examples/single_verb/input/data.csv
diff --git a/examples/single_verb/pipeline.yml b/examples/single_verb/pipeline.yml
diff --git a/examples/single_verb/run.py b/examples/single_verb/run.py
diff --git a/examples/use_built_in_workflows/__init__.py b/examples/use_built_in_workflows/__init__.py
diff --git a/examples/use_built_in_workflows/pipeline.yml b/examples/use_built_in_workflows/pipeline.yml