Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Users/amrsing/changes for browser auth fallback #1201

Draft
wants to merge 135 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
91370aa
Index, Query by context + SA integration + Placeholder + Context Swit…
Aug 19, 2024
649edcd
Minor changes + Context Swtich flow not working
dabbcomputers Aug 20, 2024
4875079
Writing edges and vertices into graphdb
Aug 20, 2024
a0f9a54
Fixed context switching issue
dabbcomputers Aug 20, 2024
5f80f66
addressed comment
dabbcomputers Aug 20, 2024
b231058
Merge pull request #2 from prateejain-linked/users/amrsing/POCChanges2
amritpalms Aug 21, 2024
d38a026
Adding azure-kusto-data as a dependency
logomachic Aug 7, 2024
a4efadb
Initial add of Kusto related file changes.
logomachic Aug 7, 2024
da4ca97
Drop & Remove db query split due to syntax error
logomachic Aug 7, 2024
c408799
Adding kusto documentation
logomachic Aug 7, 2024
303e9c4
I cleaned up the code I was working on and added TODOs to blocks of f…
logomachic Aug 9, 2024
027988a
Merge pull request #3 from prateejain-linked/KUSTO-1
sirus-ms Aug 21, 2024
16295e5
Add Reading from graphdb
Aug 21, 2024
01fb617
Some modifications to kusto flow.
sirus-ms Aug 21, 2024
c63718f
Merge pull request #4 from prateejain-linked/sirusbr
sirus-ms Aug 21, 2024
f78bd40
Add a skleton for launch.json
sirus-ms Aug 21, 2024
3fd7ada
Merge pull request #5 from prateejain-linked/sirusbr
sirus-ms Aug 21, 2024
be00a93
Fix a typo
sirus-ms Aug 21, 2024
73306be
Merge pull request #6 from prateejain-linked/sirusbr
sirus-ms Aug 21, 2024
0918186
Creating entities table in Kusto.
logomachic Aug 21, 2024
f40f101
Merge pull request #7 from prateejain-linked/KUSTO-2
sirus-ms Aug 21, 2024
a6b0689
Minor fixes
dabbcomputers Aug 21, 2024
6837a01
Merge pull request #8 from prateejain-linked/users/amrsing/MinorFixes…
sirus-ms Aug 21, 2024
116f524
minor updates: lancedb style entities
sirus-ms Aug 22, 2024
8dc4181
Merge pull request #9 from prateejain-linked/sirusbr
sirus-ms Aug 22, 2024
83798a4
syntax
sirus-ms Aug 22, 2024
2f40f69
Merge pull request #10 from prateejain-linked/sirusbr
sirus-ms Aug 22, 2024
02b3a95
Writing edges and vertices into graphdb
Aug 20, 2024
d79fa7b
Add Reading from graphdb
Aug 21, 2024
3d2be1c
Integrating with latest PR
Aug 21, 2024
16c4759
Correct file path in text files
Aug 22, 2024
d6e2645
Resolve merge conflicts
Aug 22, 2024
2c9edd7
Resolve final merge conflict
Aug 22, 2024
3fc8f8d
Merge pull request #11 from prateejain-linked/users/gbarrnsnchez/grap…
sirus-ms Aug 22, 2024
b9fbb6a
merge final_entities and final_nodes
sirus-ms Aug 22, 2024
95f8a20
Merge pull request #12 from prateejain-linked/sirusbr
sirus-ms Aug 22, 2024
621ea11
Add config parameters for graphdb
Aug 22, 2024
cae28b3
Add default values in graphrag/index/init_content.py
Aug 23, 2024
a151172
Merge pull request #13 from prateejain-linked/users/gbarrnsnchez/add_…
gbarroutlook Aug 23, 2024
d5a8e7c
Moving Pipeline Storage to Common + Export query artifacts
dabbcomputers Aug 23, 2024
8e20637
Add missing config file
Aug 23, 2024
b4d3817
Merge pull request #15 from prateejain-linked/users/gbarrnsnchez/add_…
gbarroutlook Aug 23, 2024
1f95114
Merge branch 'main' of https://github.com/prateejain-linked/graphrag …
dabbcomputers Aug 23, 2024
c46d71c
Small fixes to incline with GraphDBClient
dabbcomputers Aug 23, 2024
b9dff03
saving changes
dabbcomputers Aug 24, 2024
1088da6
Fixing minor issue in main branch
dabbcomputers Aug 24, 2024
50fa5a9
Merge pull request #16 from prateejain-linked/users/amrsing/BringingM…
amritpalms Aug 24, 2024
079c034
Only using Kusto store for all entities.
logomachic Aug 26, 2024
5d9f27a
Merge pull request #17 from prateejain-linked/KUSTO-3
sirus-ms Aug 26, 2024
f6a8f63
Merge branch 'main' of https://github.com/prateejain-linked/graphrag …
dabbcomputers Aug 26, 2024
c2f14b6
Merge branch 'users/amrsing/ExportQueryOutputArtifacts' of https://gi…
dabbcomputers Aug 26, 2024
28594c6
Include context into read and write calls for graphdb
Aug 26, 2024
aefa924
Using create_final_entities table rather than entity_description_embe…
logomachic Aug 26, 2024
8a1d17c
Optimized Search
dabbcomputers Aug 27, 2024
3e484e0
Merge pull request #18 from prateejain-linked/KUSTO-4
sirus-ms Aug 27, 2024
1c21376
Include context into read and write calls for graphdb
Aug 26, 2024
ca0dcc9
Changing similarity search (query to entity embedding search) to use …
logomachic Aug 27, 2024
a87071d
Working changes
dabbcomputers Aug 27, 2024
e51f325
Merge pull request #14 from prateejain-linked/users/amrsing/ExportQue…
amritpalms Aug 27, 2024
c0d628c
Merge pull request #19 from prateejain-linked/COSINE-VECTOR16
sirus-ms Aug 28, 2024
8bc2c71
Kusto minor edits
sirus-ms Aug 28, 2024
13eb167
Merge pull request #20 from prateejain-linked/sirusbr
logomachic Aug 28, 2024
1927e99
Merging Kusto local search into local search
logomachic Aug 28, 2024
38449c3
Merge pull request #21 from prateejain-linked/MERGE-KUSTO
sirus-ms Aug 28, 2024
0039b0d
Fixing lancedb from Sirus suggestion
logomachic Aug 28, 2024
cadd7e3
Merge pull request #23 from prateejain-linked/MERGE-KUSTO
sirus-ms Aug 28, 2024
16a5766
Add functionality for context graph creation
Aug 29, 2024
fc79c32
Solve merge conflicts
Aug 29, 2024
b1d4f22
Kusto context-switch
sirus-ms Aug 30, 2024
05e721c
Include context into read and write calls for graphdb
Aug 26, 2024
689b983
Add functionality for context graph creation
Aug 29, 2024
628a4c2
Address comments
Aug 30, 2024
b7749b1
Fix conflicts
Aug 30, 2024
0e5a4b9
Merge pull request #24 from prateejain-linked/users/gbarrnsnchez/add_…
gbarroutlook Aug 30, 2024
fd76db5
Adding community reports to Kusto
logomachic Aug 30, 2024
aa6095b
Adding configurations use_kusto_community_reports, updating - to _ co…
logomachic Aug 30, 2024
98ff552
Merge pull request #26 from prateejain-linked/COMMUNITY_REPORTS
sirus-ms Sep 3, 2024
3efb530
Seperate out setup so that context switcher can call setup & load sep…
logomachic Sep 3, 2024
4eb4ede
Merge pull request #27 from prateejain-linked/MULTI_QUERY
sirus-ms Sep 3, 2024
c80510a
Fixed bug where defaults for vector store weren't being set for index…
logomachic Sep 3, 2024
67e1033
Setup for vector store happens once per activation instead of for eve…
logomachic Sep 3, 2024
82a2034
Merge pull request #29 from prateejain-linked/MULTI_QUERY
sirus-ms Sep 3, 2024
c353d68
Arg mismatch & not calling load kusto in query anymore.
logomachic Sep 3, 2024
3ecccdf
Merge pull request #30 from prateejain-linked/MULTI_QUERY
sirus-ms Sep 3, 2024
dae1bee
Adding report_name var to query, load_ doesn't overwrite automaticall…
logomachic Sep 4, 2024
eb6cbe4
Get rid of concat in context_switcher so each file gets uploaded sepe…
logomachic Sep 4, 2024
560bbe1
Configuring in_memory embedding storage even with vector_store config…
logomachic Sep 4, 2024
ed14b6a
Merge pull request #32 from prateejain-linked/MULTI_QUERY
sirus-ms Sep 4, 2024
fa42510
Change entity ID generation
sirus-ms Sep 4, 2024
71263a4
Add graphdb calls directly where relationships are filtered (#31)
gbarroutlook Sep 4, 2024
376578e
Merge pull request #33 from prateejain-linked/fix_ids
sirus-ms Sep 5, 2024
c333009
Adding graphdb into for-loop per data_path of context b/c it should
logomachic Sep 5, 2024
1b66bd3
logs on file & stdout + unbuffered logs
Sep 5, 2024
5521da9
Merge pull request #35 from prateejain-linked/users/amrsing/LogsOnFil…
amritpalms Sep 5, 2024
3604ca2
Merge pull request #34 from prateejain-linked/MINOR_CHANGE
sirus-ms Sep 6, 2024
44374d9
Fix cli when graphdb is not enabled. (#36)
sirus-ms Sep 6, 2024
72a866b
Add graphdb parameters for local emulator support
Sep 6, 2024
2baa18d
incline to run in azure
amritpalms Sep 8, 2024
0e1df62
commenting managed identity code
amritpalms Sep 8, 2024
a92787e
minor fix
amritpalms Sep 8, 2024
4bc605c
Merge pull request #39 from prateejain-linked/users/amrsing/RagFixesF…
sirus-ms Sep 9, 2024
beb0285
Merge pull request #37 from prateejain-linked/users/gbarrnsnchez/add_…
sirus-ms Sep 9, 2024
6a13d55
fix kusto cli
sirus-ms Sep 9, 2024
dc4a1fb
Merge pull request #40 from prateejain-linked/query_cli_kusto2
gbarroutlook Sep 9, 2024
25aad8c
fix legacy
sirus-ms Sep 10, 2024
51c7720
Implement deactivation switch
sirus-ms Sep 10, 2024
8b79c00
Merge pull request #41 from prateejain-linked/query_cli_kusto3
sirus-ms Sep 10, 2024
4d35524
Merge pull request #42 from prateejain-linked/ctx_switch_deact
sirus-ms Sep 10, 2024
544f4b7
Add deactivation switch for graphdb
sirus-ms Sep 10, 2024
52ec890
Merge pull request #43 from prateejain-linked/ctx_switch_deact_gdb
sirus-ms Sep 10, 2024
a50c65c
Allowing multiple files to be indexed.
logomachic Sep 12, 2024
7232a68
Initial code for the different query paths.
logomachic Sep 16, 2024
5afc137
Merge pull request #44 from prateejain-linked/MULTI_FILE
sirus-ms Sep 16, 2024
fd8ac3f
Merge pull request #45 from prateejain-linked/PATHS
sirus-ms Sep 16, 2024
38254ef
Add missing args
sirus-ms Sep 16, 2024
b5cbe43
Update __main__.py paths type
logomachic Sep 17, 2024
62a3495
Add text units to kusto
sirus-ms Sep 17, 2024
9a03893
Text units 2
sirus-ms Sep 17, 2024
0f8c7c4
Merge pull request #47 from prateejain-linked/txt_units
sirus-ms Sep 17, 2024
951e847
minor fix
sirus-ms Sep 17, 2024
175b075
Minor fix
sirus-ms Sep 18, 2024
58d30de
Graphrag using Azure OpenAI uses Managed Identity when no API_KEY pre…
logomachic Sep 19, 2024
1f6a49c
Query & Embedding Manged Identities changes.
logomachic Sep 19, 2024
43806a2
Added the func app compatible code
Sep 23, 2024
7a87843
Added one req for windows local debug
Sep 23, 2024
3c1781a
Removing the redundant settings.yaml
Sep 23, 2024
05ee3b9
Added the func app compatible code (#50)
prateejain-linked Sep 23, 2024
2f32844
added the local settings file forcefully
Sep 23, 2024
1db67f9
Merge branch 'main' of https://github.com/prateejain-linked/graphrag …
Sep 23, 2024
cdad208
Update local.settings.json
logomachic Sep 23, 2024
d8cb19f
Merge pull request #51 from prateejain-linked/user/prateejain/graphra…
sirus-ms Sep 23, 2024
08d80b8
config for debugger
amritpalms Sep 24, 2024
569a5a2
Merge pull request #53 from prateejain-linked/users/amrsing/WorkingDe…
amritpalms Sep 24, 2024
984fa0d
DefaultAuthCredes for llm
amritpalms Sep 24, 2024
81f0bca
fix for browser not opening
amritpalms Sep 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Creating entities table in Kusto.
logomachic committed Aug 21, 2024
commit 09181864290c694c05fe23604533a848de12afe5
10 changes: 8 additions & 2 deletions graphrag/query/cli.py
Original file line number Diff line number Diff line change
@@ -276,7 +276,7 @@ def run_content_store_local_search(
description_embedding_store.load_parqs(data_dir, ["create_final_nodes", "create_final_community_reports", "create_final_text_units", "create_final_relationships", "create_final_entities"])

#TODO KQLify this. This merge of nodes & entities needs to happen in Kusto.
# entities = read_indexer_entities(final_nodes, final_entities, community_level)
create_entities_table(description_embedding_store, community_level)
# description_embedding_store = __get_embedding_description_store(
# entities=entities,
# description_embedding_store=description_embedding_store,
@@ -313,7 +313,13 @@ def run_content_store_local_search(

return True #Obviously this is a placeholder due to all the TODOs above.


# Create entities table similar to read_indexer_entities, but creating that table in Kusto, not in memory.
def create_entities_table(description_embedding_store: BaseVectorStore, community_level: int):
description_embedding_store.execute_query(f".set-or-replace entities <| create_final_nodes \
| where level <= {community_level} \
| project community=coalesce(community, 0), name=['title'], rank=degree \
| summarize community=max(community) by name, rank \
| join kind=inner create_final_entities on name")

def run_content_store_global_search(
config_dir: str | None,
6 changes: 4 additions & 2 deletions graphrag/vector_stores/azure_ai_search.py
Original file line number Diff line number Diff line change
@@ -194,5 +194,7 @@ def similarity_search_by_text(
return []

def load_parqs(self, data_path, parq_names) -> Any:
"""Load documents (Parquet files) into the vector-store."""
raise NotImplementedError("Loading Parquet files is not supported for Azure AI Search")
raise NotImplementedError("Loading Parquet files is not supported for Azure AI Search")

def execute_query(self, query: str) -> Any:
return super().execute_query(query)
5 changes: 5 additions & 0 deletions graphrag/vector_stores/base.py
Original file line number Diff line number Diff line change
@@ -83,3 +83,8 @@ def filter_by_id(self, include_ids: list[str] | list[int]) -> Any:
@abstractmethod
def load_parqs(self, data_path: str, parqs: list[str]) -> Any:
"""Load documents (Parquet files) into the vector-store."""

#TODO This is temporary until I take out the client from the vector store class
@abstractmethod
def execute_query(self, query: str) -> Any:
"""Execute a query in the vector-store."""
2 changes: 2 additions & 0 deletions graphrag/vector_stores/kusto.py
Original file line number Diff line number Diff line change
@@ -178,6 +178,8 @@ def similarity_search_by_text(
return self.similarity_search_by_vector(query_embedding, k)
return []

def execute_query(self, query: str) -> Any:
self.client.execute(self.database, f"{query}")

def load_parqs(self, data_dir, parq_names) -> Any:
data_path = Path(data_dir)
5 changes: 4 additions & 1 deletion graphrag/vector_stores/lancedb.py
Original file line number Diff line number Diff line change
@@ -121,4 +121,7 @@ def similarity_search_by_text(
return []

def load_parqs(self, data_path, parq_names) -> Any:
raise NotImplementedError("Loading Parquet files is not supported for LanceDB")
raise NotImplementedError("Loading Parquet files is not supported for LanceDB")

def execute_query(self, query: str) -> Any:
return super().execute_query(query)