Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

restructure pytg for async #250

Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
faf6339
adding async functionality to pyTG
Dyllan-Hofflich-tg Aug 9, 2024
fb7d522
modify the async directory name and re-impot to test files
Aug 21, 2024
0ac2dff
resolve conflits
Aug 26, 2024
b96596e
format code
Aug 26, 2024
481fcf0
Merge branch 'master' into GML-1843-ASYNC-PYTG
luzhoutg Aug 26, 2024
06b6660
remove typo
Aug 27, 2024
01d4ef3
remove comments
Aug 27, 2024
f0f7a12
change the test server file to default
Aug 27, 2024
b2977f6
update tag to 1.7
Aug 27, 2024
f9f216a
remove temp folder, fix typo in testserver.json, remove version and l…
Aug 27, 2024
0d7752e
add async version to createSecret in dataset
Aug 28, 2024
b5c3169
comment out unnecessary codes for testing
Aug 28, 2024
f93dabe
removed duplicate code
Aug 28, 2024
4614d43
clean up codes related to tokens
Aug 30, 2024
145318e
clean up code for udf
Aug 30, 2024
0c219c8
clean up code for udf
Aug 30, 2024
9705c9c
restructure pytg for async
Sep 5, 2024
28e2f61
GML-1870 add init
Sep 6, 2024
bff13fa
change to zulip server
Sep 6, 2024
887bb55
GML-1870 refactor pyTG
Sep 10, 2024
949ef09
Update testserver.json
parkererickson-tg Sep 11, 2024
b1c1fb0
Update test_pyTigerGraphVertex.py
parkererickson-tg Sep 12, 2024
e160dcf
Update vertex.py
parkererickson-tg Sep 12, 2024
8fe443e
fixs
parkererickson-tg Sep 12, 2024
43a8c6c
update tests
parkererickson-tg Sep 12, 2024
4c7fd86
update setup
parkererickson-tg Sep 12, 2024
304b6d7
fix
parkererickson-tg Sep 12, 2024
8d97970
remove object inheritance async
parkererickson-tg Sep 12, 2024
8b2813f
updates
parkererickson-tg Sep 12, 2024
88628d9
fixes?
parkererickson-tg Sep 13, 2024
1a2dcb6
refactor pt 1
parkererickson-tg Sep 23, 2024
254f48b
change vertex funcs to use helper functions
parkererickson-tg Sep 23, 2024
fdd887c
remove class deps on schema helper functions
parkererickson-tg Sep 23, 2024
0497044
clean query util deps
parkererickson-tg Sep 23, 2024
15ee031
fix path util logging
parkererickson-tg Sep 23, 2024
2ee907d
fix loading util funcs
parkererickson-tg Sep 23, 2024
ea49948
wip: gsql progress
parkererickson-tg Sep 23, 2024
fd8511f
udf util funcs
parkererickson-tg Sep 23, 2024
9a2b4b6
fix edge util funcs
parkererickson-tg Sep 23, 2024
900c246
fix loading and path funcs
parkererickson-tg Sep 23, 2024
c0e79ef
fix dataset util funcs
parkererickson-tg Sep 23, 2024
3702985
cleanup
parkererickson-tg Sep 24, 2024
e9d0047
comment out async ogm tests for now
parkererickson-tg Sep 25, 2024
a9e78ac
test fixes
parkererickson-tg Sep 25, 2024
7b5f03e
fix parse get token
parkererickson-tg Sep 25, 2024
762692b
comment out task group test
parkererickson-tg Sep 25, 2024
0ae3256
fix import in jwt tests
parkererickson-tg Sep 25, 2024
1187536
fix import
parkererickson-tg Sep 25, 2024
4e15744
fix tests
parkererickson-tg Sep 25, 2024
d03dc26
fix auth
parkererickson-tg Sep 25, 2024
a0b9b3d
fix datasets list test
parkererickson-tg Sep 25, 2024
3c4830e
temp disable tests for refresh token
parkererickson-tg Sep 25, 2024
48062af
fix imports
parkererickson-tg Sep 25, 2024
0d6e707
fix test import path
parkererickson-tg Sep 25, 2024
bdfc13b
fix error check tests
parkererickson-tg Sep 25, 2024
b398470
fix error check tests 2
parkererickson-tg Sep 26, 2024
8cb5315
fix edge set to dataframe
parkererickson-tg Sep 26, 2024
1d975ed
fix vs2df
parkererickson-tg Sep 26, 2024
4f29efd
fix _get_edges_by_type
parkererickson-tg Sep 26, 2024
f45d488
see if disabling edge async makes sync tests pass
parkererickson-tg Sep 26, 2024
4f97e18
temp comment edge insert/delete tests
parkererickson-tg Sep 26, 2024
a491790
add parse get ver to core base class
parkererickson-tg Sep 26, 2024
f642480
rename async test classes, fix async base
parkererickson-tg Sep 26, 2024
9b750c1
fix version check in async getUDF
parkererickson-tg Sep 26, 2024
6071f2e
comment out edge tests failing
parkererickson-tg Sep 27, 2024
a860713
fix vertex upsert_attrs
parkererickson-tg Sep 30, 2024
1d868de
fix path params
parkererickson-tg Sep 30, 2024
30a14b4
fix async upsert edges
parkererickson-tg Sep 30, 2024
181cca9
remove unstable query installed test
parkererickson-tg Sep 30, 2024
e3e20dc
fix version check in async query
parkererickson-tg Sep 30, 2024
918f1fb
fix upsert_attrs test
parkererickson-tg Sep 30, 2024
dff6558
fix utils tests
parkererickson-tg Oct 1, 2024
37e7566
fix async util tests
parkererickson-tg Oct 1, 2024
941adf7
fix locals logging in vs2df
parkererickson-tg Oct 1, 2024
59a0f50
fix safe char in pyTG vertex
parkererickson-tg Oct 1, 2024
be67a72
await vertexSetToDataFrame in tests
parkererickson-tg Oct 1, 2024
76c4295
test async ogm
parkererickson-tg Oct 3, 2024
1512f27
remove async ogm
parkererickson-tg Oct 3, 2024
70dc538
remove refresh token test
parkererickson-tg Oct 3, 2024
f5f90a1
Merge branch 'master' into GML-1870-optimize-code-reusability-in-py-t…
parkererickson-tg Oct 3, 2024
930dbb5
Revert "remove refresh token test"
parkererickson-tg Oct 3, 2024
a5d7e11
change no token required
parkererickson-tg Oct 4, 2024
ab0df27
fix token config
parkererickson-tg Oct 4, 2024
f339c77
Merge branch 'master' into GML-1870-optimize-code-reusability-in-py-t…
parkererickson-tg Oct 23, 2024
1a61971
fix error handling auth
parkererickson-tg Oct 24, 2024
55d2291
fix async vertex tests
parkererickson-tg Oct 24, 2024
57914d2
fix vertex async tests
parkererickson-tg Oct 25, 2024
c15ff4b
fix getVertexCount async test
parkererickson-tg Oct 25, 2024
1980faa
fix jwt tests
parkererickson-tg Oct 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,6 @@ site/
pyTigerGraph.egg-info
tests1.py
.ipynb_checkpoints
pyrightconfig.json
venv
requirement.txt
pyrightconfig.json
1 change: 1 addition & 0 deletions pyTigerGraph/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
from pyTigerGraph.pyTigerGraph import TigerGraphConnection
from pyTigerGraph.pytgasync.pyTigerGraph import AsyncTigerGraphConnection

__version__ = "1.7"

Expand Down
82 changes: 44 additions & 38 deletions pyTigerGraph/ai/ai.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,14 @@

from pyTigerGraph import TigerGraphConnection


class AI:
def __init__(self, conn: TigerGraphConnection) -> None:
def __init__(self, conn: TigerGraphConnection) -> None:
"""NO DOC: Initiate an AI object. Currently in beta testing.
Args:
conn (TigerGraphConnection):
Accept a TigerGraphConnection to run queries with

Returns:
None
"""
Expand All @@ -47,7 +48,7 @@ def __init__(self, conn: TigerGraphConnection) -> None:
# split scheme and host
scheme, host = conn.host.split("://")
self.nlqs_host = scheme + "://copilot-" + host

def configureInquiryAIHost(self, hostname: str):
""" DEPRECATED: Configure the hostname of the InquiryAI service.
Not recommended to use. Use configureCoPilotHost() instead.
Expand All @@ -56,8 +57,8 @@ def configureInquiryAIHost(self, hostname: str):
The hostname (and port number) of the InquiryAI serivce.
"""
warnings.warn(
"The `configureInquiryAIHost()` function is deprecated; use `configureCoPilotHost()` function instead.",
DeprecationWarning)
"The `configureInquiryAIHost()` function is deprecated; use `configureCoPilotHost()` function instead.",
DeprecationWarning)
self.nlqs_host = hostname

def configureCoPilotHost(self, hostname: str):
Expand Down Expand Up @@ -96,14 +97,17 @@ def registerCustomQuery(self, query_name: str, description: str = None, docstrin
"queries": [query_name]
}
url = self.nlqs_host+"/"+self.conn.graphname+"/upsert_from_gsql"
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None)
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None)
else:
if description is None:
raise ValueError("When using TigerGraph 3.x, query descriptions are required parameters.")
raise ValueError(
"When using TigerGraph 3.x, query descriptions are required parameters.")
if docstring is None:
raise ValueError("When using TigerGraph 3.x, query docstrings are required parameters.")
raise ValueError(
"When using TigerGraph 3.x, query docstrings are required parameters.")
if param_types is None:
raise ValueError("When using TigerGraph 3.x, query parameter types are required parameters.")
raise ValueError(
"When using TigerGraph 3.x, query parameter types are required parameters.")
data = {
"function_header": query_name,
"description": description,
Expand All @@ -112,8 +116,8 @@ def registerCustomQuery(self, query_name: str, description: str = None, docstrin
"graphname": self.conn.graphname
}
url = self.nlqs_host+"/"+self.conn.graphname+"/register_docs"
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None)
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None)

def updateCustomQuery(self, query_name: str, description: str = None, docstring: str = None, param_types: dict = None):
""" Update a custom query with the InquiryAI service.
Args:
Expand Down Expand Up @@ -142,14 +146,17 @@ def updateCustomQuery(self, query_name: str, description: str = None, docstring:
"queries": [query_name]
}
url = self.nlqs_host+"/"+self.conn.graphname+"/upsert_from_gsql"
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None)
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None)
else:
if description is None:
raise ValueError("When using TigerGraph 3.x, query descriptions are required parameters.")
raise ValueError(
"When using TigerGraph 3.x, query descriptions are required parameters.")
if docstring is None:
raise ValueError("When using TigerGraph 3.x, query docstrings are required parameters.")
raise ValueError(
"When using TigerGraph 3.x, query docstrings are required parameters.")
if param_types is None:
raise ValueError("When using TigerGraph 3.x, query parameter types are required parameters.")
raise ValueError(
"When using TigerGraph 3.x, query parameter types are required parameters.")
data = {
"function_header": query_name,
"description": description,
Expand All @@ -160,8 +167,8 @@ def updateCustomQuery(self, query_name: str, description: str = None, docstring:

json_payload = {"id": "", "query_info": data}
url = self.nlqs_host+"/"+self.conn.graphname+"/upsert_docs"
return self.conn._req("POST", url, authMode="pwd", data = json_payload, jsonData=True, resKey=None)
return self.conn._req("POST", url, authMode="pwd", data=json_payload, jsonData=True, resKey=None)

def deleteCustomQuery(self, query_name: str):
""" Delete a custom query with the InquiryAI service.
Args:
Expand All @@ -172,9 +179,9 @@ def deleteCustomQuery(self, query_name: str):
"""
data = {"ids": [], "expr": "function_header == '"+query_name+"'"}
url = self.nlqs_host+"/"+self.conn.graphname+"/delete_docs"
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None)
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None)

def retrieveDocs(self, query:str, top_k:int = 3):
def retrieveDocs(self, query: str, top_k: int = 3):
""" Retrieve docs from the vector store.
Args:
query (str):
Expand All @@ -188,8 +195,9 @@ def retrieveDocs(self, query:str, top_k:int = 3):
"query": query
}

url = self.nlqs_host+"/"+self.conn.graphname+"/retrieve_docs?top_k="+str(top_k)
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None, skipCheck=True)
url = self.nlqs_host+"/"+self.conn.graphname + \
"/retrieve_docs?top_k="+str(top_k)
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None, skipCheck=True)

def query(self, query):
""" Query the database with natural language.
Expand All @@ -204,24 +212,24 @@ def query(self, query):
}

url = self.nlqs_host+"/"+self.conn.graphname+"/query"
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None)
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None)

def coPilotHealth(self):
""" Check the health of the CoPilot service.
Returns:
JSON response from the CoPilot service.
"""
url = self.nlqs_host+"/health"
return self.conn._req("GET", url, authMode="pwd", resKey=None)

def initializeSupportAI(self):
""" Initialize the SupportAI service.
Returns:
JSON response from the SupportAI service.
"""
url = self.nlqs_host+"/"+self.conn.graphname+"/supportai/initialize"
return self.conn._req("POST", url, authMode="pwd", resKey=None)

def createDocumentIngest(self, data_source, data_source_config, loader_config, file_format):
""" Create a document ingest.
Args:
Expand All @@ -244,8 +252,8 @@ def createDocumentIngest(self, data_source, data_source_config, loader_config, f
}

url = self.nlqs_host+"/"+self.conn.graphname+"/supportai/create_ingest"
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None)
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None)

def runDocumentIngest(self, load_job_id, data_source_id, data_path):
""" Run a document ingest.
Args:
Expand All @@ -264,9 +272,9 @@ def runDocumentIngest(self, load_job_id, data_source_id, data_path):
"file_path": data_path
}
url = self.nlqs_host+"/"+self.conn.graphname+"/supportai/ingest"
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None)
def searchDocuments(self, query, method = "hnswoverlap", method_parameters: dict = {"indices": ["Document", "DocumentChunk", "Entity", "Relationship"], "top_k": 2, "num_hops": 2, "num_seen_min": 2}):
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None)

def searchDocuments(self, query, method="hnswoverlap", method_parameters: dict = {"indices": ["Document", "DocumentChunk", "Entity", "Relationship"], "top_k": 2, "num_hops": 2, "num_seen_min": 2}):
""" Search documents.
Args:
query (str):
Expand All @@ -284,9 +292,9 @@ def searchDocuments(self, query, method = "hnswoverlap", method_parameters: dict
"method_params": method_parameters
}
url = self.nlqs_host+"/"+self.conn.graphname+"/supportai/search"
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None)
def answerQuestion(self, query, method = "hnswoverlap", method_parameters: dict = {"indices": ["Document", "DocumentChunk", "Entity", "Relationship"], "top_k": 2, "num_hops": 2, "num_seen_min": 2}):
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None)

def answerQuestion(self, query, method="hnswoverlap", method_parameters: dict = {"indices": ["Document", "DocumentChunk", "Entity", "Relationship"], "top_k": 2, "num_hops": 2, "num_seen_min": 2}):
""" Answer a question.
Args:
query (str):
Expand All @@ -304,8 +312,8 @@ def answerQuestion(self, query, method = "hnswoverlap", method_parameters: dict
"method_params": method_parameters
}
url = self.nlqs_host+"/"+self.conn.graphname+"/supportai/answerquestion"
return self.conn._req("POST", url, authMode="pwd", data = data, jsonData=True, resKey=None)
return self.conn._req("POST", url, authMode="pwd", data=data, jsonData=True, resKey=None)

def forceConsistencyUpdate(self, method="supportai"):
""" Force a consistency update for SupportAI embeddings.
Args:
Expand All @@ -317,8 +325,7 @@ def forceConsistencyUpdate(self, method="supportai"):
"""
url = f"{self.nlqs_host}/{self.conn.graphname}/{method}/forceupdate/"
return self.conn._req("GET", url, authMode="pwd", resKey=None)

''' TODO: Add support in CoPilot

def checkConsistencyProgress(self, method="supportai"):
""" Check the progress of the consistency update.
Args:
Expand All @@ -330,4 +337,3 @@ def checkConsistencyProgress(self, method="supportai"):
"""
url = f"{self.nlqs_host}/{self.conn.graphname}/supportai/consistency_status/{method}"
return self.conn._req("GET", url, authMode="pwd", resKey=None)
'''
Loading