Typing and XML removal (#414)

* Runner: dataclass => pydantic BaseModel

* guard: Add overload typedefs for __call__ and parse

* Correctly type guard.py and references

* run: Fully typed

* guard: Fix parse type overload

* cli: one typehint fix

* guard: Raise RuntimeError if num_reasks is None after calling configure

* run: Reconsolidate rebased parse call

* run: Add type expecting reasks

* Remove <pydantic> plumbing

* rail: Type

* guard: Propagate typehints

* base_prompt: Correctly return None on no match

* Type schema, validators, datatypes

* format

* rail: Default input schema to None

* Allow metadata to be None instead of default dict

* Add coro checks to SaliencyCheck validator

* Refactor MockValidator to create_mock_validator

This now creates a type dynamically instead of just an instance

* datatypes: Import Self from typing_extensions instead of typing (python3.11+)

* Add typing-extensions dependency

This is already included by some other dependency, but we should also explicitly require it

* Remove PydanticReask (old pydantic plumbing)

* Fix mocked openai embeddings response

* analytics

* embedding: Typing

* document_store: Typing

* vectordb: Typing

* docs_utils: Typing

* json_utils: Typing

* logs_utils: Typing

* reask_utils: Typing

* pydantic_utils: Typing

* format

* sql_utils: Typing

* parsing_utils: Typing

* utils/misc: Typing

* pydantic_utils: Fixup

* add missing imports back in

* fix tests

* lint fixes

* ignore tests when calculating coverage

* match js pattern to mkdocs docs

* add GA

* directly add postgot and pipedrive keys

* GA stream directly in yml

* Update canonical uri

* Formatter.parse -> Template.get_identifiers

* polyfill get_identifiers for older python versions

* test_get_template_variables

* autoformat

* swap spy

* simplify test

* correct minimum version

* remove unused impot

* use safe_substitute to guard against monetary format i.e. $5

* ignore pyenv version

* test prompt.escape

* lint fix

* ignore lint error on line

* schema.FormatAttr: Remove internal XML dependence

* datatypes: Store optional as param in datatype

* text2sql: Typing

* Makefile: Add type target

* guard: Type return as Any

* llm_providers: Typing

* JsonSchema.parse: Small typing

* validators.provenancev0: Rewrite to satisfy typing

* ci: Add pyright job

* setup: Add pyright to dev dependencies

* Ignore handled missing imports

* pydantic_utils: Return instead of pass (bug?)

* setup: Require dev dependency lxml-stubs (for typing)

* validators.ProvenanceV1: Rewrite for typing

* Fix xml attr string casts

* json_utils: Take datatype dict instead of xml element

* datatypes: Abstract out name and description

* datatypes: Verify metadata over datatype

* Split validators into separate package

* schema/reask_utils: Migrate from internal XML dependence

* tests: Fix

* Rewrite pruned tree test

* format

* Refactor removeprefix for python 3.8

* Abstract out xml element string cast

* add a few UTs

* test_async: Test StringSchema.async_validate

* test_xml_utils: Add test for test_xml_element_cast

* test manifest

* schema: Remove unused func

* add pydantic chat integration test

* test_llm_providers: Add openai chat tests with basemodel

* Fix some type errors

* datatypes: set_children => set_children_from_xml

* datatypes: Remove iter methods

* datatypes: Remove xml element property

* schema: Simplify from_xml construction

* test_reask_utils: Amend test with correct validator example

* lint

* circ references, semi-passing tests

* correct string default typing

* date_format fix

* run.call: correctly pass output as llm_response

* guard.parse: Pop kwargs off api initialization (as it used to do)

* mock_llm_outputs: Add tomato cheese pizza example

* test_run: formatting

* fix regex validator

* fix validator test type sussing

* lint

* unused import removed

* typing 50% fixed

* text completion

* ignore pyright warning on np

* lint

* fix numpy typing issue

* Check for match type on init

* merge latest from public main

* ignore tpying on optional packages

* fix import path and typing on detect_secrets package

---------

Co-authored-by: Rafael Irgolic <[email protected]>
Co-authored-by: rafael <[email protected]>
Co-authored-by: Caleb Courier <[email protected]>

Author: 3 people

.github/workflows/ci.yml

@@ -42,6 +42,33 @@ jobs:
                 run: |
                     make lint
 
+    Typing:
+        runs-on: ubuntu-latest
+        strategy:
+            matrix:
+                python-version: ['3.8', '3.9', '3.10', '3.11']
+
+        steps:
+            -   uses: actions/checkout@v2
+            -   name: Set up Python ${{ matrix.python-version }}
+                uses: actions/setup-python@v2
+                with:
+                    python-version: ${{ matrix.python-version }}
+
+            -   uses: actions/cache@v2
+                with:
+                    path: ~/.cache/pip
+                    key: ${{ runner.os }}-pip
+
+            -   name: Install Dependencies
+                run: |
+                    python -m pip install --upgrade pip
+                    make dev
+
+            -   name: Static analysis with pyright
+                run: |
+                    make type
+
     Pytests:
         runs-on: ubuntu-latest
         strategy:

.gitignore

@@ -23,3 +23,4 @@ scratch/
 test.db
 test.index
 htmlcov
+.python-version

Makefile

@@ -5,6 +5,9 @@ autoformat:
 	isort --atomic guardrails/ tests/
 	docformatter --in-place --recursive guardrails tests
 
+type:
+	pyright guardrails/
+
 lint:
 	isort -c guardrails/ tests/
 	black guardrails/ tests/ --check
@@ -36,4 +39,9 @@ dev:
 full:
 	pip install -e ".[all]"
 
-all: autoformat lint docs test
+all: autoformat type lint docs test
+
+precommit:
+	# pytest -x -q --no-summary
+	pyright guardrails/
+	make lint

codecov.yml

@@ -1,2 +1,3 @@
 ignore:
-  - "guardrails/version.py"
+  - "guardrails/version.py"
+  - "tests"

docs/javascripts/pipedrive.js

@@ -1,7 +1,8 @@
+import asyncio
 import json
 import os
 from string import Template
-from typing import Callable, Dict, Optional
+from typing import Callable, Dict, Optional, Type
 
 import openai
 
@@ -63,14 +64,14 @@ def __init__(
         conn_str: str,
         schema_file: Optional[str] = None,
         examples: Optional[Dict] = None,
-        embedding: Optional[EmbeddingBase] = OpenAIEmbedding,
-        vector_db: Optional[VectorDBBase] = Faiss,
-        document_store: Optional[DocumentStoreBase] = EphemeralDocumentStore,
+        embedding: Type[EmbeddingBase] = OpenAIEmbedding,
+        vector_db: Type[VectorDBBase] = Faiss,
+        document_store: Type[DocumentStoreBase] = EphemeralDocumentStore,
         rail_spec: Optional[str] = None,
         rail_params: Optional[Dict] = None,
-        example_formatter: Optional[Callable] = example_formatter,
-        reask_prompt: Optional[str] = REASK_PROMPT,
-        llm_api: Optional[Callable] = openai.Completion.create,
+        example_formatter: Callable = example_formatter,
+        reask_prompt: str = REASK_PROMPT,
+        llm_api: Callable = openai.Completion.create,
         llm_api_kwargs: Optional[Dict] = None,
         num_relevant_examples: int = 2,
     ):
@@ -119,7 +120,7 @@ def _init_guard(
         schema_file: Optional[str] = None,
         rail_spec: Optional[str] = None,
         rail_params: Optional[Dict] = None,
-        reask_prompt: Optional[str] = REASK_PROMPT,
+        reask_prompt: str = REASK_PROMPT,
     ):
         # Initialize the Guard class
         if rail_spec is None:
@@ -144,9 +145,9 @@ def _init_guard(
     def _create_docstore_with_examples(
         self,
         examples: Optional[Dict],
-        embedding: EmbeddingBase,
-        vector_db: VectorDBBase,
-        document_store: DocumentStoreBase,
+        embedding: Type[EmbeddingBase],
+        vector_db: Type[VectorDBBase],
+        document_store: Type[DocumentStoreBase],
     ) -> Optional[DocumentStoreBase]:
         if examples is None:
             return None
@@ -167,7 +168,7 @@ def _create_docstore_with_examples(
     def output_schema_formatter(output) -> str:
         return json.dumps({"generated_sql": output}, indent=4)
 
-    def __call__(self, text: str) -> str:
+    def __call__(self, text: str) -> Optional[str]:
         """Run text2sql on a text query and return the SQL query."""
 
         if self.store is not None:
@@ -179,6 +180,12 @@ def __call__(self, text: str) -> str:
         else:
             similar_examples_prompt = ""
 
+        if asyncio.iscoroutinefunction(self.llm_api):
+            raise ValueError(
+                "Async API is not supported in Text2SQL application. "
+                "Please use a synchronous API."
+            )
+
         try:
             output = self.guard(
                 self.llm_api,
@@ -188,7 +195,11 @@ def __call__(self, text: str) -> str:
                     "db_info": str(self.sql_schema),
                 },
                 **self.llm_api_kwargs,
-            )[1]["generated_sql"]
+            )[  # type: ignore
+                1
+            ][
+                "generated_sql"
+            ]
         except TypeError:
             output = None
 

docs/javascripts/posthog.js

@@ -12,7 +12,7 @@ def compile_rail(rail: str, out: str) -> None:
     raise NotImplementedError("Currently compiling rail is not supported.")
 
 
-def validate_llm_output(rail: str, llm_output: str) -> bool:
+def validate_llm_output(rail: str, llm_output: str) -> dict:
     """Validate guardrails.yml file."""
     guard = Guard.from_rail(rail)
     result = guard.parse(llm_output)