Sync Collection with changes to prefect-collection-template (#204)

Co-authored-by: desertaxle <[email protected]>
Co-authored-by: ahuang11 <[email protected]>
Co-authored-by: Andrew <[email protected]>

Author: 4 people

.cruft.json

@@ -1,6 +1,6 @@
 {
   "template": "https://github.com/PrefectHQ/prefect-collection-template",
-  "commit": "9495ca71f74c279e9d9beff90f1d9f53f77637aa",
+  "commit": "e11a3be195f24f60ed3f564dfccb40170ee7c3fa",
   "checkout": null,
   "context": {
     "cookiecutter": {

.github/workflows/nightly-dev-tests.yml

@@ -0,0 +1,38 @@
+name: Nightly tests against Prefect's main branch
+on:
+  schedule:
+    - cron: "0 6 * * *"
+  workflow_dispatch:
+
+jobs:
+  submit-update-pr:
+    name: Run tests against Prefect's main branch
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version:
+          - "3.7"
+          - "3.8"
+          - "3.9"
+          - "3.10"
+      fail-fast: false
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: requirements*.txt
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install --upgrade --upgrade-strategy eager -e ".[dev]" "prefect @ git+https://github.com/PrefectHQ/prefect.git@main"
+
+      - name: Run tests
+        env:
+          PREFECT_API_DATABASE_CONNECTION_URL: "sqlite+aiosqlite:///./collection-tests.db"
+        run: |
+          pytest tests -vv

.github/workflows/template-sync.yml

@@ -1,7 +1,7 @@
 name: Template Synchronization
 on:
   schedule:
-    - cron: "0 0 * * *"
+    - cron: "0 6 * * *"
   workflow_dispatch:
 
 jobs:

.github/workflows/tests.yml

@@ -31,9 +31,9 @@ jobs:
 
       - name: Run tests
         env:
-          PREFECT_ORION_DATABASE_CONNECTION_URL: "sqlite+aiosqlite:///./orion-tests.db"
+          PREFECT_SERVER_DATABASE_CONNECTION_URL: "sqlite+aiosqlite:///./collection-tests.db"
         run: |
-          prefect orion database reset -y
+          prefect server database reset -y
           coverage run --branch -m pytest tests -vv
           coverage report
 

.github/workflows/windows-tests.yml

@@ -0,0 +1,34 @@
+
+name: Windows Tests
+
+on: [pull_request]
+
+jobs:
+  run-tests:
+    name: Run Tests
+    runs-on: windows-latest
+    strategy:
+      matrix:
+        # Prefect only tests 3.9 on Windows
+        python-version:
+          - "3.9"
+      fail-fast: false
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: requirements*.txt
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install --upgrade --upgrade-strategy eager -e ".[dev]"
+      - name: Run tests
+        env:
+          PREFECT_SERVER_DATABASE_CONNECTION_URL: "sqlite+aiosqlite:///./collection-tests.db"
+        run: |
+          pytest tests -vv

MAINTAINERS.md

@@ -66,7 +66,7 @@ To verify the setup was successful you can run the following:
 
 ## Developing tasks and flows
 
-For information about the use and development of tasks and flow, check out the [flows](https://orion-docs.prefect.io/concepts/flows/) and [tasks](https://orion-docs.prefect.io/concepts/tasks/) concepts docs in the Prefect docs.
+For information about the use and development of tasks and flow, check out the [flows](https://docs.prefect.io/concepts/flows/) and [tasks](https://docs.prefect.io/concepts/tasks/) concepts docs in the Prefect docs.
 
 ## Writing documentation
 
@@ -98,7 +98,7 @@ To publish a new version of your collection, [create a new GitHub release](https
 
 Upon publishing, a `docs` branch is automatically created. To hook this up to GitHub Pages, simply head over to https://github.com/PrefectHQ/prefect-aws/settings/pages, select `docs` under the dropdown menu, keep the default `/root` folder, `Save`, and upon refresh, you should see a prompt stating "Your site is published at https://PrefectHQ.github.io/prefect-aws". Don't forget to add this link to the repo's "About" section, under "Website" so users can access the docs easily.
 
-Feel free to [submit your collection](https://orion-docs.prefect.io/collections/overview/#listing-in-the-collections-catalog) to the Prefect [Collections Catalog](https://orion-docs.prefect.io/collections/catalog/)!
+Feel free to [submit your collection](https://docs.prefect.io/collections/overview/#listing-in-the-collections-catalog) to the Prefect [Collections Catalog](https://docs.prefect.io/collections/catalog/)!
 
 ## Further guidance
 

README.md

@@ -99,7 +99,7 @@ custom_example_flow = example_flow.with_options(
 
 Refer to the API documentation on the sidebar to explore all the capabilities of Prefect AWS!
 
-For more tips on how to use tasks and flows in a Collection, check out [Using Collections](https://orion-docs.prefect.io/collections/usage/)!
+For more tips on how to use tasks and flows in a Collection, check out [Using Collections](https://docs.prefect.io/collections/usage/)!
 
 ### Recipes
 
@@ -127,7 +127,7 @@ Requires an installation of Python 3.7+
 
 We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.
 
-These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to the [Prefect documentation](https://orion-docs.prefect.io/).
+These tasks are designed to work with Prefect 2.0. For more information about how to use Prefect, please refer to the [Prefect documentation](https://docs.prefect.io/).
 
 ### Contributing
 

docs/gen_blocks_catalog.py

@@ -0,0 +1,103 @@
+"""
+Discovers all blocks and generates a list of them in the docs
+under the Blocks Catalog heading.
+"""
+
+from pathlib import Path
+from textwrap import dedent
+
+import mkdocs_gen_files
+from prefect.blocks.core import Block
+from prefect.utilities.dispatch import get_registry_for_type
+from prefect.utilities.importtools import from_qualified_name, to_qualified_name
+
+COLLECTION_SLUG = "prefect_aws"
+
+
+def find_module_blocks():
+    blocks = get_registry_for_type(Block)
+    collection_blocks = [
+        block
+        for block in blocks.values()
+        if to_qualified_name(block).startswith(COLLECTION_SLUG)
+    ]
+    module_blocks = {}
+    for block in collection_blocks:
+        block_name = block.__name__
+        module_nesting = tuple(to_qualified_name(block).split(".")[1:-1])
+        if module_nesting not in module_blocks:
+            module_blocks[module_nesting] = []
+        module_blocks[module_nesting].append(block_name)
+    return module_blocks
+
+
+def insert_blocks_catalog(generated_file):
+    module_blocks = find_module_blocks()
+    if len(module_blocks) == 0:
+        return
+    generated_file.write(
+        dedent(
+            f"""
+            Below is a list of Blocks available for registration in
+            `prefect-aws`.
+
+            To register blocks in this module to
+            [view and edit them](https://docs.prefect.io/ui/blocks/)
+            on Prefect Cloud, first [install the required packages](
+            https://PrefectHQ.github.io/prefect-aws/#installation),
+            then
+            ```bash
+            prefect block register -m {COLLECTION_SLUG}
+            ```
+            """  # noqa
+        )
+    )
+    generated_file.write(
+        "Note, to use the `load` method on Blocks, you must already have a block document "  # noqa
+        "[saved through code](https://docs.prefect.io/concepts/blocks/#saving-blocks) "  # noqa
+        "or [saved through the UI](https://docs.prefect.io/ui/blocks/).\n"
+    )
+    for module_nesting, block_names in module_blocks.items():
+        module_path = f"{COLLECTION_SLUG}." + " ".join(module_nesting)
+        module_title = (
+            module_path.replace(COLLECTION_SLUG, "")
+            .lstrip(".")
+            .replace("_", " ")
+            .title()
+        )
+        generated_file.write(f"## [{module_title} Module][{module_path}]\n")
+        for block_name in block_names:
+            block_obj = from_qualified_name(f"{module_path}.{block_name}")
+            block_description = block_obj.get_description()
+            if not block_description.endswith("."):
+                block_description += "."
+            generated_file.write(
+                f"[{block_name}][{module_path}.{block_name}]\n\n{block_description}\n\n"
+            )
+            generated_file.write(
+                dedent(
+                    f"""
+                    To load the {block_name}:
+                    ```python
+                    from prefect import flow
+                    from {module_path} import {block_name}
+
+                    @flow
+                    def my_flow():
+                        my_block = {block_name}.load("MY_BLOCK_NAME")
+
+                    my_flow()
+                    ```
+                    """
+                )
+            )
+        generated_file.write(
+            f"For additional examples, check out the [{module_title} Module]"
+            f"(../examples_catalog/#{module_nesting[-1]}-module) "
+            f"under Examples Catalog.\n"
+        )
+
+
+blocks_catalog_path = Path("blocks_catalog.md")
+with mkdocs_gen_files.open(blocks_catalog_path, "w") as generated_file:
+    insert_blocks_catalog(generated_file)

docs/gen_examples_catalog.py

@@ -0,0 +1,120 @@
+"""
+Locates all the examples in the Collection and puts them in a single page.
+"""
+
+import re
+from collections import defaultdict
+from inspect import getmembers, isclass, isfunction
+from pathlib import Path
+from pkgutil import iter_modules
+from textwrap import dedent
+from types import ModuleType
+from typing import Callable, Set, Union
+
+import mkdocs_gen_files
+from griffe.dataclasses import Docstring
+from griffe.docstrings.dataclasses import DocstringSectionKind
+from griffe.docstrings.parsers import Parser, parse
+from prefect.logging.loggers import disable_logger
+from prefect.utilities.importtools import load_module, to_qualified_name
+
+import prefect_aws
+
+COLLECTION_SLUG = "prefect_aws"
+
+
+def skip_parsing(name: str, obj: Union[ModuleType, Callable], module_nesting: str):
+    """
+    Skips parsing the object if it's a private object or if it's not in the
+    module nesting, preventing imports from other libraries from being added to the
+    examples catalog.
+    """
+    try:
+        wrong_module = not to_qualified_name(obj).startswith(module_nesting)
+    except AttributeError:
+        wrong_module = False
+    return obj.__doc__ is None or name.startswith("_") or wrong_module
+
+
+def skip_block_load_code_example(code_example: str) -> bool:
+    """
+    Skips the code example if it's just showing how to load a Block.
+    """
+    return re.search(r'\.load\("BLOCK_NAME"\)\s*$', code_example.rstrip("`"))
+
+
+def get_code_examples(obj: Union[ModuleType, Callable]) -> Set[str]:
+    """
+    Gathers all the code examples within an object.
+    """
+    code_examples = set()
+    with disable_logger("griffe.docstrings.google"):
+        with disable_logger("griffe.agents.nodes"):
+            docstring = Docstring(obj.__doc__)
+            parsed_sections = parse(docstring, Parser.google)
+
+    for section in parsed_sections:
+        if section.kind == DocstringSectionKind.examples:
+            code_example = "\n".join(
+                (part[1] for part in section.as_dict().get("value", []))
+            )
+            if not skip_block_load_code_example(code_example):
+                code_examples.add(code_example)
+        if section.kind == DocstringSectionKind.admonition:
+            value = section.as_dict().get("value", {})
+            if value.get("annotation") == "example":
+                code_example = value.get("description")
+                if not skip_block_load_code_example(code_example):
+                    code_examples.add(code_example)
+
+    return code_examples
+
+
+code_examples_grouping = defaultdict(set)
+for _, module_name, ispkg in iter_modules(prefect_aws.__path__):
+
+    module_nesting = f"{COLLECTION_SLUG}.{module_name}"
+    module_obj = load_module(module_nesting)
+
+    # find all module examples
+    if skip_parsing(module_name, module_obj, module_nesting):
+        continue
+    code_examples_grouping[module_name] |= get_code_examples(module_obj)
+
+    # find all class and method examples
+    for class_name, class_obj in getmembers(module_obj, isclass):
+        if skip_parsing(class_name, class_obj, module_nesting):
+            continue
+        code_examples_grouping[module_name] |= get_code_examples(class_obj)
+        for method_name, method_obj in getmembers(class_obj, isfunction):
+            if skip_parsing(method_name, method_obj, module_nesting):
+                continue
+            code_examples_grouping[module_name] |= get_code_examples(method_obj)
+
+    # find all function examples
+    for function_name, function_obj in getmembers(module_obj, callable):
+        if skip_parsing(function_name, function_obj, module_nesting):
+            continue
+        code_examples_grouping[module_name] |= get_code_examples(function_obj)
+
+
+examples_catalog_path = Path("examples_catalog.md")
+with mkdocs_gen_files.open(examples_catalog_path, "w") as generated_file:
+    generated_file.write(
+        dedent(
+            """
+            # Examples Catalog
+
+            Below is a list of examples for `prefect-aws`.
+            """
+        )
+    )
+    for module_name, code_examples in code_examples_grouping.items():
+        if len(code_examples) == 0:
+            continue
+        module_title = module_name.replace("_", " ").title()
+        generated_file.write(
+            f"## [{module_title} Module][{COLLECTION_SLUG}.{module_name}]\n"
+        )
+        for code_example in code_examples:
+            generated_file.write(code_example + "\n")

docs/gen_home_page.py

@@ -0,0 +1,21 @@
+"""
+Copies README.md to index.md.
+"""
+
+from pathlib import Path
+
+import mkdocs_gen_files
+
+# Home page
+
+readme_path = Path("README.md")
+docs_index_path = Path("index.md")
+
+with open(readme_path, "r") as readme:
+    with mkdocs_gen_files.open(docs_index_path, "w") as generated_file:
+        for line in readme:
+            if line.startswith("Visit the full docs [here]("):
+                continue  # prevent linking to itself
+            generated_file.write(line)
+
+    mkdocs_gen_files.set_edit_path(Path(docs_index_path), readme_path)