Build a conda-smithy language server

[bollwyvl]

This fundable has not been submitted yet.

• This fundable has not been submitted yet.

Motivation

The journey of a new contributor to conda-forge (or any conda-smithy-based ecosystem) can be somewhat bewildering, looking something like:

• a single (generated, or copy-pasta'd) file in staged-recipes/recipes/{my-package}
• dozens of files in {my-package}-feedstock
• conda-forge-pinning/recipe/migrations/the-platform-i-really-want.txt
• sometimes admin-requests/requests/oops-i-broke-it.yaml
• maybe conda-forge-repodata-patches/requests/oops-someone-else-broke-it.yaml

Many of the well-formed YAML files have existing JSON Schema, and even the less-well-formed ones have bespoke parsers and linters which can provide additional insights without trial-by-CI exploration, if one knows to install the right editor tools.

Additional CLI tools, maintained by the conda-forge and broader conda communities, go a bit further, and provide broader insights, even spanning multiple top-level channels, but these generally are not visible in text editors.

Combining the insights of these schemas and tools into something that provides per key-stroke insight into contributors' "daily driver" text editors would be an effective reuse of these existing tools, but would primarily make it easier for contributors to generate quality packages.

To achieve the above, the language server protocol, for all its warts, provides the most widely-supported approach to delivering these capabilities, without requiring commitment to a specific editor's API and data model.

Expected result

A useful MVP language server would:

• provide the "big ticket" features expected from a language server from conda-smithy recipe-lint:
  • diagnostics, a.k.a "red squigglies" for more contents of syntax fields
  • completion, e.g. ctrl+space to get requirement names from a feedstock's channel_sources, with sane caching
• be easy to install from conda-forge
• be easy to configure multiple editors to use the language server
• be well-tested against a large set of in-the-wild packages

As a midterm, this package and its dependencies could be added to a dedicated environment in e.g. (staged-recipes|admin-requests)/pixi.toml#feature.conda-smithy-ls with commands like start-vscode, start-neovim, start-jupyterlab, etc.

Some more advanced "final exams" might include:

• a miniforge variant that shipped a full, ready-to-develop environment suitable for rapid onboarding in a conference workshop setting
• a zero-install, statically-hosted (e.g. RTD, GitHub Pages) browser-based tool, provisioned with emscripten-forge (or, better, conda-forge if we ever get an emscripten target)
• add more support for code actions such as
  • reformatting (with e.g. prettier)
  • refactoring (described as python, but potentially more interesting and portable as declarative json-e, jsonata or semgrep patterns)

Work items

• start a package
  • use pixi to provision, manage, build, document, and test the new package
  • start with a python-based language server on pygls
    • likely target the 2.0.0 pre-release baseline
  • wrap conda-smithy recipe-lint in language server diagnostics
  • provide offline-ready, versioned schema for all schema files we have
  • consider packaging some optimized/compressed subsets of feedstock-outputs, conda-suggest, etc. refreshed on some interval
  • maybe add a few lark grammars for e.g migrations
• release on an appropriate upstream (probably PyPI)
• release on conda-forge
• add to existing singleton repos' pixi.toml
  • staged-recipes
  • admin-requests
  • conda-forge-pinning
  • conda-forge-repodata-patches
• add as an option to conda-forge.yml which conda-smithy would realize into a local pixi environment

References

• conda-forge
  • original zulip thread
  • conda-smithy line number hints PR
  • pygls 2.0.0 alpha
• external
  • the language server protocol

Skills / personnel

• python development (see below)
• familiarity with conda-smithy and related tools and file formats
• familiarity with ancillary conda-forge repos and workflows such as staged-recipes, admin-requests, etc.
• knowledge of JSON schema, and likely some richer text parser generators such as lark
• knowledge of the language server protocol

Estimated costs and/or resources needed (optional)

Thanks to tools like pygls, something could be cobbled together very quickly, even during a conference sprint. The most challenging changes might be hacking existing tools to provide enough context (line numbers) to drive the editor experience.

A longer commitment, such as Summer of Code, or other funded activity, would allow this to be more extensively explored. For example, while python is sufficient and most easily maintainable by the conda-forge community, a rust or go-based tool would probably end up being much more performant, and therefore satisfying, editor experience for users.

[bollwyvl]

I've been poking around a bit on this: things seem ok, and (unsurprisingly) have found some stuff to package that would help:

• add pytest-lsp staged-recipes#30424

Meanwhile, on consuming conda-smithy output things are a bit less rosy, but I'll start some more issues over there.

[bollwyvl]

Opened conda-forge/conda-smithy#2342