Skip to content

Add Python package module for Python+Chapel interop #26156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 60 commits into from
Nov 21, 2024

Conversation

jabraham17
Copy link
Member

@jabraham17 jabraham17 commented Oct 29, 2024

Adds a new package module, Python, which supports calling Python code from Chapel.

For example, the following Chapel progam use the python interface to BeautifulSoup to parse HTML

import Python;
use URL;
use List;

proc main() {

  const url = 'https://chapel-lang.org/docs/main/language/spec/interoperability.html';
  var htmlReader = openUrlReader(url);
  var html = htmlReader.readAll(string);

  var interp = new Python.Interpreter();
  var mod = new Python.Module(interp, "bs4");

  var cls = new Python.Class(mod, "BeautifulSoup");
  var soup = cls(html, 'html.parser');

  var res: list(owned Python.ClassObject?);
  res = soup.callMethod(res.type, "find_all", "h3");
  for c in res {
    writeln(c!.getAttr(string, "text"));
  }
}

As an another example, this simple program compiles an runs a Python lambda from Chapel code

import Python;

config const n = 10;
config const func = "lambda x,: x + 1 if x % 2 != 0 else x";

proc apply(interp: borrowed, type t, arr, l) {
  var lambdaFunc = new Python.Function(interp, l);
  var res: [arr.domain] t;
  for i in arr.domain {
    res(i) = lambdaFunc(t, arr(i));
  }
  return res;
}
proc main() {
  var interp = new Python.Interpreter();

  var data: [1..#n] int = 1..#n;
  writeln(" data: ", data);

  var res = apply(interp, int, data, func);
  writeln("res: ", res);
}

Future work:

  • Convert classes to a hierarchy under Value so that more implementation can be shared and the Chapel interface can better imitate python
    • Add special support for common python types as sub-classes of value, to avoid round tripping values through chapel in many cases
    • Support common operators like add. These can be called today as dunder methods (.call("__add__", ...)), but it would be nice to support native operators like +
  • Add the ability to compile arbitrary chapel strings, beyond just lambdas
  • Setup python to use Chapel stdout/stderr
  • Add custom adapters for Chapel arrays to allow python functions to operate on Chapel arrays, without copying
  • Support python context managers as Chapel context managers
  • Fully support Python sets, Python dictionaries, Chapel sets, Chapel maps, Chapel associative arrays, and Chapel associative domains
  • Make sure all python objects are properly reference counted
  • Resolve TODO with LLVM IR verification: [Bug]: extern records cannot have a different ordering of fields #26235

[Reviewed by @DanilaFe]

Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
@jabraham17 jabraham17 requested a review from DanilaFe November 6, 2024 23:16
Copy link
Contributor

@DanilaFe DanilaFe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the code, not the tests yet.

Comment on lines 759 to 760
init this;
this.fn!.check();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider invoking the name/object constructor (init#2) to share some code.

proc init(interpreter: borrowed Interpreter, in lambdaFn: string) throws {
this.fnName = "<anon>";
init this;
this.fn = interpreter.compileLambda(lambdaFn);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO come back to this

Comment on lines 973 to 981
var pyArgs: args.size * c_ptr(void);
for param i in 0..#args.size {
pyArgs(i) = interpreter.toPython(args(i));
}
var pyRes;
if pyArgs.size == 1 then
pyRes = PyObject_CallOneArg(this.obj!.get(), pyArgs(0));
else
pyRes = PyObject_CallFunctionObjArgs(this.obj!.get(), (...pyArgs), nil);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is duplicated; might you be able to share it with the Function instance? Moreover, aren't functions "just" ClassObjects?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there is a lot of duplicated code right now between Function and ClassObject. I plan to introduce a class hierarchy to handle this in a future PR

Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
@jabraham17 jabraham17 requested a review from DanilaFe November 20, 2024 18:53
Copy link
Contributor

@DanilaFe DanilaFe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your changes look good and I've read-skimmed the tests, trusting you on those.

Signed-off-by: Jade Abraham <[email protected]>
Signed-off-by: Jade Abraham <[email protected]>
@jabraham17 jabraham17 merged commit 53bee29 into chapel-lang:main Nov 21, 2024
7 checks passed
@jabraham17 jabraham17 deleted the python-interop branch November 21, 2024 03:00
jabraham17 added a commit that referenced this pull request Nov 21, 2024
Fixes some new Python interop tests (added in
#26156) with COMM!=none. This
is caused by a difference between Chapel's stdio and Python's stderr.

The difference is that Chapel does some extra work in the multi-locale
case to make sure everything is printed on the right locales, whereas
Python has no native concept of locality. In this case, this just
manifests as changing whether the Python output comes first or last.

This PR also switches the interpreter to be explicitly an "isolated"
interpreter, to prevent potential issues with Python setting up its
stdio.

Tested that all Python tests pass with COMM=none and COMM=gasnet

[Reviewed by @DanilaFe]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants