Provide tools to build unchained language standalones

Add `mx_sdk_vm_ng` with a number of helper project, build tasks, and
distributions that can be used to build "unchained" language standalone.
In particular they allow building native images using a bootstrap
GraalVM (or the current stage1) without requiring the input jars and
build arguments to be part of a legacy stage1 GraalVM.

* `NativeImageLibraryProject` allows defining shared libraries built
  with native-image:
```
  "mylib": {
    "class": "NativeImageLibraryProject",
    "dependencies": ["SOME_JAR"],
    "build_args": ["-R:+InstallSegfaultHandler"],
  },
```
  There is also the `LanguageLibraryProject` subclass that has good
  defaults for truffle languages.
  By default, if the `substratevm` suite is loaded, it will use
  native-image from the current legacy stage1 GraalVM. If `substartevm`
  is not loaded, those native image projects will be ignored and liklely
  cause layout distributions that contain them to be recursively
  ignored.
  If the `BOOTSTRAP_GRAALVM` environment variable is set, it will be
  used to build those native images instead. If this is not set but
  `JAVA_HOME` contains the native-image tools, it will be used.
  These projects are build by default and respect
  `mx --native-images=...` arguments.
* `NativeImageExecutableProject` is similar but is used to build
  executables instead of shared library.
* `ThinLauncherProject` allows building a thin native launcher that can
  be used to start a JVM or a native language library:
```
"mylauncher": {
  "class": "ThinLauncherProject",
  "mainClass": "org.example.MyLauncher",
  "jar_distributions": ["MY_LAUNCHER"],
  "relative_jre_path": "../jvm", # for JVM mode
  "relative_module_path": "../modules", # for JVM mode
},
```
* `StandaloneLicenses` can be used to provide licenses that can be used
  in a layout distribution. It adapts to the current context being
  "community" or "enterprise":
```
"truffleruby_licenses": {
  "class": "StandaloneLicenses",
  "community_license_file": "LICENCE.md",
  "community_3rd_party_license_file": "3rd_party_licenses.txt",
},
```
* `JavaHomeDependency` can be used to represent a java home in the mx
  build system. It can be used in layout distributions to copy a JDK.
* `DynamicPOMDistribution` is like mx's `POMDistribution` but
  additionally supports `dynamicDistDependencies` and
  `dynamicDistDependencies` attributes which call back into the suite's
  mx extensions to dynamically retrieve a list of distributions. This
  can replace the use of `mx_register_dynamic_suite_constituents`.
```python filename="suite.py"
"MY_DISTRIBUTION": {
  "class": "DynamicPOMDistribution",
  "distDependencies": ["A_STATIC_DEPENDENCY"],
  "dynamicDistDependencies": "my_callback",
},
```
```python filename="mx_mysuite.py"
def my_callback():
  return [] if foo else ['A_DYNAMIC_DEPENDENCY']
```
* `DeliverableStandaloneArchive` can be used to make an archive for a
  standalone based on a layout "dir" distribution. It will adjust its
  name based on the current context being "community" or "enterprise".
  It ensure that the archive has a single root directory for easy
  consumption by user (multiple `tar xf ...` in the same directory will
  not conflit).
* `sdk:STANDALONE_JAVA_HOME` is a dependency that can be used in a
  layout distribution to get a JDK that can be used in a standalone.
  The `STANDALONE_JAVA_HOME` environment variable can be used to point
  at an explicit JDK. Otherwise it will be the `BOOTSTRAP_GRAALVM` if it
  is set. If `BOOTSTRAP_GRAALVM` was not set but `JAVA_HOME` contains
  the native-image tool, `JAVA_HOME` is used.
  If none of the above works, it will be similar to the JDK that used to
  be inserted into legacy standalones: a special jlink'd JDK that
  contains the sdk and libgraal if it is available.
* `sdk:TOOLS_FOR_STANDALONE` is a convenience pom distribution that can
  be used to depend on all known tools if they are available.
* `ExtractedEngineResources` can be used to obtain the engine resources
  for use in a layout distributions.
```
"my-resources": {
  "class": "ExtractedEngineResources",
  "dependencies": ['LANG_AND_RESOURCES_DISTS'],
},
...
  "layout": {
    "resources/": "dependency:my-resources/mylang/*",
  },
```
  Note that native image projects also produce copied internal resources
  if `-H:+CopyLanguageResources` is used.
* Define TRUFFLE_ATTACH_GRAALVM_SUPPORT to be used for standalones:
```
  "jvmlibs/": [
    "extracted-dependency:truffle:TRUFFLE_ATTACH_GRAALVM_SUPPORT",
    "extracted-dependency:truffle:TRUFFLE_NFI_NATIVE_GRAALVM_SUPPORT",
  ],
```

Co-authored-by: Benoit Daloze <[email protected]>

Author: gilles-duboscq

common.json

@@ -4,7 +4,7 @@
     "Jsonnet files should not include this file directly but use ci/common.jsonnet instead."
   ],
 
-  "mx_version": "7.40.2",
+  "mx_version": "7.45.0",
 
   "COMMENT.jdks": "When adding or removing JDKs keep in sync with JDKs in ci/common.jsonnet",
   "jdks": {

espresso/mx.espresso/mx_espresso.py

@@ -38,9 +38,10 @@
 import mx_espresso_benchmarks  # pylint: disable=unused-import
 import mx_sdk_vm
 import mx_sdk_vm_impl
+from mx_sdk_vm_ng import JavaHomeDependency
 from mx_gate import Task, add_gate_runner
 from mx_jackpot import jackpot
-from os.path import join, isabs, exists, dirname, relpath, basename
+from os.path import join, exists, dirname, relpath
 from import_order import verify_order, validate_format
 
 _suite = mx.suite('espresso')
@@ -587,49 +588,6 @@ def register_espresso_runtime_resources(register_project, register_distribution,
         }))
 
 
-class JavaHomeDependency(mx.BaseLibrary):
-    def __init__(self, suite, name, java_home):
-        assert isabs(java_home)
-        self.java_home = java_home
-        release_dict = mx_sdk_vm.parse_release_file(join(java_home, 'release'))
-        self.is_ee_implementor = release_dict.get('IMPLEMENTOR') == 'Oracle Corporation'
-        self.version = mx.VersionSpec(release_dict.get('JAVA_VERSION'))
-        self.major_version = self.version.parts[1] if self.version.parts[0] == 1 else self.version.parts[0]
-        if self.is_ee_implementor:
-            the_license = "Oracle Proprietary"
-        else:
-            the_license = "GPLv2-CPE"
-        super().__init__(suite, name, optional=False, theLicense=the_license)
-        self.deps = []
-
-    def is_available(self):
-        return True
-
-    def getBuildTask(self, args):
-        return mx.ArchivableBuildTask(self, args, 1)
-
-    def getResults(self):
-        for root, _, files in os.walk(self.java_home):
-            for name in files:
-                yield join(root, name)
-
-    def getArchivableResults(self, use_relpath=True, single=False):
-        if single:
-            raise ValueError("single not supported")
-        for path in self.getResults():
-            if use_relpath:
-                arcname = relpath(path, self.java_home)
-            else:
-                arcname = basename(path)
-            yield path, arcname
-
-    def post_init(self):
-        pass  # help act like a distribution since this is registered as a distribution
-
-    def archived_deps(self):
-        return []  # help act like a distribution since this is registered as a distribution
-
-
 class EspressoRuntimeResourceProject(mx.JavaProject):
     def __init__(self, suite, subDir, runtime_name, theLicense):
         name = f'com.oracle.truffle.espresso.resources.runtime'

sdk/mx.sdk/mx_sdk.py

@@ -46,6 +46,7 @@
 import mx_gate
 import mx_sdk_vm
 import mx_sdk_vm_impl
+import mx_sdk_vm_ng
 import pathlib
 import mx_sdk_benchmark # pylint: disable=unused-import
 import mx_sdk_clangformat # pylint: disable=unused-import
@@ -193,6 +194,7 @@ def upx(args):
 
 def mx_register_dynamic_suite_constituents(register_project, register_distribution):
     mx_sdk_vm_impl.mx_register_dynamic_suite_constituents(register_project, register_distribution)
+    mx_sdk_vm_ng.mx_register_dynamic_suite_constituents(register_project, register_distribution)
 
 
 def mx_post_parse_cmd_line(args):

sdk/mx.sdk/mx_sdk_vm_impl.py

@@ -60,6 +60,8 @@
 import textwrap
 import zipfile
 
+import mx_sdk_vm_ng
+
 try:
     # Use more secure defusedxml library, if available
     from defusedxml.ElementTree import parse as etreeParse
@@ -3154,7 +3156,7 @@ def get_stage1_graalvm_distribution():
     """:rtype: GraalVmLayoutDistribution"""
     global _stage1_graalvm_distribution
     if _stage1_graalvm_distribution == 'uninitialized':
-        _stage1_graalvm_distribution = GraalVmLayoutDistribution(_graalvm_base_name, stage1=True)
+        _stage1_graalvm_distribution = GraalVmLayoutDistribution(_graalvm_base_name, stage1=True, defaultBuild=False)
         _stage1_graalvm_distribution.description = "GraalVM distribution (stage1)"
         _stage1_graalvm_distribution.maven = False
     return _stage1_graalvm_distribution
@@ -3517,10 +3519,10 @@ def register_main_dist(dist, label):
                 register_distribution(debuginfo_dist)
                 main_dists[label].append(debuginfo_dist.name)
 
-    _final_graalvm_distribution = get_final_graalvm_distribution()
+    final_graalvm_distribution = get_final_graalvm_distribution()
 
     from mx_native import TargetSelection
-    for c in _final_graalvm_distribution.components:
+    for c in final_graalvm_distribution.components:
         if c.extra_native_targets:
             for t in c.extra_native_targets:
                 mx.logv(f"Selecting extra target '{t}' from GraalVM component '{c.short_name}'.")
@@ -3532,7 +3534,7 @@ def register_main_dist(dist, label):
 
     names = set()
     short_names = set()
-    needs_stage1 = False
+    needs_stage1 = mx_sdk_vm_ng.requires_native_image_stage1()
     installables = {}
     jvmci_parent_jars = []
     modified_jmods = {}
@@ -3600,10 +3602,10 @@ def register_main_dist(dist, label):
         register_project(native_image_resources_filelist_project)
 
     # Register main distribution
-    register_main_dist(_final_graalvm_distribution, 'graalvm')
+    register_main_dist(final_graalvm_distribution, 'graalvm')
 
     # Register standalones
-    needs_java_standalone_jimage = False
+    needs_java_standalone_jimage = mx_sdk_vm_ng.requires_standalone_jimage()
     for components in installables.values():
         main_component = _get_main_component(components)
         svm_support = _get_svm_support()
@@ -3622,7 +3624,7 @@ def register_main_dist(dist, label):
                         mx.warn("Skipping JVM standalone of '{}' because it contains launcher configs that are not yet supported".format(main_component.name))
                 else:
                     needs_java_standalone_jimage = True
-                    java_standalone = GraalVmStandaloneComponent(get_component(main_component.name, fatalIfMissing=True), _final_graalvm_distribution, is_jvm=True, defaultBuild=False)
+                    java_standalone = GraalVmStandaloneComponent(get_component(main_component.name, fatalIfMissing=True), final_graalvm_distribution, is_jvm=True, defaultBuild=False)
                     register_main_dist(java_standalone, 'graalvm_standalones')
 
                     # Use `main_component.library_configs` rather than `_get_library_configs(main_component)` because we
@@ -3638,7 +3640,7 @@ def register_main_dist(dist, label):
                 only_native_launchers = not main_component.launcher_configs or has_svm_launcher(main_component)
                 only_native_libraries = not main_component.library_configs or (svm_support.is_supported() and not _has_skipped_libraries(main_component))
                 if only_native_launchers and only_native_libraries:
-                    native_standalone = GraalVmStandaloneComponent(get_component(main_component.name, fatalIfMissing=True), _final_graalvm_distribution, is_jvm=False, defaultBuild=False)
+                    native_standalone = GraalVmStandaloneComponent(get_component(main_component.name, fatalIfMissing=True), final_graalvm_distribution, is_jvm=False, defaultBuild=False)
                     register_main_dist(native_standalone, 'graalvm_standalones')
 
     if needs_java_standalone_jimage:
@@ -3661,7 +3663,7 @@ def register_main_dist(dist, label):
             suite=_suite,
             name='java-standalone-jimage',
             jimage_jars=sorted(java_standalone_jimage_jars),
-            jimage_ignore_jars=sorted(_final_graalvm_distribution.jimage_ignore_jars),
+            jimage_ignore_jars=sorted(final_graalvm_distribution.jimage_ignore_jars),
             workingSets=None,
             defaultBuild=False,
             missing_export_target_action='warn',
@@ -3677,32 +3679,34 @@ def register_main_dist(dist, label):
                 else:
                     config_class = GraalVmMiscLauncher
                 for launcher_config in _get_launcher_configs(component):
-                    register_project(config_class(component, launcher_config, stage1=True))
-            for component in registered_graalvm_components(stage1=False):
-                # native properties in the final distribution also need native properties in the stage1 distribution
-                for launcher_config in _get_launcher_configs(component):
-                    register_project(GraalVmNativeProperties(component, launcher_config, stage1=True))
-                for library_config in _get_library_configs(component):
-                    register_project(GraalVmNativeProperties(component, library_config, stage1=True))
+                    register_project(config_class(component, launcher_config, stage1=True, defaultBuild=False))
+            if get_component('svm', stage1=True):
+                for component in registered_graalvm_components(stage1=False):
+                    # native properties in the final distribution also need native properties in the stage1 distribution
+                    for launcher_config in _get_launcher_configs(component):
+                        register_project(GraalVmNativeProperties(component, launcher_config, stage1=True, defaultBuild=False))
+                    for library_config in _get_library_configs(component):
+                        register_project(GraalVmNativeProperties(component, library_config, stage1=True, defaultBuild=False))
         register_distribution(get_stage1_graalvm_distribution())
 
     if register_project:
         if needs_stage1:
-            _stage1_graalvm_distribution = get_stage1_graalvm_distribution()
-            if _needs_stage1_jimage(_stage1_graalvm_distribution, _final_graalvm_distribution):
+            stage1_graalvm_distribution = get_stage1_graalvm_distribution()
+            if _needs_stage1_jimage(stage1_graalvm_distribution, final_graalvm_distribution):
                 register_project(GraalVmJImage(
                     suite=_suite,
                     name='graalvm-stage1-jimage',
-                    jimage_jars=sorted(_stage1_graalvm_distribution.jimage_jars),
-                    jimage_ignore_jars=sorted(_stage1_graalvm_distribution.jimage_ignore_jars),
+                    jimage_jars=sorted(stage1_graalvm_distribution.jimage_jars),
+                    jimage_ignore_jars=sorted(stage1_graalvm_distribution.jimage_ignore_jars),
                     workingSets=None,
                     default_to_jvmci=False,  # decide depending on the included modules
+                    defaultBuild=False,
                 ))
         final_jimage_project = GraalVmJImage(
             suite=_suite,
             name='graalvm-jimage',
-            jimage_jars=sorted(_final_graalvm_distribution.jimage_jars),
-            jimage_ignore_jars=sorted(_final_graalvm_distribution.jimage_ignore_jars),
+            jimage_jars=sorted(final_graalvm_distribution.jimage_jars),
+            jimage_ignore_jars=sorted(final_graalvm_distribution.jimage_ignore_jars),
             workingSets=None,
             default_to_jvmci=_get_libgraal_component() is not None,
         )
@@ -4575,15 +4579,16 @@ def _expand_native_images_list(only):
         return only
 
 
-def _force_bash_launchers(launcher):
+def _force_bash_launchers(launcher, build_by_default=None):
     """
     :type launcher: str | mx_sdk.LauncherConfig
     """
     launcher_name = _get_launcher_name(launcher)
 
     forced = _parse_cmd_arg('force_bash_launchers')
-    default = has_vm_suite() or forced is not None # for compatibility with legacy behavior
-    only = _parse_cmd_arg('native_images', default_value=str(default))
+    if build_by_default is None:
+        build_by_default = has_vm_suite() or forced is not None # for compatibility with legacy behavior
+    only = _parse_cmd_arg('native_images', default_value=str(build_by_default))
     only = _expand_native_images_list(only)
     if isinstance(only, bool):
         included = only
@@ -4598,15 +4603,16 @@ def _force_bash_launchers(launcher):
     return not included
 
 
-def _skip_libraries(library):
+def _skip_libraries(library, build_by_default=None):
     """
     :type library: str | mx_sdk.LibraryConfig
     """
     library_name = _get_library_name(library)
 
     skipped = _parse_cmd_arg('skip_libraries')
-    default = has_vm_suite() or skipped is not None # for compatibility with legacy behavior
-    only = _parse_cmd_arg('native_images', default_value=str(default))
+    if build_by_default is None:
+        build_by_default = has_vm_suite() or skipped is not None # for compatibility with legacy behavior
+    only = _parse_cmd_arg('native_images', default_value=str(build_by_default))
     only = _expand_native_images_list(only)
     if isinstance(only, bool):
         included = only
@@ -4748,12 +4754,15 @@ def _generate_debuginfo(image_config):
     elif isinstance(generate_debuginfo, bool):
         return generate_debuginfo
     else:
-        destination = image_config.destination if isinstance(image_config, mx_sdk.AbstractNativeImageConfig) else image_config
-        if isinstance(image_config, mx_sdk.LauncherConfig):
-            name = basename(remove_exe_suffix(destination, require_suffix=False))
-        elif isinstance(image_config, mx_sdk.LibraryConfig):
-            name = remove_lib_prefix_suffix(basename(destination), require_suffix_prefix=False)
-            generate_debuginfo = [lib[4:] for lib in generate_debuginfo if lib.startswith('lib:')]
+        if isinstance(image_config, str):
+            name = image_config
+        else:
+            destination = image_config.destination if isinstance(image_config, mx_sdk.AbstractNativeImageConfig) else image_config
+            if isinstance(image_config, mx_sdk.LauncherConfig):
+                name = basename(remove_exe_suffix(destination, require_suffix=False))
+            elif isinstance(image_config, mx_sdk.LibraryConfig):
+                name = remove_lib_prefix_suffix(basename(destination), require_suffix_prefix=False)
+                generate_debuginfo = [lib[4:] for lib in generate_debuginfo if lib.startswith('lib:')]
         return name in generate_debuginfo