"accelerator" directive is not working in Google Batch

[puerta-amarilla]

Bug report

(Please follow this template by replacing the text between parentheses with the requested information)

Expected behavior and actual behavior

(Give a brief description of the expected behavior and actual behavior)
According to the documentation, accelerator can be specified like this:

process hello {
    accelerator 4, type: 'nvidia-tesla-k80'
    ...
}

Steps to reproduce the problem

Specify as follows:

  'google-batch' {
      params.transcriptome = 'gs://rnaseq-nf/data/ggal/transcript.fa'
      params.reads = 'gs://rnaseq-nf/data/ggal/gut_{1,2}.fq'
      params.multiqc = 'gs://rnaseq-nf/multiqc'
      process {
        executor = 'google-batch'
        accelerator 4, type: 'nvidia-tesla-a100'
        container = 'docker.io/nextflow/rnaseq-nf:v1.3.1'
      }

Program output

ERROR ~ Unknown method invocation `accelerator` on _parse_closure5 type

 -- Check '.nextflow.log' file for details

In the log file:

Oct-03 03:27:02.166 [main] ERROR nextflow.cli.Launcher - Unable to parse config file: '/usr/local/google/home/jeffling/projects/Mayo clinic/nextflow_on_batch/rnaseq-nf/nextflow.config'

  No signature of method: nextflow.config.parser.v1.ConfigParserV1$_parse_closure5.accelerator() is applicable for argument types: (LinkedHashMap, Integer) values: [[type:nvidia-tesla-a100], 4]

groovy.lang.MissingMethodException: No signature of method: nextflow.config.parser.v1.ConfigParserV1$_parse_closure5.accelerator() is applicable for argument types: (LinkedHashMap, Integer) values: [[type:nvidia-tesla-a100], 4]
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.parser.v1.ConfigParserV1$_parse_closure5.doCall(ConfigParserV1.groovy:438)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:88)
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1140)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:165)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:392)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:329)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:392)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:329)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:392)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:329)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at _nf_config_d5b5b6a7$_run_closure2$_closure15$_closure18.doCall(_nf_config_d5b5b6a7:109)
	at _nf_config_d5b5b6a7$_run_closure2$_closure15$_closure18.doCall(_nf_config_d5b5b6a7)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:280)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.parser.v1.ConfigParserV1$_parse_closure5.doCall(ConfigParserV1.groovy:415)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:88)
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1140)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:165)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:392)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:329)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:392)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:329)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at _nf_config_d5b5b6a7$_run_closure2$_closure15.doCall(_nf_config_d5b5b6a7:107)
	at _nf_config_d5b5b6a7$_run_closure2$_closure15.doCall(_nf_config_d5b5b6a7)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:280)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.parser.v1.ConfigParserV1$_parse_closure5.doCall(ConfigParserV1.groovy:381)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:88)
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1140)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:165)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:645)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:628)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:392)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:329)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at _nf_config_d5b5b6a7$_run_closure2.doCall(_nf_config_d5b5b6a7:103)
	at _nf_config_d5b5b6a7$_run_closure2.doCall(_nf_config_d5b5b6a7)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:328)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:280)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.parser.v1.ConfigParserV1$_parse_closure5.doCall(ConfigParserV1.groovy:357)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:88)
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1140)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:165)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at _nf_config_d5b5b6a7.run(_nf_config_d5b5b6a7:32)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.parser.v1.ConfigParserV1$_parse_closure5.doCall(ConfigParserV1.groovy:436)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:343)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaMethod.invoke(ClosureMetaMethod.java:88)
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1140)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1007)
	at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:165)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.parser.v1.ConfigParserV1.parse(ConfigParserV1.groovy:465)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.selectMethod(IndyInterface.java:355)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.parser.v1.ConfigParserV1.parse(ConfigParserV1.groovy:291)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.selectMethod(IndyInterface.java:355)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.parse0(ConfigBuilder.groovy:432)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.selectMethod(IndyInterface.java:355)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.merge0(ConfigBuilder.groovy:417)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.selectMethod(IndyInterface.java:355)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.buildConfig0(ConfigBuilder.groovy:374)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.buildGivenFiles(ConfigBuilder.groovy:318)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.buildConfigObject(ConfigBuilder.groovy:816)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.build(ConfigBuilder.groovy:829)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:333)
	at nextflow.cli.Launcher.run(Launcher.groovy:513)
	at nextflow.cli.Launcher.main(Launcher.groovy:673)

Environment

• Nextflow version: [?]
• Java version: [?]
• Operating system: [macOS, Linux, etc]
• Bash version: (use the command $SHELL --version)

Additional context

(Add any other context about the problem here)

[bentsherman]

This is actually a syntax error. In the config file it needs to be specified like this:

process {
  accelerator = [request: 4, type: 'nvidia-tesla-a100']
}

The new syntax parser should at least give a better error message about this

[puerta-amarilla]

Got it. The documentation is incorrect then:

https://www.nextflow.io/docs/latest/reference/process.html#process-accelerator

[puerta-amarilla]

That syntax didn't work either:

I put:
process {
executor = 'google-batch'
accelerator = [request: 4, type: 'nvidia-tesla-a100']
container = 'docker.io/nextflow/rnaseq-nf:v1.3.1'
}

Got this error:

Oct-04 19:51:22.242 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_84e0c0846989c3f9: /usr/local/google/home/jeffling/.nextflow/assets/nextflow-io/rnaseq-nf/modules/fastqc/main.nf
Script_9f1b934c039656c4: /usr/local/google/home/jeffling/.nextflow/assets/nextflow-io/rnaseq-nf/modules/multiqc/main.nf
Script_93ab1d6aaf1fee24: /usr/local/google/home/jeffling/.nextflow/assets/nextflow-io/rnaseq-nf/modules/quant/main.nf
Script_ddb63d1e49bbbd23: /usr/local/google/home/jeffling/.nextflow/assets/nextflow-io/rnaseq-nf/modules/index/main.nf
Script_ff16f4ec3543575c: /usr/local/google/home/jeffling/.nextflow/assets/nextflow-io/rnaseq-nf/modules/rnaseq.nf
Script_802cb7373e975c93: /usr/local/google/home/jeffling/.nextflow/assets/nextflow-io/rnaseq-nf/main.nf
Oct-04 19:51:22.243 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Oct-04 19:51:22.243 [main] DEBUG nextflow.Session - Session await
Oct-04 19:51:25.193 [Actor Thread 9] DEBUG nextflow.util.HashBuilder - Hash asset file sha-256: /usr/local/google/home/jeffling/.nextflow/assets/nextflow-io/rnaseq-nf/bin/fastqc.sh
Oct-04 19:51:29.618 [Task submitter] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
task: name=RNASEQ:INDEX (transcript); work-dir=gs://jeff_batch_nextflow_1/scratch/de/4a7b1bad668e82ced81dbd777a835d
error [com.google.api.gax.rpc.InternalException]: io.grpc.StatusRuntimeException: INTERNAL: http2 exception
Oct-04 19:51:29.630 [Task submitter] ERROR nextflow.processor.TaskProcessor - Error executing process > 'RNASEQ:INDEX (transcript)'

Caused by:
INTERNAL: http2 exception

com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL: http2 exception
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:110)
at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1130)

....

[bentsherman]

The example you linked is for a process definition -- note the different between a process definition and a process config block

This error looks like a low-level HTTP error, I can't tell if it's related to the accelerator request. Does it happen consistently? And only when you request an accelerator?

[puerta-amarilla]

Thanks for clarifying the configuration differences.

The error only happens when I configure an accelerator using this option. If I configure an accelerator through the machine type in Instance Template, and specify the instance template as follows, no error:

process {
executor = 'google-batch'
machineType = 'template://simple-template-1'
container = 'docker.io/nextflow/rnaseq-nf:v1.3.1'
}

[bentsherman]

I haven't been able to reproduce that http error yet. I do get this error:

$ nextflow run hello
...
ERROR ~ Error executing process > 'sayHello (1)'

Caused by:
  INVALID_ARGUMENT: Accelerator field is invalid. Accelerator with type nvidia-tesla-a100 should use a2 standard machine types. Please make sure that the configuration meets this requirement: https://cloud.google.com/compute/docs/gpus#a100-gpus.

That's just a limitation of the particular example from the docs. But it seems to work when I use an older gpu type