[Build] Set general test timeout of 20min per test-project

[HannesWell]

In PDE we recently had issued with tests waiting forever which lead to builds that ran for a long time until they hit the general build timeout of multiple hours.
While this limit can of course be specified only in the PDE root pom, I think it makes sense to have a general limit by default. My proposal is to set it to 20min per-test project. This is more than sufficient for most test projects, which just run a few minutes and still would give relative fast feedback if something is flawed and hangs.

After a quick search I identified the following projects for which the limit might be too strict (in general or on slow computers):
org.eclipse.jdt.core.tests.compiler running ~45min
org.eclipse.jdt.core.tests.model running ~15min
org.eclipse.equinox.p2.tests running ~15min
org.eclipse.osgi.tests running ~10min

there we could just overwrite the general timeout and set it to a more suitable value by a similar property definition in the pom.xml or by using a corresponding pom.model.property.surefire.timeout entry in the build.properties (for already pomless projects).

What do the platform and Equinox commiters think?
@stephan-herrmann, @jarthana, @mpalat what do you think about this for JDT?

[stephan-herrmann]

For JDT I don't see a need to change anything, but if the end result is similar to the current situation then I have no objections, of course. In particular I don't have any data about how times per test project deviate from an average.

In case we get undesired timeouts for a particular build job, can the job override the timeout value of test projects?

[stephan-herrmann]

I'm seeing strange timeouts in jdt.core PR builds these days. Could those be related to changes in platform?

Have a look at https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/PR-4573/

Executions 2 and 3 claim to have hit a timeout after 90 min

In both cases things go wrong in this phase:

23:01:05  [INFO] --- tycho-packaging:5.0.1-SNAPSHOT:package-plugin (default-package-plugin) @ org.eclipse.jdt.core.tests.model ---
23:01:05  [INFO] Building jar: /home/jenkins/agent/workspace/eclipse.jdt.core-Github_PR-4573/org.eclipse.jdt.core.tests.model/target/org.eclipse.jdt.core.tests.model-3.13.600-SNAPSHOT.jar
23:02:02  
23:02:02  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-7-SelectorManager"
23:02:02  
23:02:02  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-8-SelectorManager"
23:02:02  
23:02:02  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-1-SelectorManager"
23:02:14  
23:02:14  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-10-SelectorManager"
23:02:22  
23:02:22  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-6-SelectorManager"
23:02:25  
23:02:25  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-9-SelectorManager"
23:02:26  
23:02:26  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-2-SelectorManager"
23:02:30  
23:02:30  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-0"
23:02:32  
23:02:32  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-5-SelectorManager"
23:02:37  
23:02:37  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-4-SelectorManager"
23:02:39  
23:02:39  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-1"
23:02:54  
23:02:54  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-3-SelectorManager"
23:03:06  
23:03:06  Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"
  Cancelling nested steps due to timeout
23:40:39  Sending interrupt signal to process
23:40:49  OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
23:40:49  OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
23:40:49  script returned exit code 143

I read this as

• at 23:02 we run out of memory (while building a simple jar, which shouldn't be a memory intensive operation)
• at 23:40 the VM still hasn't shut down, assumeably exception handlers have gone wild for 38 minutes
  • this is when SIGTERM is send at the OS level?
• the last lines look like another OOME occurred as a consequence of SIGTERM

So, perhaps timeout is the secondary thing here, but I haven't seen this kind of problem before, and now it happened in pretty much the same way twice in a row!

[stephan-herrmann]

So, perhaps timeout is the secondary thing here, but I haven't seen this kind of problem before, and now it happened in pretty much the same way twice in a row!

Right, this observation belongs to #3436

[HannesWell]

In case we get undesired timeouts for a particular build job, can the job override the timeout value of test projects?

Yes that's possible and in the linked PRs I've used that already to extend the timeout for the test projects listed above.
In general you have to set the forkedProcessTimeoutInSeconds property of the tycho-surefire-plugin to a suitable value.
This can be done by setting this in the configuration of that mojo in a pom.xml or by setting the surefire.timeout property.

The latter can also be done in the build.properties of a pomless projects, e.g using:
pom.model.property.surefire.timeout = 1200

In particular I don't have any data about how times per test project deviate from an average.

For the long running tests I used a value about 2-3 times of the runtimes I saw as a timeout.
We can always extend it, if that's too strict.