Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 11 additions & 65 deletions docs/extensions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -351,63 +351,27 @@ With the flag `applyGlobalTimeoutToFixtures` you can control if the global timeo
The `@Retry` extensions can be used for flaky integration tests, where remote systems can fail sometimes.
By default it retries an iteration `3` times with `0` delay if either an `Exception` or `AssertionError` has been thrown, all this is configurable.
In addition, an optional `condition` closure can be used to determine if a feature should be retried.
It also provides special support for data driven features, offering to either retry all iterations or just the failing ones.
In its standard mode it only retries the feature method execution, but this can be changed using `mode` to
also run setup and cleanup on retries. Even in this mode, the retry is only triggered if the feature method is failing
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@leonard84 is this actually the intended and expected behavior?
That only expected failures in the feature method trigger a retry I mean.
I do some UI tests using Geb with Marathon driver and it happens sometimes on GHA that the application cannot be started and thus the driver throws an exception.

This is done in an iteration interceptor before calling proceed, the iteration interceptor is within the retry iteration interceptor due to using global retry extension. But as the error happens there and not during feature method invocation, it does not cause a retry.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the current behavior. I'm not opposed to letting it cover setup as well.

Though if I'm not mistaken, you'll have problems covering cleanup without the method executions appearing in the results.

in the expected way. If the setup or cleanup is failing, the test fails immediately.

[source,groovy]
----
class FlakyIntegrationSpec extends Specification {
@Retry
def retry3Times() { ... }

@Retry(count = 5)
def retry5Times() { ... }

@Retry(exceptions=[IOException])
def onlyRetryIOException() { ... }

@Retry(condition = { failure.message.contains('foo') })
def onlyRetryIfConditionOnFailureHolds() { ... }

@Retry(condition = { instance.field != null })
def onlyRetryIfConditionOnInstanceHolds() { ... }

@Retry
def retryFailingIterations() {
...
where:
data << sql.select()
}

@Retry(mode = Retry.Mode.FEATURE)
def retryWholeFeature() {
...
where:
data << sql.select()
}

@Retry(delay = 1000)
def retryAfter1000MsDelay() { ... }
}
include::{sourcedir}/extension/RetryDocSpec.groovy[tag=example-common]
include::{sourcedir}/extension/RetryDocSpec.groovy[tag=example-a]
----

Retries can also be applied to spec classes which has the same effect as applying it to each feature method that isn't
already annotated with {@code Retry}.
already annotated with `@Retry`.

[source,groovy]
----
@Retry
class FlakyIntegrationSpec extends Specification {
def "will be retried with config from class"() {
...
}
@Retry(count = 5)
def "will be retried using its own config"() {
...
}
}
include::{sourcedir}/extension/RetryDocSpec.groovy[tag=example-b1]
include::{sourcedir}/extension/RetryDocSpec.groovy[tag=example-common]
include::{sourcedir}/extension/RetryDocSpec.groovy[tag=example-b2]
----

A {@code @Retry} annotation that is declared on a spec class is applied to all features in all subclasses as well,
A `@Retry` annotation that is declared on a spec class is applied to all features in all subclasses as well,
unless a subclass declares its own annotation. If so, the retries defined in the subclass are applied to all feature
methods declared in the subclass as well as inherited ones.

Expand All @@ -416,25 +380,7 @@ Running `BarIntegrationSpec` will execute `inherited` and `bar` with two retries

[source,groovy]
----
@Retry(count = 1)
abstract class AbstractIntegrationSpec extends Specification {
def inherited() {
...
}
}

class FooIntegrationSpec extends AbstractIntegrationSpec {
def foo() {
...
}
}

@Retry(count = 2)
class BarIntegrationSpec extends AbstractIntegrationSpec {
def bar() {
...
}
}
include::{sourcedir}/extension/RetryDocSpec.groovy[tag=example-c]
----

Check https://github.com/spockframework/spock/blob/master/spock-specs/src/test/groovy/org/spockframework/smoke/extension/RetryFeatureExtensionSpec.groovy[RetryFeatureExtensionSpec] for more examples.
Expand Down
14 changes: 8 additions & 6 deletions spock-core/src/main/java/spock/lang/Retry.java
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@


/**
* Retries the given feature if an exception occurs during execution.
* Retries the given feature if an exception occurs during execution of the feature method.
*
* <p>Retries can be applied to feature methods and spec classes. Applying it to
* a spec class has the same effect as applying it to each feature method that
Expand All @@ -50,7 +50,7 @@
/**
* Configures which types of Exceptions should be retried.
*
* Subclasses are included if their parent class is listed.
* <p>Subclasses are included if their parent class is listed.
*
* @return array of Exception classes to retry.
*/
Expand All @@ -77,12 +77,12 @@ Class<? extends Throwable>[] skipRetryExceptions() default {
* Condition that is evaluated to decide whether the feature should be
* retried.
*
* The configured closure is called with a delegate of type
* <p>The configured closure is called with a delegate of type
* {@link org.spockframework.runtime.extension.builtin.RetryConditionContext}
* which provides access to the current exception and {@code Specification}
* instance.
*
* The feature is retried if the exception class passes the type check and the
* <p>The feature is retried if the exception class passes the type check and the
* specified condition holds true. If no condition is specified, only the type
* check is performed.
*
Expand Down Expand Up @@ -113,12 +113,14 @@ Class<? extends Throwable>[] skipRetryExceptions() default {

enum Mode {
/**
* Retry the iterations individually.
* Retry only the feature method execution, setup and cleanup are not running on retries.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we also the the same sentence as in the docu

Suggested change
* Retry only the feature method execution, setup and cleanup are not running on retries.
* Retry only the feature method execution, setup and cleanup are not running on retries.
* If the setup or cleanup is failing, the test fails immediately.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it makes sense here.

The sentence in the docs is about the SETUP_FEATURE_CLEANUP mode below and there I already added the sentence like in the docs.

In the ITERATION mode setup and cleanup are not part of the retry, so why should we additionally mention that the test fails if they are failing, they are not affected in any way by the retry anyway in that mode.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Besides that it is anyway questionable whether this is actually the intended behavior, thus my question above to @leonard84

*/
ITERATION,

/**
* Retry the feature together with the setup and cleanup methods.
* Retry the iteration together with setup and cleanup.
* Even in this mode, the retry is only triggered if the feature method is failing
* in the expected way. If the setup or cleanup is failing, the test fails immediately.
*/
SETUP_FEATURE_CLEANUP
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
package org.spockframework.docs.extension

import groovy.sql.Sql
import spock.lang.Retry
import spock.lang.Shared
import spock.lang.Specification

abstract
// tag::example-common[]
class FlakyIntegrationSpec extends Specification {
// end::example-common[]
@Shared
def sql = Sql.newInstance("jdbc:h2:mem:", "org.h2.Driver")
}

class FlakyIntegrationSpecA extends FlakyIntegrationSpec {
// tag::example-a[]
@Retry
def "retry 3 times"() {
expect: true
}

@Retry(count = 5)
def "retry 5 times"() {
expect: true
}

@Retry(exceptions = [IOException])
def "only retry on IOException"() {
expect: true
}

@Retry(condition = { failure.message.contains('foo') })
def "only retry if condition on failure holds"() {
expect: true
}

@Retry(condition = { instance.field != null })
def "only retry if condition on instance holds"() {
expect: true
}

@Retry
def "retry failing feature methods"() {
expect: true

where:
data << sql.execute('')
}

@Retry(mode = Retry.Mode.SETUP_FEATURE_CLEANUP)
def "retry with setup and cleanup"() {
expect: true

where:
data << sql.execute('')
}

@Retry(delay = 1000)
def "retry after 1000 ms delay"() {
expect: true
}
}
// end::example-a[]

// tag::example-b1[]
@Retry
// end::example-b1[]
class FlakyIntegrationSpecB extends FlakyIntegrationSpec {
// tag::example-b2[]
def "will be retried with config from class"() {
expect: true
}

@Retry(count = 5)
def "will be retried using its own config"() {
expect: true
}
}
// end::example-b2[]

// tag::example-c[]
@Retry(count = 1)
abstract class AbstractIntegrationSpec extends Specification {
def inherited() {
expect: true
}
}

class FooIntegrationSpec extends AbstractIntegrationSpec {
def foo() {
expect: true
}
}

@Retry(count = 2)
class BarIntegrationSpec extends AbstractIntegrationSpec {
def bar() {
expect: true
}
}
// end::example-c[]
Loading