From all passing to 7 failed, 73 skipped

[DamirSvrtan]

Hey everyone,

I'm doing the Rails integration and until a month ago I was 99% compatible with the TCK (except the social login that we decided to skip for this release).

In the last month lots of stuff in the TCK stopped working.

I believe this commit caused those failures: 736b1ea

Since then, I've started getting:

Failed methods (hide)
 changeEndpointChangesAccountPasswordWhenPostingJson
  changeRendersFormForValidSptoken 
  cookieExpirationMatchesTokenTtl 
  refreshTokenAsAccessBearerTokenFails 
  refreshTokenAsAccessTokenFails 
  serverRefreshesAccessTokenWhenMissing 
Skipped methods (hide)
  forgotDoesNotHandleDelete 
  forgotDoesNotHandleJsonGet 
  forgotDoesNotHandlePut 
  forgotRedirectsToNextUriWhenPostingInvalidEmail 
  forgotRedirectsToNextUriWhenPostingValidEmail 
  forgotRendersForm 
  forgotRendersFormWithInvalidSptokenBanner 
  forgotSucceedsWhenPostingInvalidEmailJson 
  forgotSucceedsWhenPostingValidEmailJson 
  loginAccountDatetimePropertiesAreIso8601 
  loginDoesNotHandleDelete 
  loginDoesNotHandlePut 
  loginDoesNotRenderWrongStatusParameter 
  loginErrorsWithBadCredentialsJson 
  loginFormPreservesValuesOnPostback 
  loginFormShouldBeOrderedCorrectly 
  loginHtmlRendersErrorWithoutUsernameAndPassword
  loginJsonDoesNotHaveLinkedResources 
  loginRedirectsToNextUriOnSuccess 
  loginRendersCreatedMessage 
  loginRendersErrorOnFailure 
  loginRendersForgotMessage 
  loginRendersResetMessage 
  loginRendersUnverifiedMessage 
  loginRendersVerifiedMessage 
  loginServesHtmlForm 
  loginServesJsonViewModel 
  loginSetsCookiesHtml 
  loginSetsCookiesJson 
  loginSucceedsForJson 
  loginViewModelHasFields 
  loginWithEmailSucceeds 
  loginWithEmailSucceedsHtml 
  loginWithEmptyPasswordFails 
  loginWithEmptyStringFails 
  loginWithInvalidFacebookAccessTokenFails 
  loginWithUsernameSucceeds 
  loginWithUsernameSucceedsHtml 
  loginWithValidFacebookAccessTokenSucceeds 
  logoutDeletesCookiesHtml 
  logoutDeletesCookiesJson 
  logoutDoesNotHandleDelete 
  logoutDoesNotHandleGet 
  logoutDoesNotHandlePut 
  logoutRedirectsToNextUriOnSuccess 
  logoutRedirectsToNextUriOnUnauthenticatedRequest 
  logoutReturns200OKOnSuccess 
  logoutRevokesTokensAfterSuccessJson 
  logoutRevokesTokensHtml 
  logoutSucceedsOnUnauthenticatedJsonRequest 
  meFailsOnUnauthenticatedRequest 
  meWithBasicAuthReturnsJsonUser 
  meWithBearerAuthHasNoCacheHeaders 
  meWithBearerAuthReturnsJsonUser 
  meWithBearerAuthStripsLinkedResources 
  meWithCookieAuthHasNoCacheHeaders 
  meWithCookieAuthReturnsJsonUser 
  meWithCookieAuthStripsLinkedResources 
  oauthClientCredentialsGrantFailsWithoutAPISecret 
  oauthClientCredentialsGrantSucceeds 
  oauthDoesntSupportGet 
  oauthErrorsAreTransformedProperly 
  oauthErrorsOnEmptyRequestBody 
  oauthErrorsOnJsonRequestBody 
  oauthErrorsOnMissingGrantType 
  oauthErrorsOnUnsupportedGrantTypes 
  oauthPasswordGrantWithEmailSucceeds 
  oauthPasswordGrantWithUsernameSucceeds 
  oauthRefreshGrantFailsWithInvalidRefreshToken 
  oauthRefreshGrantWorksWithValidToken 
  testNoRedirectionStickinessHtml 
  unsignedAccessTokensShouldFail 

I've reverted to a commit before it and the only spec that's failing is testNoRedirectionStickinessHtml.

@edjiang @dogeared @omgitstom can you check this out?

I'm also on the stormpath slack, we can do more debugging over there.

[edjiang]

I haven't been following the mass of changes lately from the Java team, and it's definitely possible they made breaking changes in the TCK. I'll have to defer to @dogeared

[dogeared]

Let's setup a time after the conference I'm at to do a screenshare. We did make a lot of changes (and additions for stuff that wasn't represented).

The intention was to ensure that we wouldn't disturb other existing tests.

One thing that was introduced prior to our work that you may want to look at is that you must set a STORMPATH_APPLICATION_HREF now to run the tck tests.

[DamirSvrtan]

Hey @dogeared! I've got that env already because the Rails integration needs it.

I'm going on vacation today, we can hear from each other when I'm back two weeks from now!
I'll ping you. Thnx!

[edjiang]

This is also failing for @bretterer

[dogeared]

Just by way of follow up for @edjiang @DamirSvrtan @omgitstom - I setup a rails app following the instructions here: https://github.com/stormpath/stormpath-rails

And I ran the TCK against it and the results are 112 tests run, 4 failures, none skipped.

This was running the TCK from IntelliJ.

Now, when I first ran it from the command line, I was getting similar results that you indicated above:

STORMPATH_APPLICATION_HREF=<href> \
STORMPATH_TCK_WEBAPP_PORT=3000 \
mvn verify > /tmp/tck_results.txt

The failures were because I had the wrong ~/.stormpath/apiKey.properties set.

Once I changed the command line to this:

STORMPATH_API_KEY_FILE=<api key file> \
STORMPATH_APPLICATION_HREF=<href> \
STORMPATH_TCK_WEBAPP_PORT=3000 \
mvn verify > /tmp/tck_results.txt

Then, I got the same results as in IntelliJ - 4 failures.

The STORMPATH_API_KEY_FILE env variable is something new that we added as it is supported in the Java SDK. The intention, however, is that ~/.stormpath/apiKey.properties should take precedence if it is present. It's possible we messed something up here.

I tested locally by making sure I had a ~/.stormpath/apiKey.properties and not including theSTORMPATH_API_KEY_FILE parameter. I then got the proper results.

So, I think the root cause of the failures you saw is that the TCK is not finding the API key id and secret. Please shoot me the command line you have been using when you get back from your vacation.