Bug 137883

Summary:

Many (most?) transitions tests are very flaky, frequently failing all at once

Product:

WebKit

Reporter:

Alexey Proskuryakov <ap>

Component:

Tools / Tests

Assignee:

Nobody <webkit-unassigned>

Status:

NEW

Severity:

Normal

CC:

aestes, cdumez, dino, jonlee, rniwa, simon.fraser, webkit-bug-importer

Priority:

Keywords:

InRadar

Version:

528+ (Nightly build)

Hardware:

Unspecified

OS:

Unspecified

Bug Depends on:

58673, 114182, 114189, 114194, 114198, 114203, 114206

Bug Blocks:

Attachments:

Description	Flags
tests.txt	none

Alexey Proskuryakov

Reported 2014-10-20 09:37:44 PDT

These tests are very very flaky, and have been for quite a while. Time to start ignoring them. transitions/background-transitions.html [ Failure Pass ] transitions/border-radius-transition.html [ Failure Pass ] transitions/change-values-during-transition.html [ Failure Pass ] transitions/clip-transition.html [ Failure Pass ] transitions/color-transition-all.html [ Failure Pass ] transitions/color-transition-premultiplied.html [ Failure Pass ] transitions/color-transition-rounding.html [ Failure Pass ] transitions/created-while-suspended.html [ Failure Pass ] @@ -1,3 +1,5 @@ -PASS - "background-position" property for "box" element at 0.5s saw something close to: 10,10 -PASS - "-webkit-background-size" property for "box2" element at 0.5s saw something close to: 20,20 +CONSOLE MESSAGE: line 240: Failed to pause 'background-position' transition on element 'box' +CONSOLE MESSAGE: line 240: Failed to pause '-webkit-background-size' transition on element 'box2' +FAIL - "background-position" property for "box" element at 0.5s expected: 10,10 but saw: 20,20 +FAIL - "-webkit-background-size" property for "box2" element at 0.5s expected: 20,20 but saw: 30,30

Attachments
tests.txt (259.48 KB, text/plain) 2014-11-14 14:57 PST, Alexey Proskuryakov	no flags	Details
View All Add attachment proposed patch, testcase, etc.

Alexey Proskuryakov

Comment 1 2014-10-20 09:40:58 PDT

Marked them as flaky in <http://trac.webkit.org/changeset/174886>.

Simon Fraser (smfr)

Comment 2 2014-10-20 09:46:39 PDT

Would be nice to understand why these 8 all flake at once. Maybe some earlier test leaves bad state around?

Alexey Proskuryakov

Comment 3 2014-10-20 10:06:10 PDT

> Would be nice to understand why these 8 all flake at once. It definitely would! > Maybe some earlier test leaves bad state around? I tried running tests in exactly the same order as a failing tester on a bot did, and couldn't reproduce the issue. So it must be something more involved.

Simon Fraser (smfr)

Comment 4 2014-10-20 10:17:54 PDT

It's also interesting that the same bot hits this only sporadically. Does EWS hit it more reliably?

Alexey Proskuryakov

Comment 5 2014-10-20 10:43:04 PDT

EWS hits this in release builds, while regular bots only do in debug.

Alexey Proskuryakov

Comment 6 2014-11-13 17:07:43 PST

Something changed, and now it's 30 at once that fail, not counting ones that were already marked as flaky. This happens on EWS (Mountain Lion WK2 release) and on regular bots (Yosemite WK2 release or debug). Many of these are already marked flaky, perhaps we should mark all the rest?

Alexey Proskuryakov

Comment 7 2014-11-13 17:08:11 PST

*** Bug 113227 has been marked as a duplicate of this bug. ***

Alexey Proskuryakov

Comment 8 2014-11-14 10:06:12 PST

Skipped a lot more in <http://trac.webkit.org/r176125>, because having 30+ flaky failures breaks EWS. We should fix EWS (filed bug 138743), and we should fix this bug, but right now, marking the tests as flaky is the only way out. https://build.webkit.org/results/Apple%20Yosemite%20Debug%20WK2%20(Tests)/r176079%20(479)/results.html is an example of this problem occurring on a regular regression test bot. Looks like it's not only transitions tests that are affected, and this is a general timer problem.

Alexey Proskuryakov

Comment 9 2014-11-14 14:44:56 PST

This happens very rarely and is thus difficult to pinpoint. But looks like this started to happen for more tests about two weeks ago. Could it be <http://trac.webkit.org/changeset/175441>?

Alexey Proskuryakov

Comment 10 2014-11-14 14:57:04 PST

Created attachment 241628 [details] tests.txt This is the sequence of tests that a single worker process on a bot ran when this happened recently. But I cannot reproduce when re-running in the same order locally.

Alexey Proskuryakov

Comment 11 2015-07-25 19:21:19 PDT

This still happens, so we still can't have any test coverage for transitions. How fun is that?

Radar WebKit Bug Importer

Comment 12 2017-09-08 16:43:06 PDT

<rdar://problem/34342774>

Alexey Proskuryakov

Comment 13 2017-09-11 09:06:10 PDT

The original issue with 8 tests failing all at once doesn't seem to be happening any more. But we have many more transition tests marked as flaky now, and a few of them still are flaky. And a few are marked as flaky, but fail 100% of time. This will take some effort to clean up.

Note You need to log in before you can comment on or make changes to this bug.