Investigate Visual Regression Testing

[arschmitz]

Look into visual regression testing using image comparison

For anyone looking for the unit testing issue that was moved to #35

[sfrisk]

Has anyone looked into this at all?

[jzaefferer]

I just looked at the two references. Apparently you write "tests" sass in both cases, so that you can test mathematical functions implemented in sass, where the test provides input variables and compares the output to other variables. Based on that these are only useful if this framework ends up with lots of non-trivial calculations that can be abstracted and tested separately. Otherwise, I guess, the declarative nature of CSS leaves little room for unit testing.

Maybe a better idea is to look what kind of testing other CSS frameworks are doing. We started with a list of frameworks, but I can't find that anymore.

[arschmitz]

@jzaefferer @scottgonzalez and I have a list in a google doc of everyone we attempted to contact if thats what you mean?
or there were a few lists in the old repo related to specific things. I can get any of these lists still.

[jzaefferer]

I'm just looking for a list of source repositories of various frameworks to see what testing they do.

[arschmitz]

I can do that ill post shortly

[arschmitz]

@kristoferjoseph do you have any experience with this? I know you did a lot of testing related work on topcoat.

Also this seems like it may tie into #23 as a way to ensure the things we claim in our design language are actually true after all pre-processing.

[kristoferjoseph]

Yes we did a lot of investigations into the unit testing of CSS.

After trying everything we could come up with we ended up doing very simple css file comparison tests. I would generate CSS then hand tune it until it was what we needed then used that as the expected output for the preprocessor. These test allowed us to factor out utility classes and know the output was still what we expected. It did become a chore to keep these tests running though. For instance when a bug was found that needed to be fixed in a utility we ended up having to update all components that used that utility, which is ideally what you want, just a lot of work. I always felt there was a way to automate these test updates, but never had the time to create it.

We looked into image comparison, but found that those tests were incredibly brittle and added a lot of maintenance overhead for our tiny team. To me tests like this would only be useful when there is a way to input a design file and export preliminary CSS for refactoring ( Sketch plugin? ). This would then allow you to refactor the CSS and know you were not introducing visual differences, OR better yet you would be aware of subtle differences and be able to decide if you could live with 1px shifts etc.

At the end of the day we felt that the most important tests were around performance. These tests elevated a feedback loop that helped inform our designs.

[markelog]

Do you want to test the SASS code or design representation itself?

If both or the second one then i can come up with couple ways to do the latter.

[arschmitz]

@markelog if i understand correctly

1. is testing that the math within sass is correct, which is what the frameworks i originally mentioned were about and my intention.

• I think would help stick to some of the thinkgs @kristoferjoseph mentioned in Design language #23 like proportions and radius etc... and could be potentially useful though i'm not sure how much of this there will actually be.

2.) Testing if something is visually correct or broken by a change to the css

• This would be super useful in general if there is a good way to do it i was not aware of a good one which I think is what @kristoferjoseph was saying his experience was also. So @markelog if you could come up with something for this that would be AWESOME

[markelog]

I'm talking about second item, specifically third paragraph of the @kristoferjoseph points.

Image comparison with render results and all that, i know at least one example of successful implementation of that technic for the components library.

There is quote a lot of tools for that now, one of them can even produce code coverage data for the CSS styles, excluded areas, smart diffs, cross-browser testing and stuff like that.

It would require you to store in the repo screenshots of the design for every element you have. But it's best way to do automatic testing for this kind of libraries that am aware of.

[arschmitz]

@markelog that sounds really really interesting. Personally id like to see us at least look into this.

[cvrebert]

I'd be very interested to hear what tools others recommend for screenshot regression testing on multiple browsers.
https://github.com/webdriverio/webdrivercss looked the most promising when I researched this recently. (Full data)

[kristoferjoseph]

+1 would feel really good to be able to finally put this problem to bed either way.

[markelog]

@cvrebert Wow, that's a nice list you got there.

WebDriverJS is pretty cool, although it has some issues, like:
doesn't have "smart" diff or doesn't account for outline or box-shadow properties.

I'm not gonna repeat what you already know, but will mention tool that has escaped from your list, which is pretty similar to WebDriverJS - https://github.com/bem/gemini

Apart from the what WebDriverJS is missing, it also has code coverage, exclusion feature, has GUI (which, as far is i can tell, is not same as https://github.com/webdriverio/WebDriverJS-adminpanel), has a "smart" diff (see https://github.com/SevInf/looks-same and https://github.com/bem/gemini for more info).

But, i'm not sure if it has viewport feature, but since this is the same team from #4, i'm pretty sure it could be added if needed (if it's not already exist that is).

I have an implementation example of that tool to the components library - https://github.com/bem/bem-components:
config - https://github.com/bem/bem-components/blob/v2/.gemini.yml
test example - https://github.com/bem/bem-components/blob/v2/gemini/button.gemini.js
screenshots example - https://github.com/bem/bem-components/tree/v2/gemini/screens/button