[Architecture] Thoughts on the Karma test runner...

steve at opendirective.com steve at opendirective.com
Tue May 31 10:55:06 UTC 2016



Tony,


I'm not sure this is any use as I'm behind on my testing strategy 


 I decided to go light and minimise mocking requirements by using FP as much as possible. the intention is to use light weight flexible tools which alow me to build up mocking etc without getting in the way. (like using brunch for build)


I decided to just use tape, browser-run and faucet for unit tests. Testling also looks interesting. 


 https://github.com/OpenDirective/spikes/blob/master/aws-cognito-user-pool-sync/package.json


For http endpoints, supertest looks good, based on superagent. Also nightwatchjs based on webdriver looks right for acceptance./ functional. Those together should give end to end assuming apis exist.


STEVE


Get Outlook for Android



From: Tony Atkins

Sent: Tuesday 31 May 11:17

Subject: [Architecture] Thoughts on the Karma test runner...

To: architecture at lists.gpii.net Architecture



Hi, All:



In our recent discussions around Testem, Antranig asked about the Karma test runner.  I briefly spiked working with this with a single QUnit test in one of my projects, and wanted to share what I'd learned.



First, and most importantly, their way of working is pretty heavily tailored to unit tests, and to Angular in particular.  Check their FAQ:



https://karma-runner.github.io/0.13/intro/faq.html



They specifically say that their focus is not end-to-end tests.  Although people have tried working around this with various runners, they do not support first-class mechanisms for starting your own test fixtures before running your tests and stopping them afterward.  This is a problem for much of our end-to-end testing, as we need to do things like spin up kettle/gpii.express/gpii.pouch instances before running our tests.



The second major problem causes real problems when testing client-side components:



https://github.com/karma-runner/karma-qunit/issues/18



Basically, they do not have a clean mechanism for loading HTML fixtures, or for loading a whole HTML file with its own separate includes.  They plan to provide minor relief for this, but only in the form of a single "context" HTML file common to all tests:



https://github.com/karma-runner/karma/issues/2151



I was hoping to work around this using the "test suites" function provided with infusion, but has thus far proved impossible to use that function with their runner.  Their runner loads its own much newer QUnit, which is not compatible with the improvements added in infusion:



https://issues.fluidproject.org/browse/FLUID-5911



Even if it were compatible, or if we manage to figure out a workaround, my impression is that we would be unwise to use Karma for anything but unit tests.  Their focus is simply not a good fit for the range of things we need to test.



By comparison, Testem makes some things (notably client side code coverage reports) a bit trickier, but exposes enough low-level controls that we can abstract out the complexity into a series of helper components.



The only other technology I have heard of that I would like to explore is Intern, which would give us better options in testing keyboard navigation and browser back/forward buttons.  It seems like it also has all of the low-level hooks we need to build our own fixtures before tests run, and to tear them down afterward, which is encouraging.



Anyway, just wanted to share what I'd learned while it was fresh.  Please comment, especially if you have experience with other tools that might help.



Cheers,




Tony


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gpii.net/pipermail/architecture/attachments/20160531/7b21b13b/attachment.html>


More information about the Architecture mailing list