Build an async test runner with hooks, retries, and skips
An end-to-end suite talks to a real staging server. Setup is slow (create a database once, log in before every test), some checks are flaky and need a retry, and every step is async. You are building the small runner that drives it.
Implement createSuite(). Each call returns a new, independent suite:
const suite = createSuite();
suite.beforeAll(fn) // runs once, before the first test
suite.afterAll(fn) // runs once, after the last test
suite.beforeEach(fn) // runs before every attempt of every test
suite.afterEach(fn) // runs after every attempt of every test
suite.test(name, fn, options?) // options: { retries?: number, skip?: boolean }
await suite.run()
// => { passed, failed, skipped, results: Array<{ name, status, attempts, error? }> }
The methods work when destructured: const { test, beforeEach } = suite;.
run() does this, in order:
beforeAll hooks, in declaration order;afterAll hooks, in declaration order.beforeAll and afterAll hooks do not run either.beforeEach hook, then the test function, then every afterEach hook.beforeEach hook fails, the remaining beforeEach hooks and the test function are skipped.afterEach hooks always all run, even after a failure.retries (default 0) is how many extra attempts a failing test gets. A test with retries: 2 runs at most 3 times and stops at the first passing attempt.beforeAll and afterAll hooks are called with the suite context: one plain object, empty at the start of run().beforeEach hooks, test function and afterEach hooks are all called with it. Changes to it are never seen by other attempts or by the suite context.beforeAll hook fails, the remaining beforeAll hooks are skipped and no test function or each-hook runs. Every test that is not skipped fails with attempts: 0 and that hook’s error. The afterAll hooks still run.afterAll hook fails, the remaining afterAll hooks still run, and the results are not changed.run() returns a promise that always resolves. results has one entry per declared test, in declaration order:
name: the test’s name.status: 'passed', 'failed' or 'skipped'.attempts: how many attempts ran (0 for skipped tests).error: only on failed results, from the last attempt (or the failing beforeAll hook). Use the thrown value’s message if it is an Error, otherwise String(value). Other results have no error key.passed, failed and skipped count the results with each status. A skipped test (skip: true) never runs its function or any each-hook.
const suite = createSuite();
const log = [];
let flaky = 0;
suite.beforeAll(async (ctx) => { ctx.db = 'staging'; log.push('create db'); });
suite.beforeEach((ctx) => { ctx.user = 'mina'; log.push('login'); });
suite.afterEach(() => log.push('logout'));
suite.afterAll(() => log.push('drop db'));
suite.test('loads profile', async (ctx) => {
log.push(`profile ${ctx.user}@${ctx.db}`);
});
suite.test('syncs cart', async () => {
flaky++;
if (flaky < 2) throw new Error('503');
}, { retries: 1 });
suite.test('exports csv', () => {}, { skip: true });
await suite.run();
// => {
// passed: 2, failed: 0, skipped: 1,
// results: [
// { name: 'loads profile', status: 'passed', attempts: 1 },
// { name: 'syncs cart', status: 'passed', attempts: 2 },
// { name: 'exports csv', status: 'skipped', attempts: 0 },
// ],
// }
// log => ['create db', 'login', 'profile mina@staging', 'logout',
// 'login', 'logout', 'login', 'logout', 'drop db']