Building Cypress Tests can be easy and fun, but debugging the failing test sometimes might take a bit. To support QA and Engineers in your company it'd be great to get easy-to-read repro steps of when the test fails. In this article I'll guide you through setting Custom Errors for Cypress.
Building a Cypress Test is a lot of fun especially if you end up creating a comprehensive test that will ultimately run on CI on every development iteration. But the work of a QA Engineer or a Software Engineer is not easy and fun all the time. I can't count the number of hours I've spent trying to figure out why something is not working. The good part has always been that I had automation tests running in CI that notified me before I pushed my code to production. But usually the time between "something failed on CI" and actually realising how to reproduce those steps was long, or at least longer than I expected it to be. I had to spend 10 minutes on checking CI artifacts for screenshots, loading up local server on the feature branch, running the script locally, and coming up with repro steps for the Engineer to the bug that was found with one of the Cypress tests.
10 minutes doesn't sound that bad, right? Now multiply that by only 6 instances like that and my day is shorter by 1 hour. I knew there had to be a way to shorten things up. The thing that took the most of my time was of course finding those repro steps. Let's read the code I had for my test:
Alright, the code is straighforward in here, sure. But in reality our tests are much more complex! And if something goes wrong here we have to track down the steps by checking the running test vs the code and figuring out where exactly did it break.
First of all I like to have things nicely grouped in the test. So even if a Junior-Junior Engineer looks at that it would be clear what a specific section does. Let's re-format that code with comments and blocks and while we're at it let's make sure the test will fail by asserting incorrect pathname.
// cypress/integration/mytest.spec.js
describe('Custom Cypress errors', () => {
before(() => {
// Go to qacourse.dev
cy.visit('https://www.qacourse.dev')
})
it('Go to Articles from Homepage', () => {
// In the navigation get the Articles link
cy
.get('nav[role="navigation"]')
.contains('Articles')
.as('articleslink')
// Assert the link is visible
cy
.get('@articleslink')
.should('be.visible')
// Click on the link
cy
.get('@articleslink')
.click()
// Assert the url path has changed to /articles
cy.location().should(loc => {
expect(loc.pathname).to.eq('/articles123')
})
})
})
Yeap, much more clear. However it still doesn't give us repro steps in case stuff fails, right...? Here's how the default Cypress Error looks.
And sure it does show the place in code it failed, but we still don't get repro steps immediately. Let's work on that right now!
New command - cy.step('...')
Let's open cypress\support\commands.js file which is the home for any custom commands we'd like to have and use in our Cypress environment. Those commands can then be called by chaining from cy.myCustomCommand(). We can add the following code there - I'll also add comments here and there to explain what each part is doing.
// cypress/support/commands.js
// New command will be called using cy.step('...')
// and whatever is passed inside the function will
// ultimately be the description of a step.
Cypress.Commands.add('step', description => {
// maximum amount of steps to hold
const MAX_ITEMS_IN_STACK = 5
// Cypress.env('step') will contain an array
// of steps made so far so we get the list
// of what was done in order. Let's store that
// in a constant and make it an empty array if it
// doesn't exist yet.
const arr = Cypress.env('step') || []
// Let's add our new step to the steps stack
arr.push(description)
// If we have more steps than the max we want to store
// let's shift an array which removes first element
// which happens to be the oldest step.
if (arr.length > MAX_ITEMS_IN_STACK) {
arr.shift()
}
// Now we can store the steps back in the Cypress.env()
// so it's accessible everywhere in the test run.
Cypress.env('step', arr)
})
Update test file
Now that we have that command, let's actually convert all of our comments we made in the test file into the cy.step() function.
// cypress/integration/mytest.spec.js
describe('Custom Cypress errors', () => {
before(() => {
cy.step('Go to qacourse.dev')
cy.visit('https://www.qacourse.dev')
})
it('Go to Articles from Homepage', () => {
cy.step('In the navigation get the Articles link')
cy
.get('nav[role="navigation"]')
.contains('Articles')
.as('articleslink')
cy.step('Assert the link is visible')
cy
.get('@articleslink')
.should('be.visible')
cy.step('Click on the link')
cy
.get('@articleslink')
.click()
cy.step('Assert the url path has changed to /articles')
cy.location().should(loc => {
expect(loc.pathname).to.eq('/articles123')
})
})
})
Custom Error
Nice! But we don't actually get those steps anywhere in case Cypress fails :( Well, we have to create the custom error function for that. Let's open cypress/support/index.js and add the following function first:
// cypress/support/index.js
// new function that takes 3 arguments
const createCustomErrorMessage = (error, steps, runnableObj) => {
// Let's generate the list of steps from an array of strings
let lastSteps = "Last logged steps:\n"
steps.map((step, index) => {
lastSteps += `${index + 1}. ${step}\n`
})
// I decided to keep the following as an array
// for easier customization. But basically in the end
// I'll be building the text from the array by combining those
// and adding new line at the end
const messageArr = [
`Context: ${runnableObj.parent.title}`, // describe('...')
`Test: ${runnableObj.title}`, // it('...')
`----------`,
`${error.message}`, // actual Cypress error message
`\n${lastSteps}`, // additional empty line to get some space
// and the list of steps generated earlier
]
// Return the new custom error message
return messageArr.join('\n')
}
This function helped us generate that custom error message, but we still need a way to throw that error when Cypress fails. That's where the Cypress.on('fail', ...) function comes to play. Right below the new function we can add the following block of code:
// cypress/support/index.js
// When the test fails, run this function
Cypress.on('fail', (err, runnable) => {
// let's store the error message by creating it using our
// custom function we just made earlier. We need to pass
// "err" and "runnable" that we get from Cypress test fails
// but we have to remember to also pass our steps. In case
// no steps were provided we have to provide either empty string
// or some form of a message to help understand what's going on.
const customErrorMessage = createCustomErrorMessage(
err,
Cypress.env('step') || ['no steps provided...'],
runnable,
)
// Our custom error will now be defaulted back to the
// original default Cypress Error
const customError = err
// BUT we will change the message we're presenting to our custom one
customError.message = customErrorMessage
// aaaand let's throw that error nicely
throw customError
})
Done!
Aaaaand we're done :) Let's run the test that's supposed to fail now and see the error popping up with a list of repro steps. You can do many things from here, including exporting those steps as a JSON object using Cypress .writeFile() function and use that in CI to create a new ticket or comment on the PR, or - as I do - simply leave it like that. Engineers now can get immediate Repro steps of an error directly from the screenshot that's in the Artifacts on CI.
Me? I can grab my next coffee, take a sip, and work on another great automation that will make everyone's - including mine - lifes easier.
