TestinGil - Agile SW Testing Consultancy

Today's the day. Fighting Flakiness webinar goes live in a few hours.
Seven suspects. Real code. One tool that does the detective work for you.

If you like spending hours debugging flaky tests, this is definitely not for you. But if you're like me?

Last chance to register:
https://us02web.zoom.us/webinar/register/8517677800361/WN_Wo6ezTHfR3Smru0VRXxsSg

I promised you a story. And there it is.

A real story, that happened at night, and has shaken our perception on reality.

Ok, it wasn't that dramatic. But, we were young, and didn't know that the universe has played a joke on our confidence.

But it is a cautionary tale about belief, trust, and a cleaning crew.
All the good ingredients for a flaky test discussion.

Check out the video: https://youtu.be/yE5hnvk2VT0

Want to get a handle on reality again? Join my free webinar — Modern API Testing: Fighting Flakiness. June 3rd, 5 days away.

Register: https://us02web.zoom.us/webinar/register/8517677800361/WN_Wo6ezTHfR3Smru0VRXxsSg

June 3rd, 3PM CET / 9AM EDT.

Flaky Tests? Check Your Time and Date Assumptions! our tests pass every morning. Every afternoon. Like clockwork.Then...

In my examples in unit testing class, I always show an expiration calculation code. Checking if the current date is before or after a set expiration date.

And I ask about edge cases. And the people look at the code, and discuss and they come up with interesting cases.

And they write tests and review them, and run them, and everyone's happy, because now we have tests that will cover us in case of a code change. They will alert us if the code no longer does what it intends to do.

And you look at those tests and you want to almost hug them.

Until one test starts getting iffy. It mostly passes. But sometimes, in the dark of night it fails. And you look at the code - it didn't change. And the test - didn't either. And you run it again, and it passes, of course.

The only problem, is that in the dark of night, dates tend to change. And that may happen as our test runs, although mostly it doesn't run around midnight.

But who thinks about time, at the time of writing the tests?

There's a reason we mock dates and times to test logic. We want control and consistency. But system tests - API, UI - they don't have that luxury. And when they run, even if we don't intend it, may have an impact on the result.

Let me turn this to you. What's your craziest time related testing story?

What happens when AI generate tests for you?

First, you feel joy - So much time saved on something I don't like to do.
Then, you feel dread - I need to read all that?
Then you say - tests look fine, ship it.

And most of them do, and they pass. Until they don't. And you run those pesky things again. And they pass.

Stupid idiot bot got me flaky tests.

I talk about assumptions we make when we write tests, which cause the flakiness. Now, bots don't assume, right? They are not biased. They were just trained on good tests by other people.

Who had their assumptions.

Yes, using AI to generate tests, comes with assumption baggage. Turns out, weeding out assumptions in tests doesn't go away, even if we didn't write them.

How do we do that? Join my webinar "Modern API Testing: Fighting Flakiness" on June 3rd, and find out. I'll also show you a tool I built that helps me do that.

Regardless who wrote the tests.

Register: https://us02web.zoom.us/webinar/register/8517677800361/WN_Wo6ezTHfR3Smru0VRXxsSg
June 3rd, 3PM CET / 9AM EDT for free.

What makes a flaky test flaky?

It's the surprise and the repetition. You're surprised to see it, and even more surprised when you run it again, and it passes.

Why the surprise? Because we expect order and consistency. We like that in our automation. That's what we have it in the first place.

Ok, but where does the surprise come from?

Think about it - we're surprised when we expect something, and get something else. In fact, we assume something is going to happen, and it doesn't.

I have a video coming up on Thursday, that tells a story about how my team was hit by a "no one will ever do that." Like unplug our CI server during a run.

Each test we write has assumptions baked in. We block out any possible interference, because - who would interfere here?

But then something interferes. Another test, another operation at the same time. And then we spend the afternoon digging through logs.

What's your "no one will ever do that" story?