YOUHANA NASEIM: Hello, everyone. How are you all today? My name is Youhana, and I’m here to talk to you about testing. In the business of testing, we are all in search of the next bug that’s about to happen. So I’m about to show you some bugs. Please, if you recognize them, raise your hand. So let’s start with the first bug. Can you see the bug? [Laughter] 

Yes. How about that actually here? This is actually a Java function. It’s very simple. It’s a binary search of a function that actually has a catastrophic bug in hiding that can cause pretty catastrophic problems in production. Can you spot the bug? 

Actually, if you look deep enough, there is a hidden index out-of-bound exception, or an overflow, in this method. If midvalue gets too large because high and low are too big, this could cause a buffer overflow and an index out of bound. There is a safer alternative to how to write this. These sort of arithmetic bugs are just sometimes very subtle and hard to spot as an engineer. 

In fact, during my time at Microsoft, one time I was attending a friend’s engagement party. My team called me and said, “Hey, we’re discovering an index out-of-bound exception in production. Can we get your permission to patch that bug?” And I said, “Sure. Go ahead and patch the bug. Are you confident with your fix?” And they said, “Yes. We’re 100% confident. This is good.” I said, “Okay. Great. Please go patch it.” Then 15 minutes later, they call and say, “Whoops. Our bug—our fix had a bug that’s also an arithmetic bug that we couldn’t spot.” I had to leave my friend’s engagement party. I basically spend the next eight hours on a call managing a high-severe incident. So, similar to bugs in nature, bugs in production are laying in hiding. They are lurking, and we need to go and find them. Sometimes they appear in the worst of times. 

Actually, this is a really bad problem. Fifty percent of our time as engineering teams is spent in preventing the next bug. We spend time in war rooms, hunting issues in production, just holding production together to make sure that the next bug doesn’t happen. On an average, engineers spend 52 minutes a day just coding and the rest of their time finding the next bug. 

For us in fintech, we can lose up to an average of $9,000 per minute of outages, according to Visa, up to $500,000 per hour. It can even get worse than that. Catastrophe is complex, and catastrophe is just around the corner. We are all one bug away from losing our business. Knight Capital Group had a bug in their trading system that erroneously started buying securities, costing them $140 million, losing their market-leading position, and they had to be sold to their competition. 

TSB Bank in England had a bug in their upgrade system resulting in them having to lock out their customers for a couple of weeks out of their accounts, losing them $150 million and churning 80,000 users. Catastrophe is just around the corner. 

So how do we deal with this catastrophe? Or how do we try to discover these issues? We use what I call Testing 1.0. If we test, we actually write some limited scenarios—scenario tests with unit tests or integration tests. We test in production. We rely on humans. We rely on defined scenarios and alarms that humans have come up with, and we rely on humans to do all of the job. 

That is actually not good enough. Testing 1.0 is failing us. It is not good enough in discovering the next catastrophe. We need to fix that. But let’s start by asking the question, why is it not good enough? So, if we pause for a moment and think about software and how complex it has become and try to imagine, what if we want to map every possible behavior our software could take into a tree? Call that the behavioral execution space—everything, every possible behavior, everything that could ever happen. That tree will be massive, probably the size of the universe. 

But for a moment, let’s imagine that tree is as small as the tree I’m showing on the screen. Some of the spots on this tree are bugs—behaviors that we don’t want, are not desired—and we need to discover them. So, with unit tests, they are easy to write sometimes if we write them. But they only cover specific spots on that tree. They only are limited to testing specific units and components in isolation. 

With integration tests, they might be stronger, but they only cover paths along the tree that we have thought about. And they are sometimes hard to write and hard to maintain because as our software evolves, it becomes harder to keep those up and running. If we do both, and we feel confident in our tests, and we deploy our stuff to production, our software gets overwhelmed because it hasn’t thought about a lot of things that could happen in production. 

So what do we do as an industry? We said, “Let us test in production.” Nothing is like production. It tests the full system and its dependencies. It’s manageable. We can pay billions of dollars in observability costs and complicate our deployment trains and do a lot of feature flags and manage it. But here’s my hot take. This is not good enough. Testing in production is not good enough. It’s not fast enough in finding issues. This is why we keep finding catastrophes in the software that we’re running. 

When was the last time you shipped something to only fail—three days later or a week later—to discover, “Aha, my fix, like the thing I did a week ago, was wrong.” Also, in production, we only find issues that we’re looking for if we have the right alarms or the right observability. I have shipped bugs where, like, were lurking there for months to discover, “Oh my god, we have been writing the wrong data.” It’s expensive. Rollout can get expensive. I have been stuck in rollout trains all the time. 

Managing production with complex instant management and expensive observability is a lot. This is why we spend 57% of our time as teams doing that. It’s also risky. We risk customers. We cause customer impact. And even if we do it all right and we find issues in production, it’s sometimes very hard to reproduce. I have worked with teams that had bugs in production that are very catastrophic but lingered in there for weeks and months because we just couldn’t reproduce it and don’t know what the heck is going on. 

So how about AI? AI is going to fix it, right? Well, here’s the thing. AI is not creative enough to think about how to redo testing. AI just does more of the same. It can generate more unit tests. It can generate more integration tests. Maybe it generates them better. Maybe it generates a lot more. But it’s not creative enough. It’s not good enough. It’s the same broken techniques. It cannot rethink testing. Maybe it will help us manage production. Maybe it will help us fix bugs. But it’s not creative enough to rethink testing. AI is just not going to solve it. So Testing 1.0 is very, very limited. 

What if we just step back as an industry and say, “Let us rethink testing.” Let us do this over. Let us build Testing 2.0. So what will be our desires? First, we would want that environment, that Testing 2.0, to be able to test our full system, like production. Two, we would want it to actually generate the tests itself, a large diverse set of tests that it has by itself rather than us thinking through it because we’re not good at it. Three, we actually want this environment to be faster than production, ahead of production, more hostile than production, not behind production. We want to be able to discover issues that are going to happen in production before we deploy, not after we deploy.

Fourth, we need it to be able to validate software expectations, like know what our software is expected to do and be able to detect, “Aha, this is not expected. This is a bug.” Finally is that whenever we find a bug, it’s able to reproduce that bug and help us fix it rather than us chasing smoke.

Okay. So here’s the secret I’m about to reveal. This is actually not science fiction. This is possible today. This is a top secret I’m going to share that we have done at our company, Antithesis, and other people have also embraced this kind of new testing technique. 

So I’m going to walk you through how to build that. There’s two ways to go about that. You can build it yourself if you code your software in a specific way, or you can rely on a platform like Antithesis, which democratizes that for everyone. Anybody could bring their software to be tested with Testing 2.0 with Antithesis. I’ll do that in five steps. 

Step number one, you want to run your software deterministically. You want to make your software behave the same way, given the same exact input. This will enable us to test a lot of tests at the same time, and will help us also replay issues as we go. So, if you’re doing this yourself, you can code your software in a way where you control determinism. You control randomness. You can make threading predictable. This is actually possible today with multiple libraries. Or you can rely on a platform like Antithesis. 

What we have done is that we built a deterministic hypervisor. What that does is that it’s able to run virtual machines deterministically. So, if you package your software in a form of containers and all your system in a form of containers and run that in our hypervisor, your software automatically becomes deterministic. And we’re able to do that for anybody. 

Step number two. We humans are not good at thinking through all the possible test cases. Let the computer do that. Let the machine generate your tests for you. If you’re doing it yourself, you can build a test generator, make it repeatable, run it in a loop. At Antithesis, what we have done is that we have built an intelligent software explorer. 

Remember the tree I showed you in the very beginning? The intelligent software explorer basically generates inputs and test cases that run through the software. Another thing it does is that we actually collect the output coming out of these runs and feed that back into the intelligent explorer so that it’s able to understand how much of the behavior we have covered. Can it expand that behavior? What other test cases could it do to provoke nodes that haven’t been covered in the tree to be able to maximize the coverage so that it’s able to uncover as many of the unknown states as possible?

Step number three, make it chaotic by default. Meaning what? I had a smart manager say once, “If something can go wrong, it will go wrong in production. It’s just a matter of time.” So when you’re building software, if you’re doing it yourself, you can rely on a technique called bugification, which basically states that as you’re writing code, you have a mindset of what can go wrong. So make that thing actually go wrong sometimes in your test runs. 

If you’re doing a call to another service, make that fail sometimes. If you’re doing a system call, make that return the wrong thing sometimes. In our deterministic hypervisor in our virtual machine, we have actually created a thing called fault injector, which basically makes that environment chaotic and hostile by default. So when your software is running in the simulation, it’s running in a hostile environment. Everything that can happen in production is actually happening in the simulation. But it’s happening faster, so we can find issues quicker. 

So things around network problems, latency, partitioning, bad nodes, hardware problems where sometimes the server crashes or sometimes the server disappears, other hostilities around resource contention like CPU starvation or threading problems are all happening all the time. So we are forcing error handling. We’re forcing dependencies to sometimes fail. We’re making the file system sometimes fail. And if you’re doing that yourself, you should do that, as well, because that makes sure that we can uncover issues sooner. Bad things that can happen in production are happening all the time—are happening with a lot more frequency inside the simulation. 

Step number four, we need to let the simulation or the environment know when to discern if there’s a bug or not. So for that, we’re going to rely on techniques and concepts that are defined in property-based testing. What does that mean? It means that instead of defining scenarios of inputs and outputs when you write unit tests and integration tests, you are actually defining what are the properties and the expectations of your software. 

I want to talk to you about two types of those properties. The first type is what I call system-level test properties. As we are architecting systems, we actually define what kind of requirements we want from them, what kind of correctness requirements, what kind of durability requirements, what kind of requirements. And we write those in documents. It’s time to codify that and define system-level properties, things like, “Hey, my database needs to be durable.” Or, “My server should never crash.” Or, “My API calls should never return 500 errors.” Or, “I shouldn’t be seeing exceptions.” Or, “If it’s successful to 100, okay. It needs to have this or that.” So system-level properties are things that are across the whole system. 

The second kind of properties I want to talk to you about is what I like to call inline test properties. All of us as engineers, as we are writing code, we actually have a state of mind of what do we expect the software to do. We sometimes capture that in comments or logs. What we’re basically saying is let’s elevate that. Let’s capture that in assertions or expectations. As you are coding, let us write some expectations of how do we expect the software to behave.

For example, if we go back to the function I talked about in the past, if we had a check in there that says mid has always been within the list bounds, we would have caught that bug if we ran through the simulation because we would have generated a test case where low and high are too large and caused an index-out-of-bound exception. So let us convert our comments and logs into smarter assertions and expectations. 

The second kind of inline expectations I want to talk to you about is what I call “sometimes conditions.” Since we are relying on the computer to generate the test cases and since we are—as we are writing code, sometimes we can think of what are the edge cases I want to make sure I test, things like I want to make sure that I test that the list is sometimes as large as INT_MAX. Or I want to test that my function will work if the item I’m looking for is at the edge of the array, either the beginning or the very end. 

So these are cases we have thought about as humans as we’re writing code and we want to test. Let us code that, as well, because what’s going to happen is that, as the computer is generating test cases and these expectations will pass, we will get an ease of mind that, yes, the computer has actually tested those. It gives us confidence.

Step number five—so if you’re doing it yourself, you can detect these assertion or expectation failures, or you can analyze the output of those to figure out when bugs happen. 

At Antithesis, what we have done is that, as your software is running inside the deterministic hypervisor, we’re collecting all the data. We’re collecting all the log information, all the behaviors the software has been taking. And we store that in the form of a database, and we call these events. And we run some analysis on these events and say, “Hey, have any of our runs detected violations in these either system-level properties or code-level properties?” These violations are actually things we didn’t expect, and these are bugs. 

Last step is support replays. Now, if we find a bug, we want to be able to replay this bug. If we have done step one correctly and made the software deterministic, this means that, if we replay the same exact inputs that cause these bugs, we’re going to be able to reproduce the bug. And that’s exactly what happens in Antithesis. We keep track of all the inputs that we have replayed through the software. And if we find a bug, with a click of a button, replay all of that again and find that bug 100% of the time. If you have done that correctly yourself, you’re able to reproduce that. 

So, Testing 2.0 is actually real. And it’s here. And it’s about testing your software inside a simulation. That simulation is running your full system. It’s relying on the compute to generate test cases, not the humans. It’s faster and more hostile than production, so we’re able to detect, to find issues that were going to happen in production and in the future now. It’s smart enough to be able to validate our expectations of the software, not just specific scenarios and symptoms. And lastly, it’s able to reproduce any bug we find deterministically 100% of the time. 

So has anybody actually used this? The answer is absolutely yes. The impacts of Testing 2.0 are profound. Testing is one of those things that, if you do correctly, accelerates everything about your software development cycle. It makes your coding better. You release with more confidence in everything. I want to tell you about four particular impacts we have seen for customers that have used this. 

One, since you have—in your code, you have added expectations on properties. Your code now is self-describing. It describes what’s expected from it. And it’s also self-defending. When was the last time you fixed a bug in component A, rolled out of production, and realized, “Ah, this other thing is broken because of my change. I did not know that. Let’s throw it back and try again and over and over and over.” No longer with this technique, because your system is self-fortifying itself. It’s constantly testing the whole system all the time and discovering the expectations all over the place. 

The second impact. You shift the discovery of incidents left. You are discovering incidents on high-severity issues before production in your simulation. Our friends at Ramp use this technique and are able to trigger high-impact complex-code paths in the simulation reliably, very, very frequently, and discover issues sooner because of this technique. Some things that fail in production once a month? They are able to provoke that in the simulation constantly a lot of the time. 

Our friends at Mongo, actually during a test period, where they are also relying on simulation testing for their database—and guess what? During a testing period, the simulation testing found 100 critical SEV0, SEV1 bugs in the software; 3x more than conventional testing. Seventy-seven out of those 100 were missed by Testing 1.0 that they actually do correctly and do a lot of. Because we can reproduce issues deterministically, it’s very easy to actually fix these bugs. 

Impact number three—boost your release confidence. Turso, for example, rewrote their SQLite database in Rust and were able to roll that out seamlessly without issues because they rely on simulation testing.

That all brings us to impact number four, which is, hey, we’re testing a lot better in a lower-stress environment, accelerating in our engineering teams 3x, like PingThings. Because we are rethinking how testing is done, we are able to accelerate our teams. 

Testing 2.0 helps us self-fortify our systems, find instances faster and sooner, boost release confidence and accelerate our teams. I actually want to show you this in action. This is a visualization of a simulation. Every horizontal line is a test case that’s generated. This is how it looks like with two of the faults that are being introduced in this environment. 

It’s pretty chaotic. This is actually what it looks like. There are more than 20 types of faults being generated because these are things that are going to happen in production, because catastrophe is complex. And catastrophe is just around the corner. If your software can survive the simulation, it can survive anything in production. It’s rock solid. 

Testing 2.0, or autonomous testing, is here and now. And it’s a new world that a lot of people are moving to. Thank you all for listening to me.