For clarification, I don't mean automated testing, test suites, or TDD. I mean more like HAL: "Dave, I have detected an error in one of my circuits." I.e., software that, when run in normal production mode, updates its test fixtures with the latest environmental changes, and then runs its test suite on itself. If it passes, then the full production mode continues.
This is an idea I've had which would be a really nice advance for my field: parsing and scraping public government legal texts. I'm wondering what others have done like this.
Unfortunately, when I search for "self-testing software" I get only _automated testing_ ... even Martin Fowler using self-testing as a synonym for it: https://martinfowler.com/bliki/SelfTestingCode.html.
Thanks!
----
Details for the curious:
I have the problem of false positives when (say) the Nevada legislature changes the format of a statute. My program completes without error, but the output isn't correct. (My take-away: I've allowed an illegal state to become representable, and my code needs to change.)
But I noticed something interesting: when I update my text fixtures (they're full copies of a few of the Nevada HTML input files), I get test failures! This is great --- no false positives here. And it's nice confirmation for me that I've got a good test suite. (I did write this code using TDD and was careful not to code ahead of my tests.)
And so I had a thought: I could automate this. When I run the app ( https://github.com/public-law/nevada-revised-statutes-parser ) for real, it can update these fixture files and re-run the test suite to see if it still holds up against any recent changes. This wouldn't be a 100% guarantee of correctness because the source texts may have changed in some way that happens to avoid my tests. But it definitely would have saved me in this situation --- instead of finding the error way down the line with the data already in use.
A few yeas ago I used SpamBayes. The program calculates two values independently:
* The "probability" that a message is spam
* The "probability" that a message is ham (not spam)
They aren't real probabilities and they don't add up to one. Usually you get one value that is close to zero and one that is close to one.
Sometimes you get two values that are close to zero and it means that the email is of an unknown type (and you must classify it so the program learns to classify similar messages) and sometimes you get two values that are close to one and it means that the email is of a weird type (and you must classify it so the program learns to classify similar messages).
So the programs "knows" when it is confused.