There are two appropriate variations of the identical Product (e.g. database engine) that I need to examine to one another for a similar enter situation (SQL question).
The issues that I need to deal with at first is the instability and stochastic nature of the System the Product is operating on:
- CPU throttling,
- random context switching,
- third-party processes operating apart,
- reminiscence allocator web page faults,
- filesystem fragmentation (a minimum of when loading the binary in reminiscence),
- and so forth.
All of these things probably could also be dominated out by a superb probabilistic mannequin, however I do not know find out how to invent the great one.
Let me formulate the duty:
- approximate check imply time is 500ms, which is akin to a scientific error of the System,
- decide with likelihood p1 that the brand new model is quicker than outdated one,
- decide with likelihood p2 that the brand new model is slower than outdated one,
- run each variations on given situation as few instances as doable.
construct up such mannequin? Is it doable in runtime or the parameters needs to be tuned in simulations? cope with completely different Methods, i.e. if I run each of variations on completely different machines each time? (Articles, literature or scholar hyperlinks are welcome)
Additionally could also be are there different approaches for efficiency testing, like full CPU cycle depend, or different probabilistic approaches?
I need to run each of variations for not more than 7 instances for each. I’ve a formulae representing some Mannequin, like
bool NewFaster(new_run_time1, new_run_time2, …, old_run_time1, old_run_time2, …)
and it is confirmed that the outcome
true is true with likelihood 95%, and the outcome
false is true with likelihood 80%.