Author: Dann Corbit
Date: 13:46:22 08/15/05
Go up one level in this thread
On August 15, 2005 at 16:39:35, Uri Blass wrote: >On August 15, 2005 at 16:14:58, Robert Hyatt wrote: > >>On August 15, 2005 at 16:08:25, Uri Blass wrote: >> >>>On August 15, 2005 at 15:53:18, Robert Hyatt wrote: >>> >>>>On August 15, 2005 at 15:49:30, Uri Blass wrote: >>>> >>>>>I read the following comment: >>>>>"It's easy to broke things in a complex system like a chess program and it's >>>>>very dificult to discover bugs, they are likely not to be discovered with just a >>>>>few days of testing." >>>>> >>>>>My opinion is that it is not correct. >>>>> >>>>>There may be changes that it is a big risk to do at the last moment without >>>>>testing but at least some evaluation changes or implementing contempt factor can >>>>>be done with no risk by good programmers. >>>>> >>>>>part of the test may be playing blitz games and if implementing the contempt >>>>>factor does not cause problems in blitz games and give the expected result then >>>>>the programmer can be almost sure that there is no problem with the new code. >>>>> >>>>>There is no need to use more than some hours of testing to get hundred of 1+0 >>>>>time control games and if change in the evaluation did not cause problems in 1+0 >>>>>time control games it will probably not cause problems also in longer time >>>>>control(changes in the search seems to me something with bigger risk). >>>>> >>>>>Do you have examples that show that my opinion is wrong and changing things in >>>>>the last moment cause problems in games inspite of the fact that no problem were >>>>>discovered in tests. >>>>> >>>>>Note that I think that even changes in the move generator or in the search that >>>>>seem more risky to me have probability of less than 1% to create problems if few >>>>>hours of testing do not discover bugs in case that the programmer has good tools >>>>>to detect bugs. >>>>> >>>>>What is your opinion? >>>>>Is it risky to make changes in the last moment or is it only a problem of >>>>>programmers who did not generate good tools to detect bugs and need to play many >>>>>games at long time control for that purpose. >>>>> >>>>>Uri >>>> >>>> >>>>Pick up _any_ good book on software engineering. Last minute changes are >>>>_always_ a bad idea, because some percentage of them will either introduce >>>>direct bugs, or contribute to unexpected bugs... >>>> >>>>Even if you are good enough to make changes that cause no problems 90% of the >>>>time, that remaining 10% will be more than enough to keep you out of first place >>>>every time... >>> >>>If you have probability of 90% not to create bug in the first place and >>>probability of 99% to discover the bug in case that it happens in a short time >>>thanks to good tools to detect bugs then practically the probability to have >>>buggy version is only 0.1% >> >>that is not what I said. I said that even if 90% of your changes don't have >>bugs, that last 10% is enough to kill you. So I am assuming that after you make >>changes, 90% of the time either there are no bugs, or you find them and fix them >>prior to competition. 10% of the time you don't know there is a bug. >> >>But feel free to program however you want. Computer chess lore is full of "last >>minute change blunders". >> >> >>> >>>I think that reading about bugs from changes in the last moment and tests that >>>were done to detect bugs that did not help may be interesting because maybe the >>>problem was simply not doing the correct tests. >>> >> >>That's always the case. But "doing the correct tests" is not a cut-and-dried >>exercise, otherwise no software would ever be released to production with bugs >>inside. > >There are clear tests that can be recommended and the question is what is the >probability for bugs in case that programmers do the relevant tests. > >I think that movei should have more tests to detect bugs. > >Based on looking at fruit's code I see that there is significant amount of code >in order to detect errors that is not in Movei that is checking no illogical >information in the data structure. > >I think that the main problem that I have from not having this type of code is >not discovering the fact that I have a bug(because in most cases I can find that >I have a bug thanks to crash in test suites or bad results in test suites) but >the fact that after knowing that I have a bug because of some new code I need a >long time to find the bug. > > >I have some code to detect bugs that as far as I know is not in fruit or other >programs and it is a function to calculate sum of perft in pgn file. > >The idea is to take a pgn file and calculate the sum of perft(k) on all the >positions in the pgn file. > >If I do a change in the move generator then the first thing that I test to >verify no bugs is running the function on k=1,k=2,k=3 to see if I get the >correct numbers. This idea is a very good idea. You could even format it as an assert() like Fabien does and then the debugger will break into the problem when it occurs.
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.