Author: Uri Blass
Date: 13:39:35 08/15/05
Go up one level in this thread
On August 15, 2005 at 16:14:58, Robert Hyatt wrote: >On August 15, 2005 at 16:08:25, Uri Blass wrote: > >>On August 15, 2005 at 15:53:18, Robert Hyatt wrote: >> >>>On August 15, 2005 at 15:49:30, Uri Blass wrote: >>> >>>>I read the following comment: >>>>"It's easy to broke things in a complex system like a chess program and it's >>>>very dificult to discover bugs, they are likely not to be discovered with just a >>>>few days of testing." >>>> >>>>My opinion is that it is not correct. >>>> >>>>There may be changes that it is a big risk to do at the last moment without >>>>testing but at least some evaluation changes or implementing contempt factor can >>>>be done with no risk by good programmers. >>>> >>>>part of the test may be playing blitz games and if implementing the contempt >>>>factor does not cause problems in blitz games and give the expected result then >>>>the programmer can be almost sure that there is no problem with the new code. >>>> >>>>There is no need to use more than some hours of testing to get hundred of 1+0 >>>>time control games and if change in the evaluation did not cause problems in 1+0 >>>>time control games it will probably not cause problems also in longer time >>>>control(changes in the search seems to me something with bigger risk). >>>> >>>>Do you have examples that show that my opinion is wrong and changing things in >>>>the last moment cause problems in games inspite of the fact that no problem were >>>>discovered in tests. >>>> >>>>Note that I think that even changes in the move generator or in the search that >>>>seem more risky to me have probability of less than 1% to create problems if few >>>>hours of testing do not discover bugs in case that the programmer has good tools >>>>to detect bugs. >>>> >>>>What is your opinion? >>>>Is it risky to make changes in the last moment or is it only a problem of >>>>programmers who did not generate good tools to detect bugs and need to play many >>>>games at long time control for that purpose. >>>> >>>>Uri >>> >>> >>>Pick up _any_ good book on software engineering. Last minute changes are >>>_always_ a bad idea, because some percentage of them will either introduce >>>direct bugs, or contribute to unexpected bugs... >>> >>>Even if you are good enough to make changes that cause no problems 90% of the >>>time, that remaining 10% will be more than enough to keep you out of first place >>>every time... >> >>If you have probability of 90% not to create bug in the first place and >>probability of 99% to discover the bug in case that it happens in a short time >>thanks to good tools to detect bugs then practically the probability to have >>buggy version is only 0.1% > >that is not what I said. I said that even if 90% of your changes don't have >bugs, that last 10% is enough to kill you. So I am assuming that after you make >changes, 90% of the time either there are no bugs, or you find them and fix them >prior to competition. 10% of the time you don't know there is a bug. > >But feel free to program however you want. Computer chess lore is full of "last >minute change blunders". > > >> >>I think that reading about bugs from changes in the last moment and tests that >>were done to detect bugs that did not help may be interesting because maybe the >>problem was simply not doing the correct tests. >> > >That's always the case. But "doing the correct tests" is not a cut-and-dried >exercise, otherwise no software would ever be released to production with bugs >inside. There are clear tests that can be recommended and the question is what is the probability for bugs in case that programmers do the relevant tests. I think that movei should have more tests to detect bugs. Based on looking at fruit's code I see that there is significant amount of code in order to detect errors that is not in Movei that is checking no illogical information in the data structure. I think that the main problem that I have from not having this type of code is not discovering the fact that I have a bug(because in most cases I can find that I have a bug thanks to crash in test suites or bad results in test suites) but the fact that after knowing that I have a bug because of some new code I need a long time to find the bug. I have some code to detect bugs that as far as I know is not in fruit or other programs and it is a function to calculate sum of perft in pgn file. The idea is to take a pgn file and calculate the sum of perft(k) on all the positions in the pgn file. If I do a change in the move generator then the first thing that I test to verify no bugs is running the function on k=1,k=2,k=3 to see if I get the correct numbers. Uri
This page took 0 seconds to execute
Last modified: Thu, 15 Apr 21 08:11:13 -0700
Current Computer Chess Club Forums at Talkchess. This site by Sean Mintz.