From: Mike Woodworth <mike at divergentmedia dot com>
hey all,
i was tracking down a bug in a chunk of declares the other day, when
i found something which took me awhile to understand - i had written
a function to convert rb integers to fixed datatype. it read like this:
function int2fixed (i as integer) as integer
return i * &hFFFF
end
simple... easy to read... gave the wrong answers?? it appears &h is
returning a double, causing my int to be multiplied in double space
leading to rounding errors. when i switched it to use bitshiftleft
the problems went away.
BitShiftLeft? A smoking gun...
If you're trying to shift left 32 bits, then you don't want to
multiply by &hFFFF. You want to multiply by &h10000. i.e. you're off
by 1, possibly generating the appearance of round-off.
Paul Rodman
_______________________________________________
Unsubscribe or switch delivery mode:
<http://www.realsoftware.com/support/listmanager/>
Search the archives of this list here:
<http://support.realsoftware.com/listarchives/lists.html>
|