Re: [libreoffice-users] a little math problem with Calc

Hi,

alex sato schrieb:
> Hi,
>
> I have a problem with functions I made.
>
> Let's say I create the function:
>
> Function Test (n)
> dim r(3)
> r(0)=int(n)
> r(1)=n-r(0)
> r(2)=r(1)*100
> r(3)=int(r(2))
> Test = r
> End Function
>
> Then, in a empty spreadsheet take the following steps:
> -select A1:D1
> -press [F2]
> -Insert the formula =TEST(E1)
> -press [CTRL]+[SHIFT]+[ENTER]
> -insert 11,18 in E1 and see the result in cell D1
>
> I think the result should be 18 but I'm getting 17
> Where is the error?

It is the normal behavior of data type double. 11,18 has indefinite 
digits in dual system. So the stored value is cut somewhere. Because 
there are enough binary digits to round the dual number to 15 decimal 
digits, you do not notice it. But when you subtract the integral part, 
the others digits will shift left and new digits will be generated at 
the end, which does not represent the original value. The calculated 
difference becomes a little bit to small. After multiply with 100 you do 
not have 18,00 but 17,99... And that results in 17. The problem becomes 
more visible, when the integral part is larger.

You should not use Basic functions, but use Calc functions instead. They 
are often tweaked to avoid such problems. You can call Calc functions 
inside your Macro.

To get the first two decimal digits as integer value, it seems to be 
more accurate to use
        r(3)=(n * 100) mod 100
in your macro.

Kind regards
Regina



--
For unsubscribe instructions e-mail to: users+help@global.libreoffice.org
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/global/users/
All messages sent to this list will be publicly archived and cannot be deleted