#37768 - 11/10/11 10:12 PM
Re: adding a float numbers
[Re: Raphael Lalonde Lefebvre]
|
OL Newbie
Registered: 11/10/11
Posts: 19
|
Raphael,
I have a variable defined as a measure. Here is some of the code- &field6:=trim(regionline(1.63,9.29,1.97,9.43)) If(&conditionA) &dollar1:=&dollar1+(strtofloat(&field6)) else &dollar2:=&dollar2+(strtofloat(&field6)) endif
&dollar2 is at 7.835 on the next record &field6 is 0.640. The value of should be 8.475, but instead &dollar2 is 8.47999. I am doing no multiplication.
Any ideas?
Thanks
|
Top
|
|
|
|
#37770 - 11/11/11 12:39 AM
Re: adding a float numbers
[Re: dwal]
|
OL Expert
Registered: 08/25/00
Posts: 354
Loc: Objectif Lune Montréal
|
Are you trying to use the measure type to make currency calculations? That's a VERY bad idea. There is another data type called Currency that can be used for, well, dealing with currency or any fixed-point value. I very strongly suggest that you change your code to use the Currency type instead, or, even better, do currency calculations at the source if possible.
As you experienced, the measure type (really, a single-precision floating point number) does not round up as humans would expect. Furthermore, it only has 7-8 significan digits. That means that, for a measure, 700000000 is the same as 700000001. There is a reason why it is called a "measure", after all.
There are tons of web sites that explain the why's, do's and don'ts of floating point number, so I won't bother. But if you really want to dig, the Wikipedia article on floating point is a good start.
_________________________
Yannick Fortin, Team OL
|
Top
|
|
|
|
#37771 - 11/11/11 08:15 AM
Re: adding a float numbers
[Re: Yannick Fortin]
|
OL Newbie
Registered: 11/10/11
Posts: 19
|
I had started off using the currency format. However, my totals were low. I believed this to be the case as some of the amounts include a 1/2 a penny. For example, I was adding numbers like 0.515
Does the currency format allow me to add up 3 decimal places instead of just two?
|
Top
|
|
|
|
#37772 - 11/11/11 09:25 AM
Re: adding a float numbers
[Re: dwal]
|
OL Expert
Registered: 10/14/05
Posts: 4956
Loc: Objectif Lune Montreal
|
dwal, If I just use this code:
define(&dollar2, measure, 0.0)
define(&field6, measure, 0.0)
&dollar2 := 7.835
&field6 := 0.640
&dollar2:=&dollar2+&field6
show(floattostr(&dollar2))
And force the values at 7.835 and 0.640, I get the proper value, 8.475. I am thinking that the value you see in your database field "field6" might have a lot of digits after the decimal point, and it's having troubles rounding it as a result, for the reasons that Yannick explained. What's the actual value of the field? And currencies only allow two digits to be displayed. It's meant to display dollar amounts, which only have two digits. Regards, Raphaël Lalonde Lefebvre
|
Top
|
|
|
|
#37775 - 11/11/11 04:06 PM
Re: adding a float numbers
[Re: Raphael Lalonde Lefebvre]
|
OL Newbie
Registered: 11/10/11
Posts: 19
|
I am printing out the variable &field6 which is a string variable and I only have 3 digits after the decimal point. When print out the string variable &field6 the value is 0.640.
|
Top
|
|
|
|
|
|