ISTech Support Forum

ISTech Support Forum
http://www.istechforum.com/YaBB.pl
Evo-ERP and DBA Classic >> Sales >> Line Item Extended Cost Calculation
http://www.istechforum.com/YaBB.pl?num=1340733897

Message started by daletone on 06/26/12 at 11:04:57

Title: Line Item Extended Cost Calculation
Post by daletone on 06/26/12 at 11:04:57

We are looking for any insight into how the Extended Cost (BKARINVL.PEXT) is calculated.

In theory it seems pretty simple - multiply the quantity by the price and apply a discount. There is something else going on though as I will explain below:

10 pcs * $10.3076 with no discount = $103.08 (rounds up)
10 pcs * $10.3075 with no discount = $103.07 (rounds down)
10 pcs * $10.3074 with no discount = $103.07 (rounds down)

now it would seem that any total of 5 tenths of a cent or less would round down, but then we have:

50 pcs * $2.0615 with no discount = $103.08 (rounds up)

The theoretical extended cost is the same as one of the lines above but the result is different. This type of behavior is repeatable and happens in both DBA and Evo. Is there a system setting for decimal places that we should be adjusting?

Title: Re: Line Item Extended Cost Calculation
Post by daletone on 07/09/12 at 12:44:05

As it turns out this error is credited to the use of floating point numbers (which as one website put it "Using Floating point numbers for accounting is a sin").

As best as I can understand the calculation of the extended cost is somehow incorporating the use of a single precision floating point number in conversion or variable storage. Floating point numbers are good for storing rational data that has many long digits such as analog sensor values, but it becomes a burden when dealing with finite values such as dollars and cents. There are many websites that are dedicated to the topic of rounding errors and methods of rounding with floating point numbers.

This occurs for us very infrequently but it is still a shortcoming of the system that could catch the unsuspecting user at any time. Consider yourself warned.