daletone
Browser
Posts: 19
Gender: 
|
As it turns out this error is credited to the use of floating point numbers (which as one website put it "Using Floating point numbers for accounting is a sin").
As best as I can understand the calculation of the extended cost is somehow incorporating the use of a single precision floating point number in conversion or variable storage. Floating point numbers are good for storing rational data that has many long digits such as analog sensor values, but it becomes a burden when dealing with finite values such as dollars and cents. There are many websites that are dedicated to the topic of rounding errors and methods of rounding with floating point numbers.
This occurs for us very infrequently but it is still a shortcoming of the system that could catch the unsuspecting user at any time. Consider yourself warned.
|
Pages: 1
Line Item Extended Cost Calculation (Read 534 times)
