Home Search

DriveWorks Pro 21
Info: Floating Point Precision (KB13103003)

Send Feedback

Info: Floating Point Precision

This document discusses the way DriveWorks performs calculations, explains why sometimes you may see unexpected results, and provides tips on how they can be avoided.

Floating Point Precision

DriveWorks was designed in accordance to the IEEE Standard for Binary Floating-Point Arithmetic (IEEE 754). The standard defines how floating-point numbers are stored and calculated. The IEEE 754 standard is widely used because it allows-floating point numbers to be stored in a reasonable amount of space and calculations can occur relatively quickly.

Common Examples of Error Due to Floating Point Calculation

Example 1: Loss of Precision When Using Very Large Numbers

Variable NameValue
LargeNumberA1.2E+50
LargeNumberB1.2E+100
LargeNumberAplusLargeNumberB1.2E+100

The resulting value of LargeNumberAplusLargeNumberB is 1.2E+100, the same value as LargeNumberB. This is because DriveWorks stores 15 digits of precision. At least 100 digits of precision would be required to calculate the formula above.

Example 2: Loss of Precision When Using Very Small Numbers

Variable NameValue
SmallNumberA0.000123456789012345
SmallNumberB1
SmallNumberAplusSmallNumberB1.00012345678901

The resulting value in cell A1 is 1.00012345678901 instead of 1.000123456789012345. This is because DriveWorks stores 15 digits of precision. At least 19 digits of precision would be required to calculate the formula above.

Example 3: Repeating Binary Numbers

Many combinations of arithmetic operations on floating-point numbers may produce results that appear to be incorrect by very small amounts.

For example, the equation:

=1*(.5-.4-.1)

may be evaluated to the quantity (-2.78E-17), or -0.0000000000000000278 instead of 0.

This is due to the fact that the IEEE 754 standard requires numbers to be stored in binary format. Not all decimal numbers can be converted exactly to binary, as in the case of 0.1. The conversion caused the loss of precision.

Correcting Precision Errors

When working with numbers that calculate to more than 15 digits of precision it is necessary to obtain the required precision from the outset.

DriveWorks provides the following functions that will round numbers to a specified number of digits:


Knowledge Base Article Ref:KB13103003