Testing Floating Point Numbers Accuracy

This should be a no brainier for anyone who has been coding for awhile but computer math can be funky at times.  To demonstrate how accuracte a floating point number is I opened up my python interperter and ran this mathimatical gem.

>>> float(1/998001)
1.002003004005006e-06

As you can see python gives up pretty quickly calculating this number. Dividing 1 by 998001 will create a decimal sequence of every three digit number in order.

Now lets take a look a python’s decimal data type.

from decimal import Decimal
>>> Decimal(1/998001)
Decimal('0.000001002003004005005955287066314596255978131011943332850933074951171875')

Ok, the decimal length is bigger but it sequence turns to garbage at around the same point where the floating point number gave up. There are a few ways to improve your programming math but the given answer will still be bound by the constants of being computed by a computer.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s