# Decimal values and math - Inconsistent results

Posts: 216
In working on my home system I have my OTA channels stored as DOUBLEs in an array. When performing operations on them to get the multicast part of the channel I am getting unreliable results.
```DEFINE VARIABLE

VOLATILE DOUBLE tv_channels[] = { 2.2, 6.2, 6.3, 8.2, 8.3, 11.2, 19.2, 23.2 }

DEFINE_EVENT

BUTTON_EVENT[dvUI,0]
{
PUSH:
{
STACK_VAR INTEGER x
STACK_VAR INTEGER wholeNum
STACK_VAR INTEGER decimalTimesTen

STACK_VAR DOUBLE testVar
STACK_VAR DOUBLE decimal

FOR(x=1; x <= 8; x++)
{
testVar = tv_channels[x]

wholeNum = TYPE_CAST(testVar)

decimal = TYPE_CAST(testvar - wholeNum)
decimalTimesTen = TYPE_CAST(decimal * 10)

SEND_STRING 0, "'testVar    = ',FTOA(testVar)"
SEND_STRING 0, "'decimal    = ',FTOA(decimal)"
SEND_STRING 0, "'decimalX10 = ',ITOA(decimalTimesTen)"
}
}
}
```

When I run this code the ouput consistently produces the following results:
```Line      1 (16:14:45)::  testVar    = 2.2
Line      2 (16:14:45)::  decimal    = 0.2
Line      3 (16:14:45)::  decimalX10 = 2
Line      4 (16:14:45)::  testVar    = 6.2
Line      5 (16:14:45)::  decimal    = 0.2
Line      6 (16:14:45)::  decimalX10 = 1
Line      7 (16:14:45)::  testVar    = 6.3
Line      8 (16:14:45)::  decimal    = 0.3
Line      9 (16:14:45)::  decimalX10 = 3
Line     10 (16:14:45)::  testVar    = 8.2
Line     11 (16:14:45)::  decimal    = 0.2
Line     12 (16:14:45)::  decimalX10 = 1
Line     13 (16:14:45)::  testVar    = 8.3
Line     14 (16:14:45)::  decimal    = 0.3
Line     15 (16:14:45)::  decimalX10 = 3
Line     16 (16:14:45)::  testVar    = 11.2
Line     17 (16:14:45)::  decimal    = 0.2
Line     18 (16:14:45)::  decimalX10 = 1
Line     19 (16:14:45)::  testVar    = 19.2
Line     20 (16:14:45)::  decimal    = 0.200001
Line     21 (16:14:45)::  decimalX10 = 2
Line     22 (16:14:45)::  testVar    = 23.2
Line     23 (16:14:45)::  decimal    = 0.200001
Line     24 (16:14:45)::  decimalX10 = 2
```

Notice the when testVar is 2.2, 6.3 or 8.3 the result is exactly as expected but in the rest of the cases it is not. This wouldn't surprise me too much if I was working with floating point values of varying precision but this kind of inconsistency with numbers with a single decimal place has me stymied.

Anyone care to test my code and confirm/debunk my findings?

I'm leaning now towards storing my presets in a structure with the channel and the multicast values stored separately as integers. That will certainly avoid this problem but in the mean time this issue is of interest to me regardless of how I solve the problem.

• Posts: 4,368
I've never had much luck with floats or doubles. I tried your code with floats and double and both were identical results as your post. Since I've never had luck using floats/double with math I always multiple x 10 if I'm just looking at 1 place past the decimal, 100 for 2 places. Here's how I would deal with it:
```DEFINE_EVENT    //BUTTON_EVENT[vTEST,0]

BUTTON_EVENT[vTEST,0]

{
PUSH:
{
STACK_VAR INTEGER x
STACK_VAR INTEGER wholeNum
STACK_VAR INTEGER testVar
STACK_VAR INTEGER decimal

FOR(x=1; x <= 8; x++)
{
testVar = TYPE_CAST(float_test_channels[x] * 10);
wholeNum = testVar / 10;
decimal  = testVar % 10;

SEND_STRING 0, "'testValue  = ',FTOA(float_test_channels[x])"
SEND_STRING 0, "'testVar    = ',ITOA(testVar)"
SEND_STRING 0, "'wholeNum   = ',ITOA(wholeNum)"
SEND_STRING 0, "'decimal    = ',ITOA(decimal)"
}
}
}
```
```Line      4 (19:47:56)::  testValue  = 2.2
Line      5 (19:47:56)::  testVar    = 22
Line      6 (19:47:56)::  wholeNum   = 2
Line      7 (19:47:56)::  decimal    = 2
Line      8 (19:47:56)::  testValue  = 6.2
Line      9 (19:47:56)::  testVar    = 62
Line     10 (19:47:56)::  wholeNum   = 6
Line     11 (19:47:56)::  decimal    = 2
Line     12 (19:47:56)::  testValue  = 6.3
Line     13 (19:47:56)::  testVar    = 63
Line     14 (19:47:56)::  wholeNum   = 6
Line     15 (19:47:56)::  decimal    = 3
Line     16 (19:47:56)::  testValue  = 8.2
Line     17 (19:47:56)::  testVar    = 82
Line     18 (19:47:56)::  wholeNum   = 8
Line     19 (19:47:56)::  decimal    = 2
Line     20 (19:47:56)::  testValue  = 8.3
Line     21 (19:47:56)::  testVar    = 83
Line     22 (19:47:56)::  wholeNum   = 8
Line     23 (19:47:56)::  decimal    = 3
Line     24 (19:47:56)::  testValue  = 11.2
Line     25 (19:47:56)::  testVar    = 112
Line     26 (19:47:56)::  wholeNum   = 11
Line     27 (19:47:56)::  decimal    = 2
Line     28 (19:47:56)::  testValue  = 19.2
Line     29 (19:47:56)::  testVar    = 192
Line     30 (19:47:56)::  wholeNum   = 19
Line     31 (19:47:56)::  decimal    = 2
Line     32 (19:47:56)::  testValue  = 23.2
Line     33 (19:47:56)::  testVar    = 232
Line     34 (19:47:56)::  wholeNum   = 23
Line     35 (19:47:56)::  decimal    = 2
```
• Posts: 1,624
Not sure what you are trying to accomplish, but have you tried

decimalTimesTen = decimal * 10.0

decimalTimesTen = TYPE_CAST(decimal * 10)
Paul
• Posts: 966
What you are seeing are rounding errors in full force. This is to be expected when you are dealing with IEE754 floating point number, of which a double in NetLinx is. For a full rundown on why this happens have a read of what every computer scientist should know about floating-point arithmetic.

For your channels as the 'decimal' component is really just a convient way to represent the two parts of the channel address your idea of using a structure with two individual integer values sounds like a great solution.
• Posts: 1,624
PhreaK wrote: »
What you are seeing are rounding errors in full force. This is to be expected when you are dealing with IEE754 floating point number, of which a double in NetLinx is. For a full rundown on why this happens have a read of what every computer scientist should know about floating-point arithmetic.

I doubt that is happening here. He isn't going to see relative errors with these values. Good link though, all programmers run into this issue eventually.
Paul
• Posts: 216
a_riot42 wrote: »
Not sure what you are trying to accomplish, but have you tried

decimalTimesTen = decimal * 10.0

decimalTimesTen = TYPE_CAST(decimal * 10)
Paul

It will throw a compile error without the type cast. Keep in mind this is just code to see if I could get reliable results from floating point math.

Now I understand that there will be rounding errors when working with floating point numbers but I honestly expected numbers with a single decimal value to be usable when doing mathmatical operations on them.

The point of this post was to find out if floating point math in Netlinx is really that unusable. And, if so, to also give a heads up to anyone who hasn't been down that road from wasting their time trying to make it work.
• Posts: 966
Bigsquatch wrote: »
I honestly expected numbers with a single decimal value to be usable when doing mathmatical operations on them.
So does everyone when they first start dealing with floating point math. While these values can be discreetly represented in base 10 it is important to remember that computers will represent them in base 2. So when you store 0.2 in a double it is actually 0.200000000000000011102230246251565404236316680908203125. When you display it with ftoa(..) it will then round be rounded to 6 digits of precision, displaying 0.2. Similarly you cannot represt 1/3 in decimal, however you might approximate and display it as 0.333333.
• Posts: 1,624
Bigsquatch wrote: »
It will throw a compile error without the type cast. Keep in mind this is just code to see if I could get reliable results from floating point math.

Yes I see that you called the variable decimalTimesTen but defined it as an integer so that was confusing. Try declaring decimalTimesTen as a double as well and it will compile. You can cast it to an integer right before printing if needed or use ftoa instead.
Bigsquatch wrote: »
Now I understand that there will be rounding errors when working with floating point numbers but I honestly expected numbers with a single decimal value to be usable when doing mathmatical operations on them.

They are perfectly usable, you just have to be careful when converting from integers to floats/doubles and know how casting can result in data loss.
Bigsquatch wrote: »
The point of this post was to find out if floating point math in Netlinx is really that unusable. And, if so, to also give a heads up to anyone who hasn't been down that road from wasting their time trying to make it work.

Its certainly not a waste of time to understand floating point numbers and Netlinx works fine with them. But in the case of TV stations, I always use strings since there is no reason to use numbers.
Paul
• Posts: 2,908
a_riot42 wrote: »
But in the case of TV stations, I always use strings since there is no reason to use numbers.
I couldn't agree more.

More than likely, you'll need to wind up:
1) converting each digit into an ascii representation or some other kind of value to send to the tuner / display via serial
2) pulsing each digit individually in an IR sequence

Doing a simple atoi() on each character is by far easier (though maybe more expensive) than some round about mathematical equation, but then again - maybe not!
• Posts: 470
PhreaK wrote: »
What you are seeing are rounding errors in full force. This is to be expected when you are dealing with IEE754 floating point number, of which a double in NetLinx is. For a full rundown on why this happens have a read of what every computer scientist should know about floating-point arithmetic.

For your channels as the 'decimal' component is really just a convient way to represent the two parts of the channel address your idea of using a structure with two individual integer values sounds like a great solution.

I like this way of dealing with it if the values can be changed at runtime from a touchpanel. A structure with a major and minor component of the channel number will simplify things quite a bit.

The other option is to store all of the values as strings.
• Posts: 1,624
I use the dot notation since its so common for stations, but not as a decimal point, similar to the way IP addresses are notated. It just separates the major and minor portions of the string. Structures can't be passed to modules so I avoid them unless there is no alternative.
Paul
• Posts: 861
Is this for a TV settopbox?
If you need something like a major and minor number to pulse two IR channels, why not use the XCH command?

/me is probably missing something
• Posts: 216
I didn't do a very good job of writing my original post. The only thing I was really interested in was the inaccuracy of the mathematical operations. I'm not interested in explaining what I was trying to do because it's irrelevant.

I've stored and retrieved favorites different ways in dozens of different projects at my previous job doing residential AMX. Not once did I use floats and since this was my own system to do with as I please I thought "why not?" My thinking was that floats are the data type that most closely represents the data being stored. Now I have my answer.