Hello all,

I have a situation with undesired rounding in calculations using the D format.

My code is basically..

DEFINE FILE TEST
A/I4 = 3;
B/I4 = 9;
C/I4 = 6;
END

TABLE FILE TEST
PRINT
COMPUTE VAR1/D6.1 = (A / B) * 100;
COMPUTE VAR2/D6.1 = (A / C) * 100;

COMPUTE RESULT/D6.1 = VAR1 / VAR2;
.....
END

I'm using D6.1 because the IBI documentation said that when using the D format, the numbers and precision are stored to the max length the field will allow, without rounding, and that rounding occurs on display.

So if VAR1 was 1.69 and VAR2 was 5.86, I want the calculation to be on 1.6 and 5.8 . So I used the D6.1 because I thought that's what would happen. The result of the calculation should also be the same format. Before I even displayed RESULT, I did an FTOA and some other conversion so what was displayed was a non-rounded value.

I know that nothing worked right though because I did this:

PRINT
COMPUTE VAR1/D6.1 = (A / B) * 100;
COMPUTE VAR1TEST/D6.2 = (A / B) * 100;

COMPUTE VAR2/D6.1 = (A / C) * 100;
COMPUTE VAR2/D6.2 = (A / C) * 100;

COMPUTE RESULT/D6.1 = VAR1 / VAR2;
COMPUTE RESULT/D6.2 = VAR1TEST / VAR2TEST;

.....
END

I did the display conversion on both RESULTs before displaying and they were the same, and diving 5.86 by 1.69 is different than 5.8 by 1.6

Does this make sense? Does anyone have any ideas on how to be able to use one decimal place in my calculation without having to worry about rounding? I wish I could control rounding..

If you will do a SEARCH on the Forum...there are many discussions on rounding. Search on the ROUND OR ROUNDING.

Basically, use the P format instead of D format. Reasons and techniques are described in the many postsyou can find with the search utility as Prarie says.

Prairie request the way to go to get the information you seek. Darin's solution is the solution to use. I might also suggst subtracting a .05 from each number before the calculation.


ira
aix 533 wf 533(d) 537 (p)