[CODE] Rounding packed fields

In FOCUS for VMS, I have two P15.4 fields that need to be compared to see if they are equal. What's the best way to round them up to 2 decimal places for this comparison?

For example, if FIELD_A = 200.2149 and FIELD_B = 200.2200, then after the rounding they should be considered equal with both fields having the value of 200.22.

Thank you,

John

This message has been edited. Last edited by: JohnB,

WF 7.7.03, Windows 7, HTML, Excel, PDF

Posts: 225 | Location: San Francisco Bay Area, California | Registered: October 26, 2006

<Dane>

posted

Try this and then you can review the Tech Memo's

DEFINE FILE EMPDATA
FIELD_A/P15.4 = 200.2149 ;
FIELD_B/P15.4 = 200.2200;
FIELD_1/P15.2 = FIELD_A + .0001 ;
FIELD_2/P15.2 = FIELD_B ;
END
TABLE FILE EMPDATA
PRINT LN
FIELD_A
FIELD_B
FIELD_1
FIELD_2
WHERE FIELD_1 EQ FIELD_2
END
See these Tech Memo's
http://techsupport.informationbuilders.com/tech/ibm/ibm_tmo_TM7815_p1.html
http://techsupport.informationbuilders.com/tech/ibm/ibm_tmo_TM7815_p1.htm2

GamP

Virtuoso

posted

Hide Post

I think the best and most certain way to do this is to:
1. add a small number to the field(s), to be sure it rounds to the right number.
2. convert the field(s) to alpha, using FTOA and the format of '(D15.2)'.
3. do the equality check on these converted alpha field(s).

Hope this helps....

GamP

- Using AS 8.2.01 on Windows 10 - IE11.
	in Focus since 1988

Posts: 1961 | Location: Netherlands | Registered: September 25, 2007

Ignored post by GamP posted

Show Post

j.gross

Virtuoso

posted

Hide Post

(Dane's post)
Those two tech memos are specifically labeled Focus for Mainframe, and dated 1990, so I'd be wary of relying on them for other platforms.

(GamP's)
Even if, on a given platform, a calculated P field were held in memory and stored as D without rounding, the formatting of D fields for display (whether in a report or invoked by FTOA) rounds the value. Adding an epsilon is unnecessary, and could only bring harm. I'd recommend just your steps 2 and 3 to compare rounded character representations (or an eqivalent cast to an I field after scaling and rounding)

- Jack Gross
WF through 8.1.05

Posts: 1925 | Location: NYC | In FOCUS since 1983 | Registered: January 11, 2005

Ignored post by j.gross posted

Show Post

JohnB

Platinum Member

posted

Hide Post

Dane,

The company I'm with has no tech support from IBI. Is there some way you can post those tech memos?

GamP,

I did something similar to your approach, and in testing it worked fine. In yesterday's production run, it was inconsistent. I'll try your approach.

What happens when you do the following: FIELD_X/P15.2 = FIELD_A; (when FIELD_A is a P15.4)? Does packed round up correctly?

WF 7.7.03, Windows 7, HTML, Excel, PDF

Posts: 225 | Location: San Francisco Bay Area, California | Registered: October 26, 2006

Ignored post by JohnB posted

Show Post

GinnyJakes

Expert

posted

Hide Post

Play around with this code and you will see what happens when you alter SCALE4.

DEFINE FILE CAR
SCALE4/P15.4=12345.6555;
SCALE2/P15.2=SCALE4;
END
TABLE FILE CAR
PRINT SCALE4 SCALE2
BY COUNTRY
END

It rounds up according to the rounding rules for P fields.

Ginny
---------------------------------
Prod: WF 7.7.01 Dev: WF 7.6.9-11
Admin, MRE,self-service; adapters: Teradata, DB2, Oracle, SQL Server, Essbase, ESRI, FlexEnable, Google

Posts: 2723 | Location: Ann Arbor, MI | Registered: April 05, 2006

Ignored post by GinnyJakes posted

Show Post

GinnyJakes

Expert

posted

Hide Post

Here is additional information:

quote:
Rounding in Packed Decimal Format: Format P
In packed-decimal format (format type P), each byte contains two digits, except the last byte, which contains a digit and the sign (D for negative numbers, C for positive). Packed fields are comparable to COBOL COMP-3.

Packed field values are rounded to the number of digits specified in the field format before they are stored.

When the number of decimal places input is greater than the number that can be stored, P field values are rounded first, then stored or displayed.

Packed fields are precisely accurate when sufficient decimal places are available to store values. Otherwise, since values are rounded before being stored, accuracy cannot be improved by increasing the number of digits displayed. For example, if 123.78 is input to a packed field with 1 decimal place, 123.8 is stored. If the field's format is then changed to P6.2 using a COMPUTE or DEFINE, 123.80 will be displayed. If the field's format is changed to P6.2 in the Master File, 12.38 is displayed.

Ginny
---------------------------------
Prod: WF 7.7.01 Dev: WF 7.6.9-11
Admin, MRE,self-service; adapters: Teradata, DB2, Oracle, SQL Server, Essbase, ESRI, FlexEnable, Google

Posts: 2723 | Location: Ann Arbor, MI | Registered: April 05, 2006

Ignored post by GinnyJakes posted

Show Post

Tom Flynn

Expert

posted

Hide Post

quote:
For example, if FIELD_A = 200.2149 and FIELD_B = 200.2200, then after the rounding they should be considered equal with both fields having the value of 200.22.

Field_A/P15.2 rounded will equal 200.21, not 200.22. Packed fields round correctly.

 
DEFINE FILE EMPDATA
FIELD_A/P15.4 = 200.2149 ;
FIELD_B/P15.4 = 200.2200;
FIELD_C/P15.4 = 200.2150;
FIELD_D/P15.4 = 200.2249;
FIELD_E/P15.4 = 200.2301;
FIELD_A1/P15.2 = FIELD_A ;
FIELD_B1/P15.2 = FIELD_B ;
FIELD_C1/P15.2 = FIELD_C ;
FIELD_D1/P15.2 = FIELD_D ;
FIELD_E1/P15.2 = FIELD_E ;
END
TABLE FILE EMPDATA
PRINT LN
FIELD_A
FIELD_A1
FIELD_B
FIELD_B1
FIELD_C
FIELD_C1
FIELD_D
FIELD_D1
FIELD_E
FIELD_E1
IF RECORDLIMIT EQ 5
END
-EXIT

 
PAGE 1 
 
LASTNAME   FIELD_A  FIELD_A1 FIELD_B   FIELD_B1 FIELD_C   FIELD_C1 FIELD_D   FIELD_D1 FIELD_E   FIELD_E1 
VALINO    200.2149  200.21   200.2200  200.22   200.2150  200.22   200.2249  200.22   200.2301  200.23 
BELLA     200.2149  200.21   200.2200  200.22   200.2150  200.22   200.2249  200.22   200.2301  200.23 
CASSANOVA 200.2149  200.21   200.2200  200.22   200.2150  200.22   200.2249  200.22   200.2301  200.23 
ADAMS     200.2149  200.21   200.2200  200.22   200.2150  200.22   200.2249  200.22   200.2301  200.23 
ADDAMS    200.2149  200.21   200.2200  200.22   200.2150  200.22   200.2249  200.22   200.2301  200.23

Tom Flynn
WebFOCUS 8.1.05 - PROD/QA
DB2 - AS400 - Mainframe

Posts: 1972 | Location: Centennial, CO | Registered: January 31, 2006

Ignored post by Tom Flynn posted

Show Post

GinnyJakes

Expert

posted

Hide Post

TOM is correct. 200.2149 will never round to 200.22 in a single operation. Here is a two stepper that works:

DEFINE FILE CAR
SCALE4/P15.4=200.2149;
SCALE3/P15.3=SCALE4;
SCALE2/P15.2=SCALE3;
END
TABLE FILE CAR
PRINT SCALE4 SCALE2 SCALE3
BY COUNTRY
END

Ginny
---------------------------------
Prod: WF 7.7.01 Dev: WF 7.6.9-11
Admin, MRE,self-service; adapters: Teradata, DB2, Oracle, SQL Server, Essbase, ESRI, FlexEnable, Google

Posts: 2723 | Location: Ann Arbor, MI | Registered: April 05, 2006

Ignored post by GinnyJakes posted

Show Post

JohnB

Platinum Member

posted

Hide Post

Thank you all for the info. I will use your two-stepper, Ginny.

I am wondering if there's anything else with these fields that could affect the accuracy. They start out as alpha fields before becoming packed:

  
   FIELD_A_1/D15.4 = ATODBL(FIELD_IN_DATA_FILE, '14', FIELD_A_1);
   FIELD_A_2/P15.4 = FIELD_A_1;

Could this conversion from decimal to packed cause any inaccuracies that would affect the rounding?

This message has been edited. Last edited by: JohnB,

WF 7.7.03, Windows 7, HTML, Excel, PDF

Posts: 225 | Location: San Francisco Bay Area, California | Registered: October 26, 2006

Ignored post by JohnB posted

Show Post

Please Wait. Your request is being processed...

Read-Only Topic

Focal Point

Focal Point Forums

WebFOCUS/FOCUS Forum on Focal Point

[CODE] Rounding packed fields

Focal Point | Privacy Statement | Terms of Service