|
Go
 |
New
|
Search
|
Notify
|
Tools
|
Reply
|
|
|
Platinum Member |
In FOCUS for VMS, I have two P15.4 fields that need to be compared to see if they are equal. What's the best way to round them up to 2 decimal places for this comparison?
For example, if FIELD_A = 200.2149 and FIELD_B = 200.2200, then after the rounding they should be considered equal with both fields having the value of 200.22. Thank you, John This message has been edited. Last edited by: JohnB, WF 7.6.5, Windows XP, HTML, Excel, PDF |
||
|
|
Member |
Try this and then you can review the Tech Memo's
DEFINE FILE EMPDATA FIELD_A/P15.4 = 200.2149 ; FIELD_B/P15.4 = 200.2200; FIELD_1/P15.2 = FIELD_A + .0001 ; FIELD_2/P15.2 = FIELD_B ; END TABLE FILE EMPDATA PRINT LN FIELD_A FIELD_B FIELD_1 FIELD_2 WHERE FIELD_1 EQ FIELD_2 END See these Tech Memo's http://techsupport.informationbuilders.com/tech/ibm/ibm_tmo_TM7815_p1.html http://techsupport.informationbuilders.com/tech/ibm/ibm_tmo_TM7815_p1.htm2 |
|||
|
|
Master |
I think the best and most certain way to do this is to:
1. add a small number to the field(s), to be sure it rounds to the right number. 2. convert the field(s) to alpha, using FTOA and the format of '(D15.2)'. 3. do the equality check on these converted alpha field(s). Hope this helps.... GamP
|
|||||
|
|
Master |
(Dane's post)
Those two tech memos are specifically labeled Focus for Mainframe, and dated 1990, so I'd be wary of relying on them for other platforms. (GamP's) Even if, on a given platform, a calculated P field were held in memory and stored as D without rounding, the formatting of D fields for display (whether in a report or invoked by FTOA) rounds the value. Adding an epsilon is unnecessary, and could only bring harm. I'd recommend just your steps 2 and 3 to compare rounded character representations (or an eqivalent cast to an I field after scaling and rounding) - Jack Gross WF 7.6.7, Win |
|||
|
|
Platinum Member |
Dane,
The company I'm with has no tech support from IBI. Is there some way you can post those tech memos? GamP, I did something similar to your approach, and in testing it worked fine. In yesterday's production run, it was inconsistent. I'll try your approach. What happens when you do the following: FIELD_X/P15.2 = FIELD_A; (when FIELD_A is a P15.4)? Does packed round up correctly? WF 7.6.5, Windows XP, HTML, Excel, PDF |
|||
|
|
Virtuoso |
Play around with this code and you will see what happens when you alter SCALE4.
DEFINE FILE CAR SCALE4/P15.4=12345.6555; SCALE2/P15.2=SCALE4; END TABLE FILE CAR PRINT SCALE4 SCALE2 BY COUNTRY END It rounds up according to the rounding rules for P fields. Ginny --------------------------------- Prod: WF 7.6.5 with 7.6.6 WFRS; AIX 5.2; WebSphere 6.1.0.15 Dev: WF 7.6.5 with 7.6.6 WFRS; AIX 5.3; WebSphere 6.1.0.15 Primarily self-service; adapters: Teradata, DB2, Oracle, SQL Server, Essbase, ESRI, FlexEnable |
|||
|
|
Virtuoso |
Here is additional information:
Ginny --------------------------------- Prod: WF 7.6.5 with 7.6.6 WFRS; AIX 5.2; WebSphere 6.1.0.15 Dev: WF 7.6.5 with 7.6.6 WFRS; AIX 5.3; WebSphere 6.1.0.15 Primarily self-service; adapters: Teradata, DB2, Oracle, SQL Server, Essbase, ESRI, FlexEnable |
|||
|
|
Master |
Field_A/P15.2 rounded will equal 200.21, not 200.22. Packed fields round correctly. DEFINE FILE EMPDATA FIELD_A/P15.4 = 200.2149 ; FIELD_B/P15.4 = 200.2200; FIELD_C/P15.4 = 200.2150; FIELD_D/P15.4 = 200.2249; FIELD_E/P15.4 = 200.2301; FIELD_A1/P15.2 = FIELD_A ; FIELD_B1/P15.2 = FIELD_B ; FIELD_C1/P15.2 = FIELD_C ; FIELD_D1/P15.2 = FIELD_D ; FIELD_E1/P15.2 = FIELD_E ; END TABLE FILE EMPDATA PRINT LN FIELD_A FIELD_A1 FIELD_B FIELD_B1 FIELD_C FIELD_C1 FIELD_D FIELD_D1 FIELD_E FIELD_E1 IF RECORDLIMIT EQ 5 END -EXIT PAGE 1 LASTNAME FIELD_A FIELD_A1 FIELD_B FIELD_B1 FIELD_C FIELD_C1 FIELD_D FIELD_D1 FIELD_E FIELD_E1 VALINO 200.2149 200.21 200.2200 200.22 200.2150 200.22 200.2249 200.22 200.2301 200.23 BELLA 200.2149 200.21 200.2200 200.22 200.2150 200.22 200.2249 200.22 200.2301 200.23 CASSANOVA 200.2149 200.21 200.2200 200.22 200.2150 200.22 200.2249 200.22 200.2301 200.23 ADAMS 200.2149 200.21 200.2200 200.22 200.2150 200.22 200.2249 200.22 200.2301 200.23 ADDAMS 200.2149 200.21 200.2200 200.22 200.2150 200.22 200.2249 200.22 200.2301 200.23 Tom Flynn WebFOCUS 5.2.2 thru 7.6.x Windows, Unix, MVS |
|||
|
|
Virtuoso |
TOM is correct. 200.2149 will never round to 200.22 in a single operation. Here is a two stepper that works:
DEFINE FILE CAR SCALE4/P15.4=200.2149; SCALE3/P15.3=SCALE4; SCALE2/P15.2=SCALE3; END TABLE FILE CAR PRINT SCALE4 SCALE2 SCALE3 BY COUNTRY END Ginny --------------------------------- Prod: WF 7.6.5 with 7.6.6 WFRS; AIX 5.2; WebSphere 6.1.0.15 Dev: WF 7.6.5 with 7.6.6 WFRS; AIX 5.3; WebSphere 6.1.0.15 Primarily self-service; adapters: Teradata, DB2, Oracle, SQL Server, Essbase, ESRI, FlexEnable |
|||
|
|
Platinum Member |
Thank you all for the info. I will use your two-stepper, Ginny.
I am wondering if there's anything else with these fields that could affect the accuracy. They start out as alpha fields before becoming packed: FIELD_A_1/D15.4 = ATODBL(FIELD_IN_DATA_FILE, '14', FIELD_A_1); FIELD_A_2/P15.4 = FIELD_A_1; Could this conversion from decimal to packed cause any inaccuracies that would affect the rounding? This message has been edited. Last edited by: JohnB, WF 7.6.5, Windows XP, HTML, Excel, PDF |
|||
|
| Previous Topic | Next Topic | powered by eve community |
 | Please Wait. Your request is being processed... |
|
|||
| Site | Focal Point | ||
| Servlet | eve2da016 | ||
| Version | 1.2.22 build 9062 | ||
| Module | Forums 4.0.3 | ||
| Stylesheet | "ibi" | ||
| Wordlet Set | "Default Wordlet Set" | ||
|
Copyright © 1996-2008 Information Builders, leaders in enterprise business intelligence.