Focal Point Banner
Community Center Education Summit Technical Support User Groups
Let's Get Social!

Facebook Twitter LinkedIn YouTube
Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [CLOSED] Decimal value getting truncated (789.56 is coming as 789.50)
Go
New
Search
Notify
Tools
Reply
  
[CLOSED] Decimal value getting truncated (789.56 is coming as 789.50)
 Login/Join
 
Member
posted
I am reading a decimal fields from a table but in the file(wp format) the decimal portion is getting rounded off.
if the data is 789.56 in the table it is coming as 789.50 in the file. Im not using any round off function, please suggest what could be the problem.

This message has been edited. Last edited by: <Kathryn Henning>,


Focus for Unix
Windows, All Outputs
 
Posts: 6 | Registered: January 04, 2013Reply With QuoteReport This Post
Virtuoso
posted Hide Post
Does this only happen in WP format? What about HTML or EXL2K?

Can you please show us the ACTUAL and USAGE clauses in your WF synonym for that field, as well as the piece of code you're using to display it?



Prod/Dev: WF Server 8008/Win 2008 - WF Client 8008/Win 2008 - Dev. Studio: 8008/Windows 7 - DBMS: Oracle 11g Rel 2
Test: Dev. Studio 8008 /Windows 7 (Local) Output:HTML, EXL2K.
 
Posts: 1533 | Registered: August 12, 2005Reply With QuoteReport This Post
Member
posted Hide Post
I am using only WP format.
Below is the code that I am using. In the below code when I see in the ddgp2.wp file the data is ok(783.38), but in the file ddgprs.wp it shows (783.30) which is not correct.

SET NODATA=0
TABLE FILE BLUSGREC
SUM
-*
GPRS_HOME_CHG/P14.2 OVER
GPRS_INT_ROAM_CHG/P14.2
-*
ACROSS REP_SORT AS ''
-*
WHERE REPORT_SEQ_NO EQ &BLCR_SEQ OR &BLHO_SEQ OR &UPCS_SEQ OR &RPR_SEQ
-*
ON TABLE SET PAGE NOPAGE
ON TABLE SAVE AS DDGP2 FORMAT WP
END
-RUN

FILEDEF DDMAS DISK ddgp2.mas ( RECFM V LRECL 100
-RUN

-WRITE DDMAS FILENAME=DDGP2 ,SUFFIX=FIX
-WRITE DDMAS SEGNAME=ONE
-WRITE DDMAS FIELDNAME=FILL1 , ,A2 ,A2 , $
-WRITE DDMAS FIELDNAME=TOT_NAME , ,A17 ,A17 , $
-WRITE DDMAS FIELDNAME=FILL1 , ,A1 ,A1 , $
-WRITE DDMAS FIELDNAME=BC_AMOUNT , ,D14.2M,A14 , $
-WRITE DDMAS FIELDNAME=FILL1 , ,A2 ,A2 , $
-WRITE DDMAS FIELDNAME=ST_AMOUNT , ,D14.2M,A14 , $
-WRITE DDMAS FIELDNAME=FILL1 , ,A2 ,A2 , $
-WRITE DDMAS FIELDNAME=HF_AMOUNT , ,D14.2M,A14 , $
-*
FILEDEF DDGP2 DISK ddgp2.wp
-RUN
-*
TABLE FILE DDGP2
PRINT *
COMPUTE LN_INDX1/I2=LN_INDX+1; NOPRINT
COMPUTE LN_INDX/I2=DECODE LN_INDX1 ( 1 2
2 5 ELSE 0 );
WHERE TOT_NAME NE ' ' AND TOT_NAME OMITS '-'
ON TABLE HOLD AS DDGPRS2
END
-RUN
JOIN CLEAR *
JOIN LN_INDX IN DDGPRS2 TO LN_INDX IN DDTHREGP AS J5
-*
DEFINE FILE DDGPRS2
TOT_NAME/A21=TOT_NAME;
CYC_DIFF/D17.2M=BC_AMOUNT-ST_AMOUNT;
CYC_DIFF_P/P7.2=IF ST_AMOUNT EQ 0 AND CYC_DIFF GT 0 THEN 100 ELSE
IF ST_AMOUNT EQ 0 THEN 0 ELSE (CYC_DIFF/ST_AMOUNT)*100;
HOB_DIFF/D17.2M=HF_AMOUNT+CYC_DIFF;
HOB_DIFF_P/P7.2=IF ST_AMOUNT EQ 0 AND HOB_DIFF GT 0 THEN 100 ELSE
IF ST_AMOUNT EQ 0 THEN 0 ELSE (HOB_DIFF/ST_AMOUNT)*100;
TH_SN/A1=IF TH_VALUE EQ 0 THEN ' ' ELSE
IF TH_VALUE LT ABS(HOB_DIFF_P) THEN '*' ELSE ' ' ;
END
TABLE FILE DDGPRS2
PRINT
LN_INDX AS '' IN 0
TOT_NAME AS '' IN +0
BC_AMOUNT AS '' IN +0
ST_AMOUNT AS '' IN +0
CYC_DIFF AS '' IN +0
CYC_DIFF_P AS '' IN +0
HF_AMOUNT AS '' IN +0
HOB_DIFF AS '' IN +0
HOB_DIFF_P AS '' IN +0
TH_SN AS '' IN +0

ON TABLE SET PAGE NOPAGE
ON TABLE SAVE AS DDGPRS FORMAT WP
END
SAVE AS DDCSVGP
-RUN
-*
JOIN CLEAR *
JOIN GR_IND WITH LN_INDX IN DDGPRS2 TO LN_INDX IN DDTHREGP AS J6
-*
DEFINE FILE DDGPRS2
GR_IND/I2=7;
TOT_NAME/A21='GRAND_TOTAL';
END
TABLE FILE DDGPRS2
SUM
FST.GR_IND AS '' IN 0
TOT_NAME AS '' IN +0
BC_AMOUNT AS '' IN +0
ST_AMOUNT AS '' IN +0
COMPUTE CYC_DIFF/D17.2M=BC_AMOUNT-ST_AMOUNT; AS '' IN +0
COMPUTE CYC_DIFF_P/P7.2=IF ST_AMOUNT EQ 0 AND CYC_DIFF GT 0 THEN 100 ELSE
IF ST_AMOUNT EQ 0 THEN 0 ELSE (CYC_DIFF/ST_AMOUNT)*100;
AS '' IN +0
HF_AMOUNT AS '' IN +0
COMPUTE HOB_DIFF/D17.2M=HF_AMOUNT+CYC_DIFF; AS '' IN +0
COMPUTE HOB_DIFF_P/P7.2=IF ST_AMOUNT EQ 0 AND HOB_DIFF GT 0 THEN 100 ELSE
IF ST_AMOUNT EQ 0 THEN 0 ELSE (HOB_DIFF/ST_AMOUNT)*100;
AS '' IN +0
COMPUTE TH_SN/A1=IF FST.TH_VALUE EQ 0 THEN ' ' ELSE
IF FST.TH_VALUE LT ABS(HOB_DIFF_P) THEN '*' ELSE ' ' ;
AS '' IN +0
-*
BY GR_IND NOPRINT
-*
ON TABLE SET PAGE NOPAGE
ON TABLE SET HOLDLIST PRINTONLY
ON TABLE SAVE AS DDGPRS FORMAT WP
END
SAVE AS DDCSVGP
-RUN
-*


Focus for Unix
Windows, All Outputs
 
Posts: 6 | Registered: January 04, 2013Reply With QuoteReport This Post
Virtuoso
posted Hide Post
Try using the D format instead of P.
P has different rounding behaviour, probably due to how it gets packed.


WebFOCUS 8.1.03, Windows 7-64/2008-64, IBM DB2/400, Oracle 11g & RDB, MS SQL-Server 2005, SAP, PostgreSQL 11, Output: HTML, PDF, Excel 2010
: Member of User Group Benelux :
 
Posts: 1667 | Location: Enschede, Netherlands | Registered: August 12, 2010Reply With QuoteReport This Post
Member
posted Hide Post
After making the below changes I am getting very weird results. $783.30 value is coming as $78.00.

GPRS_HOME_CHG/D13.2 OVER
GPRS_INT_ROAM_CHG/D13.2


Focus for Unix
Windows, All Outputs
 
Posts: 6 | Registered: January 04, 2013Reply With QuoteReport This Post
Virtuoso
posted Hide Post
Isn't that because you're writing a D13.2 to your WP file, but are defining it as D14.2M in your master?

Why are you using WP files for temporary data anyway? If you'd use binary or FOCUS format HOLD files, the master gets generated for you.
I suspect that those formats store the data more efficiently (both performance and size) as well, but I'm not sufficiently familiar with the WP format.


WebFOCUS 8.1.03, Windows 7-64/2008-64, IBM DB2/400, Oracle 11g & RDB, MS SQL-Server 2005, SAP, PostgreSQL 11, Output: HTML, PDF, Excel 2010
: Member of User Group Benelux :
 
Posts: 1667 | Location: Enschede, Netherlands | Registered: August 12, 2010Reply With QuoteReport This Post
Member
posted Hide Post
Thanks for you help.
Since I cannot change the format at this point so I'll wait for someone who can help me with this.


Focus for Unix
Windows, All Outputs
 
Posts: 6 | Registered: January 04, 2013Reply With QuoteReport This Post
Virtuoso
posted Hide Post
Then why did you change it into this instead of D14.2?!?
quote:
GPRS_HOME_CHG/D13.2 OVER
GPRS_INT_ROAM_CHG/D13.2


Come to think of it, you might want to check the documentation on whether P (packed) format numerics and D (decimal) format numerics interpret 14.2 in the same way. I don't think they do.

I seem to recall that P14.2 means a number with 14 positions before the decimal point (and 2 after, obviously), while D14.2 means 14 positions total, of which 2 are decimals. The decimal dot might count as a position as well in the decimal format. That would mean that the equivalent of a P14.2 is not a D14.2, but rather a D16.2 or a D17.2!

Regardless, you really should check the documentation on these types. It should explain the behaviour you're seeing.

This message has been edited. Last edited by: Wep5622,


WebFOCUS 8.1.03, Windows 7-64/2008-64, IBM DB2/400, Oracle 11g & RDB, MS SQL-Server 2005, SAP, PostgreSQL 11, Output: HTML, PDF, Excel 2010
: Member of User Group Benelux :
 
Posts: 1667 | Location: Enschede, Netherlands | Registered: August 12, 2010Reply With QuoteReport This Post
Expert
posted Hide Post
quote:

Reference: FORMAT WP
Description: Captures the entire report output, including headings, footings, and subtotals, and creates a text file that can easily be incorporated into most word processing packages.

This surely means you will have rows of different types, column titles, two different data rows...

For testing purposes, you should try to create a demo fex using an IBI-supplied file as input.

I have never created a WP file to be used as input for further processing.

This message has been edited. Last edited by: Francis Mariani,


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Reply With QuoteReport This Post
Expert
posted Hide Post
SET ASNAMES=ON
SET HOLDLIST=PRINTONLY

SET NODATA=0

FILEDEF DDGP2 DISK ddgp2.wp
-RUN

TABLE FILE CAR
SUM
WEIGHT/P14.2 OVER
HEIGHT/P14.2

ACROSS SEATS AS ''

ON TABLE SET PAGE NOPAGE
ON TABLE HOLD AS DDGP2 FORMAT WP
END
-RUN

FILEDEF DDMAS DISK ddgp2.mas ( RECFM V LRECL 100
-RUN

-WRITE DDMAS FILE = DDGP2, SUFFIX=FIX, $
-WRITE DDMAS
-WRITE DDMAS SEGMENT = DDGP2, SEGTYPE=S0, $
-WRITE DDMAS   FIELD = FILLER0     , USAGE=A02  , ACTUAL=A02, $
-WRITE DDMAS   FIELD = MEASURE_TYPE, USAGE=A06  , ACTUAL=A06, $
-WRITE DDMAS   FIELD = FILLER1     , USAGE=A02  , ACTUAL=A02, $
-WRITE DDMAS   FIELD = MEASURE1    , USAGE=P14.2, ACTUAL=A14, $
-WRITE DDMAS   FIELD = FILLER2     , USAGE=A02  , ACTUAL=A02, $
-WRITE DDMAS   FIELD = MEASURE2    , USAGE=P14.2, ACTUAL=A14, $
-WRITE DDMAS   FIELD = FILLER3     , USAGE=A02  , ACTUAL=A02, $
-WRITE DDMAS   FIELD = MEASURE3    , USAGE=P14.2, ACTUAL=A14, $
-WRITE DDMAS   FIELD = FILLER4     , USAGE=A02  , ACTUAL=A02, $

TABLE FILE DDGP2
PRINT
MEASURE_TYPE
MEASURE1/D14.2M
MEASURE2/D14.2M
MEASURE3/D14.2M

WHERE MEASURE_TYPE IN ('WEIGHT', 'HEIGHT')
END


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Reply With QuoteReport This Post
Guru
posted Hide Post
For a discussion of rounding, take a look at Describing Data with WebFOCUS Language pages 114-115 in the WF77 version.
Also note that the D denotation does NOT stand for decimal but for Double precision floating point.
A quick review of how numbers are manipulated might be in order.
HTH


jimster06
DevStu WF 7.6.11
W7
HTML, PDF, EXL2K
 
Posts: 252 | Location: USA | Registered: April 15, 2003Reply With QuoteReport This Post
Master
posted Hide Post
I believe that packed decimal format truncates trailing decimal places exactly as indicated in the format. Double precision format preserves many - I forget how many - decimal places in the data but only displays the number indicated in the format.

So: 12345.5678 with P8.2 format is stored and displayed as 12345.56, but with D8.2 is stored as 12345.5678 and displayed as 12345.56. All calculations on the latter are done on the full decimal value.


WebFOCUS 7.7.05 Windows, Linux, DB2, IBM Lotus Notes, Firebird, Lotus Symphony/OpenOffice. Outputs PDF, Excel 2007 (for OpenOffice integration), WP
 
Posts: 674 | Location: Guelph, Ontario, Canada ... In Focus since 1985 | Registered: September 28, 2010Reply With QuoteReport This Post
  Powered by Social Strata  
 

Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [CLOSED] Decimal value getting truncated (789.56 is coming as 789.50)

Copyright © 1996-2018 Information Builders, leaders in enterprise business intelligence.