Can anyone explain why my calculation for "PTM_NOT_SHIPPED_IMAGES" is wrong? It's calculating a value of 1,273,022, but it should be 1,273,333 (276,211 * 4.61).
-SET &Processed_Images = '2232254'; -SET &Processed_Stmts = '484338'; -SET &PTM_Stmts = '208127'; TABLE FILE CAR SUM CAR NOPRINT COMPUTE IMGS_PER_STMT/D3.2 = &Processed_Images / &Processed_Stmts; AS 'Images per Statement' COMPUTE PTM_NOT_SHIPPED/D12 = &Processed_Stmts - &PTM_Stmts; AS 'Statements Not Shipped' COMPUTE PTM_NOT_SHIPPED_IMAGES/D12 = PTM_NOT_SHIPPED * IMGS_PER_STMT; AS 'Images Not Shipped' ENDThis message has been edited. Last edited by: Joel Elscott,
Change D3.2 to D5.2...
Images per Statement Statements Not Shipped Images Not Shipped 4.61 276,211 1,273,022
WebFOCUS 8.1.05 - PROD/QA
DB2 - AS400 - Mainframe
I tried D5.2 and D3.2 but get same result of 1,273,022.
4.61 * 276,211 should equal 1,273,333.
Try /P format instead of /D such as /P3.2
Change them all to /P
WF versions : Prod 8.2.04M gen 33, Dev 8.2.04M gen 33, OS : Windows, DB : MSSQL, Outputs : HTML, Excel, PDF
In Focus since 2007
That worked! Thank you!
Dare I ask why Packed vs. Decimal would make a difference?
It's the way that the data is held.
Packed decimal format will only store the decimal component to the number of decimal places given within the format (in your case, 2 decimal places) rounding as per rules.
Double Precision format (not decimal) will hold the result at more decimal places than given in the format - which is essentially a "usage" format in this instance.
So, with double precision, your calculation is using 4.60887644579 etc. and not 4.61 which is what packed decimal format provides.
Thank you Tony, I appreciate your detailed explanation!
|Powered by Social Strata|