 |
|||||
|
As of December 1, 2020, Focal Point is retired and repurposed as a reference repository. We value the wealth of knowledge that's been shared here over the years. You'll continue to have access to this treasure trove of knowledge, for search purposes only. Join the TIBCO Community
Former myibi community members should have received an email on 8/3/22 to activate their user accounts to join the community. Check your Spam folder for the email. Please get in touch with us at community@tibco.com for further assistance. Reference the community FAQ to learn more about the community. | |||||
Read-Only TopicGo | Search | Notify | Tools |
| Quick Question on Rounding | Login/Join |
| Gold member |
I was trying to set a standard I could follow at our site for two types of numbers. I was hoping to make the statement that all Surrogate Key values in the database could/should be defined as P11. These are basically sequence generated numbers that will have no decimal precision. This seems to work fine. Now for Fact Table data and metrics my intent is to keep as many decimals as possible and only do the rounding once on the way out to the screen after the calculation. I have gone through the WebFocus 5.2 manual and it seems that Decimal datatype is my best bet. Is this correct? Is there anything special I need to know of when setting these. I'd like all calcs to work at least out to 4 decimals. Do I need to make all metric elements a D20.4 to enable the calc or will a D20.2 (which is how most elements are stored in our Oracle tables) also carry and handle more than just those two digits. It confuses me that some keep the decimals in the background and some are actually rounding as you go. WF 5.3.5 / SOLARS 2.9 / Apache / Tomcat / Oracle (9.2/10g) | ||
|
| Gold member |
After doing some quick tests it looks like the Decimal format type will in fact keep the decimals behind the scene and round at the end. However, because most of the calcs require multiple phases I will need them rounded to no less than 4 decimals to be used later on as multipliers etc. for other calcs. I assume that for that application a D20.4 will hold just about anything our database can throw at it. If I have the ACTUAL=D8 and the USAGE=D20.4 will there be problems if the numbers get large? WF 5.3.5 / SOLARS 2.9 / Apache / Tomcat / Oracle (9.2/10g) | |||
|
| Guru |
Interestingly enough, the WF71 Describing Data with WebFOCUS Language (p5-15) says that the maximum display size for double precision floating point is 15 digits. If you want to maintain precision for very large or very small numbers, consider the use of packed decimal. There is also a discussion of rounding in the same manual, p5-23. Note that up until WF52 calculations in DEFINEs etc were done in Double Precision Floating Point and then assigned to however the variable was defined. This led to some interesting situations from time-to-time ie there was a slight wobble. Since 5.2 and higher, if the variable is PD the calculation is PD. HTHThis message has been edited. Last edited by: jimster06, | |||
|
| Expert |
i agree with you. i have found that D keeps decimal positions 'behind the scene' as you say, and presents the rounded version according to the format you specify. much like excel.
| |||||
|
| Platinum Member |
Some online documentation with examples to illustrate a little more about Formats, Storage, Display and Rounding: - Rounding in Calculations and Conversions - Data Storage and Display Regards, Mikel WebFOCUS 8.1.05, 8.2.01 | |||
|
| Powered by Social Strata |
 | Please Wait. Your request is being processed... |
Read-Only TopicCopyright © 1996-2020 Information Builders
| View $GS_USERNAME's Public Profile |
| Add $GS_USERNAME to my Ignore List |
| View Recent Posts by $GS_USERNAME |
| Notify me of New Posts by $GS_USERNAME |