April 24, 2008, 10:13 PM
JaniceLoading Data option
I have a measure that is a YTD measure. Every month I only get the data for that month. I want to be able to add that month's data and ignore the others. By choosing the option to Add new and increment old, it works because I am not adding anything to the old. However if there was a data error and we need to "reload" the current month, I am stuck. If I put it to replace, it wipes out all the old data and I don't want to increment. I would need an option that says replace new and do nothing with old. Do you have another suggestion?
Thanks.
April 25, 2008, 10:48 AM
Bob Jude FerranteSounds as if you need a new feature. Something like "Replace new, leave old."
Right now the only way to do this would be to clear out the data in the table. But in the case of a customer I would not recommend that.
You could run a process before the load to do a MODIFY and clear out new data based on last load date. Since measures that are not changed are not updated, the old data dates would not reflect the load date. So that's a possible technique for now.
thanks
April 25, 2008, 10:53 AM
Bob Jude FerranteAnother thing to consider. What you're asking for sounds a bit like rollback. RDBMSs do this already. If you've got the rollback option turned on in the RDBMS, you can roll the data back to the last good load, then re-do the load with the good data.
Try that one on.
April 25, 2008, 04:09 PM
JaniceUnfortunately we do not have a RDBMS behind this measure, so the "Replace new, leave old" sounds really nice!
April 25, 2008, 10:21 PM
Bob Jude FerranteBut you do have an RDBMS behind PMF. The rollback I am describing is in PMF's data mart, not on the source, of course.
May 06, 2008, 11:36 AM
Endre PekarikBob,
Do we only "commit" when all data loaded successfully into PMF?
If so, Janice should be ok.
\But if Janice finds out that the data was loaded, but dirty and needs to reload current, month, she is in trouble.
Endre
May 06, 2008, 11:56 AM
Bob Jude FerranteIf there are load errors, each measure load can be set up to either ignore errors and always load the "good" part of the measure data, or to automatically halt and not load on errors.
If you don't want PMF to load data if there are any errors, you must select the error option on the measure loader for each measure you want to protect from erroneous loads, so as to not allow any data to be loaded on errors.
If you have already half-loaded erroneous measures into the mart, and you disabled the error checking routine, which disables your protection if the measure load isn't 100% kosher, it's still not a total loss if you use the RDBMS's save and rollback technology to roll the data mart to a last known good version. That's what I'd recommend as a backup process in the future, besides turning error checking on for your measure loads.
Hope this helps.