Focal Point
[DISCUSSION]Data Integrity/ETL Process Best Practices

This topic can be found at:
https://forums.informationbuilders.com/eve/forums/a/tpc/f/7971057331/m/4767085486

November 15, 2016, 01:35 PM
mcp
[DISCUSSION]Data Integrity/ETL Process Best Practices
We are new users who have extensive Data warehousing already established which we are using as the source for our WebFocus synonyms. We have had a few instances of job failures in the night and have inadvertently sent out reports with missing data and have had Hyperstage builds that have completed without returning any data.

I did some research and have considered querying our SQL Agent Job Statuses and then creating pre-processing fexes that can kill the reportcaster and Hyperstage builds but wanted to see if anyone else has implmentted any other ways of handling data.

I am looking for best practices on how other organizations are handling this process.

This message has been edited. Last edited by: <Emily McAllister>,


8.015M
Windows 10
Everything
November 17, 2016, 03:18 PM
eric.woerle
The way I used to handle this in the past is I would set up a schedule to run a report. I would set the first time that I wanted it to run and have be a non re-occurring schedule. Then I would have an ETL job that was set at the end of the ETL run that would update the next run time. Say set it to 5 minutes ahead of the current time. Reportcaster would then recognize this as a jump that needs to be executed and execute the job. Here is an old link from when I was working out how to do this ( http://forums.informationbuild...=355105693#355105693 ). In that example I needed to burst a report, so I would cycle through the values, and then set the necessary paramaters.

I think the table that you are going to want to update is the botsched table. This is the FOCUS code I used back then. The tables might have changed slightly since then. But I doubt enough to cause any issues

 
DEFINE FILE BOTSCHED
NOWDATETIME_A/HYYMDS=HGETC(8, 'HYYMDS');
NOWDATETIME/HYYMDS=HADD(NOWDATETIME_A, 'minute', -3, 8, 'HYYMDS');
NOWDATE/YYMD=&YYMD;
STARTDATE/YYMD=19700101;
DAYSDIFF/D8=DATEDIF(STARTDATE, NOWDATE, 'D');
MSDIFF/D32=DAYSDIFF*24*60*60*1000;
ADDOFFSET/D12=4*60*60*1000;
NOWMILLISEC/D12 = HTIME(8, NOWDATETIME, 'D12.2');
NEEDTOADD/D13=NOWMILLISEC+ADDOFFSET+MSDIFF;
-*NEXTRUNTIME_NEW/A17=FTOA(NEEDTOADD, '(D13c)', 'A17');
NEXTRUNTIME_T/A13 =  FTOA(NEEDTOADD, '(D13c)', 'A13');
NEXTRUNTIME_NEW/A32 = '0000000000000000000' | NEXTRUNTIME_T;
END
 
SET ASNAMES = ON
TABLE FILE BOTSCHED
PRINT SCHEDULEID NEXTRUNTIME_NEW AS NEXTRUNTIME
WHERE SCHEDULEID = 'S14i1ies4f05'; 
ON TABLE HOLD AS SCHDHOLD
END
-RUN
 
MODIFY FILE BOTSCHED
 
FIXFORM FROM SCHDHOLD
 
   MATCH SCHEDULEID
    ON MATCH UPDATE NEXTRUNTIME
    ON NOMATCH REJECT
    DATA ON SCHDHOLD
    END 
-RUN 



Eric Woerle
8.1.05M Gen 913- Reporting Server Unix
8.1.05 Client Unix
Oracle 11.2.0.2