Focal Point
[SOLVED]How to write DataMigrator logs to Vertica DB

This topic can be found at:
https://forums.informationbuilders.com/eve/forums/a/tpc/f/1381057331/m/6117013486

October 21, 2016, 01:09 PM
slee
[SOLVED]How to write DataMigrator logs to Vertica DB
We have a number of flows and processes running daily through DataMigrator. We use Datamigrator to control both our ETL process as well to control the processing of our various analytics performed on our Vertica databases. Because of this, we'd like to retain logs for up to a month (or longer) to reference in case we need to debug a faulty procedure.

Instead of storing these log files on the local computer, we were wondering if we could instead direct the logs to write to a table in our Vertica Database. We could handle any archiving/retention policies easily there as well as save the space on our hard drive.

I've found a brief mention in the docs of the potential to do something like this, but haven't been able to find anything detailed. Has anyone had any experience with doing something like this?

This message has been edited. Last edited by: Tamra,


DataMigrator 7.7.06
Windows Server 2012
October 25, 2016, 09:39 AM
dhagen
I've done this for very much the same reasons, but with a slight change. I don't believe that you can use anything other than the existing logging mechanism. So we left that alone. What we did was set up a reoccurring job to extract the log changes and load them into an alternate DB. We keep that live logs as small as possible, and delete data from the external log repository weekly.


"There is no limit to what you can achieve ... if you don’t care who gets the credit." Roger Abbott
October 25, 2016, 10:26 AM
slee
Thanks dhagen. After a couple of days mulling it over. That's the approach we decided to take as well. Was hoping we could implement a more direct approach, but alas, this'll have to do.


DataMigrator 7.7.06
Windows Server 2012
December 12, 2017, 10:35 AM
Doug
Darryl, slee, or anyone,

Does anyone have the details (a fex?) to make this happen?

Thanks in advance, Doug




   In FOCUS Since 1983 ~ from FOCUS to WebFOCUS.
   Current: WebFOCUS Administrator at FIS Worldpay | 8204, 8206