I'm probably breaking a few rules by cross-posting this here and on the iWay Products forum, but I'm not sure how active the iWay forum is, so please forgive me.

I would like to know where Data Migrator Logs are stored?

When I right-click a task and select "View Log", the date selection pop-up window never pops up - I cancel the request because it takes too long.

Is there a location where every execution of a task has its own log file, or is there one big log file?

I'd like to delete old logs to speed up the date selection pop-up window.

This is for IWAY 533 DataMigrator Server.

Thank you,

This message has been edited. Last edited by: Kerry,

Francis,

Is your DM repository DB2? If yes, there should be log tables like there are in Report Caster. They used to be named CMLOG and CMLOG2. There should be a utility for purging that comes with the product.

Ginny is correct - the logs can be configured much the same way as RC. They can be stand-alone FOCUS tables or can be configured to use the RDBMS of choice. Our warehouse in DB2 on Linux stores all of the scheduling, logs, etc. I can't remember back to 53x, but I think that the console has a "Manage Log and Statistics" utility like the 71x console does and you can clean up the logs from there.

Ginny and Darin, thanks very much for the info.

I have inherited a completely undocumented environment and I currently do not have access to the server that the iWay DataMigrator Server is on. I imagine the utility is an exe or something, just like the MRE migration tools.

I think I've found the log database. It's a FOCUS DB in directory Drive:\ibi\dm\ibi\srv53\dm\catalog called etllog.foc and it's 1,867,088 KBs!

I will have to delete a lot of data from this db, it's probably got stuff there since the beginning of time.

Thank you.

Francis, you will need the DM client. Also you might want to download the manual for your release from the IBI site.

If they are FOCUS databases, you might want to do a REBUILD REORG with a date option.

I took a look at the DM Client and did not find any utility to purge the log. I'm about to get the documentation.

On the DM client, you should see Data Migrator Utilities, then Manage Log and Statistics. Under that, you should see Recreate. That's what you want. It will delete the existing log and recreate a new one for you. If you want a back up of the existing one, use Backup first, then Recreate. I responded to your other post on the iWay forum and just now saw this one so I see you found the log on your own. It's almost filled up so good thing you found it. There is a 2gb limit.

Good luck.

Jessica Bottone

Jessica,

I don't know which version you're using for sure (please update your profile so we can see your products, version, platforms, etc.) but I think you must be at least 71x if not 76x. I don't believe the items and functionality you describe is the same in the 53x version(?!) that Francis is using.

I intentionally don't put my information in my profile because I work for many different clients and it's always different. When I ask questions, I always supply the pertinent information pertaining to the question. As for Francis' question, you are correct that the version I'm currently working on is 7.1 however, I worked in 5.3 just a little over a year ago and recalled the process was the same. I checked my notes from where I wrote up for that client on how to recreate logs and my notes indicate the process is the same. So needless to say, I'm a little confused. :-) I guess my best advice to Francis at this point is to refer to the documentation for your release.

Regards,

Jessica Bottone

Okay, went back and read my notes better. That's what I get for just skimming them. :-) While the steps you take are the same, the utilities are under the Web Console in 5.3 and not under the DM Console as it is in 7.1. My apologies.

Jessica Bottone

Jessica, Thank you for your posting. I'll look into this on Monday morning. I'll update my post in the other forum once I've figured everything out.

It's too bad that you only have the choice of backing up and erasing the log file - purging up to a specified date would have been a nice option, leaving some prior executions of tasks in the log would be handy for problem solving.

I think a dump /rebuild may be a better idea - I can select what to delete and what to leave.

Cheers everyone - have a good weekend.

quote:

Way

Francis,
Here is a copy of my comments - from iWay Products section.
Francis, in release 533 DataMigrator does not have the Log and Statistics management at Data Management Console. In order to clean or create a new (better for your situation) “Log and Statistics”, you need to go Web Console --> Procedures --> DM Utilities [...].

You mentioned “date selection pop-up window“ when “right-click a task”. The date selection will come up from the Reports of Logs and Statistics.

Please notice, that the physical location of the files for ETL log and statistics is rather in a catalog under “dm” folder, e.g. ibi/srv##/dm##/catalog/. However, ETL log and statistics files ought to be managed via DM utilities – not manually.

Marina

Marina,

Thank you for your comments.

I found the DM Utilities and ran "Backup" to backup the log database.

When I tried to run "Clean", I got the error "(ICM18261) User not authorized for ETLLOG/ETLSTATS delete: SYSTEM"

I then tried "Recreate" and that worked.

Interesting that I could not Clean the database but was able to Recreate it.

I also ran into something simlar a while back. However the permissions were set up, I could not "Clean" which used an SQL DELETE (or FOCUS MODIFY), but I could recreate which did an SQL TRUNCATE (or FOCUS CREATE). I'm also interested to know how privileges were established for this. Haven't had any issues at all with our latest install.

Thanks very much to everyone.

I have one more question: the logs are stored in FOCUS databases, but the Data Flows are individual etg and fex files. It appears we can't join to another table to get names to the Data Flows in the log. I do see the Impact Analysis DM Reports that somehow report on task information, how is this done?

I had a quick look at the documentation but I assume that this type of thing is not discussed there.

Thank you,

I've never used the impact analysis reports so I can't say how they work. I would be surprised to hear they work off the logs since the logs only contain information on what has been run since the last time the log was recreated, and there could be many more .fex and .etg files. However, I can tell you this - the log files have an associated etllog.mas. And from that, you can report on it. For me, the .MAS is located in ibi/srv##/home/catalog. Take a look at it, display out some of the data and see if maybe it has what you're looking for.

Regards,

Jessica Bottone

This message has been edited. Last edited by: Jessica Bottone,

I'm not 100% positive, but I believe the Impact Analysis reports works much like they do in MRE, where it actually opens and parses the procedures in the specified path, looking for the desired field or file. It does take a little while - the more procedures you have the longer it takes. I've never looked, but it would be interesting to find the procedures that are actually run for Impact Analysis and see what they contain.

Darin and Francis,
Did you sign up to the server with Securities OFF or Securities OPSYS? You could have check your user’s roles; and keep in mind that you are using different releases: 53 and 71.
Speaking of Impact Analysis, I was wondering what exactly you need to do. Are you trying to retrieve the “REQ_NAME” field from the etllog table? If there is something specific you need to do, please ask with details. I’ll try to help.

Thanks.
Marina

Frances, I was digging through the ibi tech site today and found this (see below). I remembered this post and recalled that you wished there were other choices for the ETL log than just completely purging and rebuilding. Well, apparently there is. I recall that your version is 5.3 but maybe this will work for you too. I found ETLDELLS.FEX in the srv##/home/catalog directory. Instead of purging everything older than 7 days ago, I just hardcoded a value for DDATE, ran it, and it did delete everything older than that date. Pretty cool!

***************************
How do you schedule an ETLLOG cleaning in 7.12?

Case: 13322022

Product: Data Migrator Release: 7.1.0 07.01

Primary OS: Windows Date: 05/12/21

Case information refers to the Product, Release, and OS for which this question was asked. The solution may apply to other Products, Releases, and Operating Systems.

Symptom: How do you schedule an ETLLOG cleaning in 7.12?

Problem: How to automatically delete rows from the server log and statistics table?

Solution: It may be useful for a server to automate deleting rows from the server log and statistics tables. For example, you can delete all rows more than a week old on a daily basis.

1. Create a stored procedure with the following lines:

-SET &DDATE=AYMD(&YYMD,-7,'I8');
-SET &DDATE = EDIT (&DDATE,'9999/99/99') ;
EX ETLDELLS DDATE=&DDATE, DTIME=00:00:00, SUSTART=Y, ETLSTART=Y

2. Create a new process flow.

3. Drag the procedure into the flow and connect it to the Start object.

4. Drag a Schedule icon into the workspace.

5. Double-click the Schedule icon and set a schedule for running the flow.

6. Save the flow.

Jessica,

Thanks very much - this is very valuable information. Now I can set up a scheduled ETL Log clean-up.

Regards,

Francis.

A very valuable discussion...

Our logfiles are growing very fast (I do not know why, no time to find out) and what we do once a month is recreating them.
This scheduled cleaning process much better.

Where is that ETLDELLS process stored Jessica?

Frank, on our server it's located here: D:\ibi\srv53\home\catalog\etldells.fex

Thanks, I'll take a look tomorrow. It will be ...\srv71\... I suppose.

Frank,

I also found the program in D:\ibi\dm\ibi\srv53\home\catalog which is the location of our DataMigrator server.

For me, it's stored in C:\ibi\srv71\home\catalog

I have found this fex Jessica, I created the cleaning procedure, but when I run it....well the logfile is cleaned. All the records are gone, so it does start but does not use the variables.

Wow. Do you happen to know what range of dates you had in your log at the time, and what did you set the date variable to?

I can tell you this: I'm currently on release 7.1.5, my log file contained information from 9/10/2008 to current, and I set my code to look like this:

 
-SET &ECHO=ALL;
SET WIDTH = 132
SET PANEL = 99999
SET ASNAMES =ON
 
-*- -SET &DDATE=AYMD(&YYMD,-7,'I8');
-*- -SET &DDATE = EDIT (&DDATE,'9999/99/99') ;
-*- EX ETLDELLS DDATE=&DDATE, DTIME=00:00:00, SUSTART=Y, ETLSTART=Y
 
EX ETLDELLS DDATE=2008/09/11, DTIME=00:00:00, SUSTART=Y, ETLSTART=Y
 

And everything before 9/11/2008, which was only 9/10/2008, was deleted. All else remained. I checked before I ran ETLDELLS.FEX and after.