Focal Point Banner


As of December 1, 2020, Focal Point is retired and repurposed as a reference repository. We value the wealth of knowledge that's been shared here over the years. You'll continue to have access to this treasure trove of knowledge, for search purposes only.

Join the TIBCO Community
TIBCO Community is a collaborative space for users to share knowledge and support one another in making the best use of TIBCO products and services. There are several TIBCO WebFOCUS resources in the community.

  • From the Home page, select Predict: WebFOCUS to view articles, questions, and trending articles.
  • Select Products from the top navigation bar, scroll, and then select the TIBCO WebFOCUS product page to view product overview, articles, and discussions.
  • Request access to the private WebFOCUS User Group (login required) to network with fellow members.

Former myibi community members should have received an email on 8/3/22 to activate their user accounts to join the community. Check your Spam folder for the email. Please get in touch with us at community@tibco.com for further assistance. Reference the community FAQ to learn more about the community.


Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [SOLVED] Data Migrator Logs - Where are they stored?

Read-Only Read-Only Topic
Go
Search
Notify
Tools
[SOLVED] Data Migrator Logs - Where are they stored?
 Login/Join
 
Expert
posted
I'm probably breaking a few rules by cross-posting this here and on the iWay Products forum, but I'm not sure how active the iWay forum is, so please forgive me.

I would like to know where Data Migrator Logs are stored?

When I right-click a task and select "View Log", the date selection pop-up window never pops up - I cancel the request because it takes too long.

Is there a location where every execution of a task has its own log file, or is there one big log file?

I'd like to delete old logs to speed up the date selection pop-up window.

This is for IWAY 533 DataMigrator Server.

Thank you,

This message has been edited. Last edited by: Kerry,


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Report This Post
Expert
posted Hide Post
Francis,

Is your DM repository DB2? If yes, there should be log tables like there are in Report Caster. They used to be named CMLOG and CMLOG2. There should be a utility for purging that comes with the product.


Ginny
---------------------------------
Prod: WF 7.7.01 Dev: WF 7.6.9-11
Admin, MRE,self-service; adapters: Teradata, DB2, Oracle, SQL Server, Essbase, ESRI, FlexEnable, Google
 
Posts: 2723 | Location: Ann Arbor, MI | Registered: April 05, 2006Report This Post
Virtuoso
posted Hide Post
Ginny is correct - the logs can be configured much the same way as RC. They can be stand-alone FOCUS tables or can be configured to use the RDBMS of choice. Our warehouse in DB2 on Linux stores all of the scheduling, logs, etc. I can't remember back to 53x, but I think that the console has a "Manage Log and Statistics" utility like the 71x console does and you can clean up the logs from there.


Regards,

Darin



In FOCUS since 1991
WF Server: 7.7.04 on Linux and Z/OS, ReportCaster, Self-Service, MRE, Java, Flex
Data: DB2/UDB, Adabas, SQL Server Output: HTML,PDF,EXL2K/07, PS, AHTML, Flex
WF Client: 77 on Linux w/Tomcat
 
Posts: 2298 | Location: Salt Lake City, Utah | Registered: February 02, 2007Report This Post
Expert
posted Hide Post
Ginny and Darin, thanks very much for the info.

I have inherited a completely undocumented environment and I currently do not have access to the server that the iWay DataMigrator Server is on. I imagine the utility is an exe or something, just like the MRE migration tools.

I think I've found the log database. It's a FOCUS DB in directory Drive:\ibi\dm\ibi\srv53\dm\catalog called etllog.foc and it's 1,867,088 KBs!

I will have to delete a lot of data from this db, it's probably got stuff there since the beginning of time.

Thank you.


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Report This Post
Expert
posted Hide Post
Francis, you will need the DM client. Also you might want to download the manual for your release from the IBI site.

If they are FOCUS databases, you might want to do a REBUILD REORG with a date option.


Ginny
---------------------------------
Prod: WF 7.7.01 Dev: WF 7.6.9-11
Admin, MRE,self-service; adapters: Teradata, DB2, Oracle, SQL Server, Essbase, ESRI, FlexEnable, Google
 
Posts: 2723 | Location: Ann Arbor, MI | Registered: April 05, 2006Report This Post
Expert
posted Hide Post
I took a look at the DM Client and did not find any utility to purge the log. I'm about to get the documentation.


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Report This Post
Platinum Member
posted Hide Post
On the DM client, you should see Data Migrator Utilities, then Manage Log and Statistics. Under that, you should see Recreate. That's what you want. It will delete the existing log and recreate a new one for you. If you want a back up of the existing one, use Backup first, then Recreate. I responded to your other post on the iWay forum and just now saw this one so I see you found the log on your own. It's almost filled up so good thing you found it. There is a 2gb limit.

Good luck.

Jessica Bottone


Data Migrator 5.3, 7.1, 7.6
WebFOCUS 7.1, 7.6, 7.7
SQL Server, Oracle, DB2
Windows
 
Posts: 126 | Registered: January 18, 2007Report This Post
Virtuoso
posted Hide Post
Jessica,

I don't know which version you're using for sure (please update your profile so we can see your products, version, platforms, etc.) but I think you must be at least 71x if not 76x. I don't believe the items and functionality you describe is the same in the 53x version(?!) that Francis is using.


Regards,

Darin



In FOCUS since 1991
WF Server: 7.7.04 on Linux and Z/OS, ReportCaster, Self-Service, MRE, Java, Flex
Data: DB2/UDB, Adabas, SQL Server Output: HTML,PDF,EXL2K/07, PS, AHTML, Flex
WF Client: 77 on Linux w/Tomcat
 
Posts: 2298 | Location: Salt Lake City, Utah | Registered: February 02, 2007Report This Post
Platinum Member
posted Hide Post
I intentionally don't put my information in my profile because I work for many different clients and it's always different. When I ask questions, I always supply the pertinent information pertaining to the question. As for Francis' question, you are correct that the version I'm currently working on is 7.1 however, I worked in 5.3 just a little over a year ago and recalled the process was the same. I checked my notes from where I wrote up for that client on how to recreate logs and my notes indicate the process is the same. So needless to say, I'm a little confused. :-) I guess my best advice to Francis at this point is to refer to the documentation for your release.

Regards,

Jessica Bottone


Data Migrator 5.3, 7.1, 7.6
WebFOCUS 7.1, 7.6, 7.7
SQL Server, Oracle, DB2
Windows
 
Posts: 126 | Registered: January 18, 2007Report This Post
Platinum Member
posted Hide Post
Okay, went back and read my notes better. That's what I get for just skimming them. :-) While the steps you take are the same, the utilities are under the Web Console in 5.3 and not under the DM Console as it is in 7.1. My apologies.

Jessica Bottone


Data Migrator 5.3, 7.1, 7.6
WebFOCUS 7.1, 7.6, 7.7
SQL Server, Oracle, DB2
Windows
 
Posts: 126 | Registered: January 18, 2007Report This Post
Expert
posted Hide Post
Jessica, Thank you for your posting. I'll look into this on Monday morning. I'll update my post in the other forum once I've figured everything out.

It's too bad that you only have the choice of backing up and erasing the log file - purging up to a specified date would have been a nice option, leaving some prior executions of tasks in the log would be handy for problem solving.

I think a dump /rebuild may be a better idea - I can select what to delete and what to leave.

Cheers everyone - have a good weekend.


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Report This Post
Silver Member
posted Hide Post
quote:
Way

Francis,
Here is a copy of my comments - from iWay Products section.
Francis, in release 533 DataMigrator does not have the Log and Statistics management at Data Management Console. In order to clean or create a new (better for your situation) “Log and Statistics”, you need to go Web Console --> Procedures --> DM Utilities [...].

You mentioned “date selection pop-up window“ when “right-click a task”. The date selection will come up from the Reports of Logs and Statistics.

Please notice, that the physical location of the files for ETL log and statistics is rather in a catalog under “dm” folder, e.g. ibi/srv##/dm##/catalog/. However, ETL log and statistics files ought to be managed via DM utilities – not manually.

Marina
 
Posts: 37 | Registered: February 13, 2007Report This Post
Expert
posted Hide Post
Marina,

Thank you for your comments.

I found the DM Utilities and ran "Backup" to backup the log database.

When I tried to run "Clean", I got the error "(ICM18261) User not authorized for ETLLOG/ETLSTATS delete: SYSTEM"

I then tried "Recreate" and that worked.

Interesting that I could not Clean the database but was able to Recreate it.


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Report This Post
Virtuoso
posted Hide Post
I also ran into something simlar a while back. However the permissions were set up, I could not "Clean" which used an SQL DELETE (or FOCUS MODIFY), but I could recreate which did an SQL TRUNCATE (or FOCUS CREATE). I'm also interested to know how privileges were established for this. Haven't had any issues at all with our latest install.


Regards,

Darin



In FOCUS since 1991
WF Server: 7.7.04 on Linux and Z/OS, ReportCaster, Self-Service, MRE, Java, Flex
Data: DB2/UDB, Adabas, SQL Server Output: HTML,PDF,EXL2K/07, PS, AHTML, Flex
WF Client: 77 on Linux w/Tomcat
 
Posts: 2298 | Location: Salt Lake City, Utah | Registered: February 02, 2007Report This Post
Expert
posted Hide Post
Thanks very much to everyone.

I have one more question: the logs are stored in FOCUS databases, but the Data Flows are individual etg and fex files. It appears we can't join to another table to get names to the Data Flows in the log. I do see the Impact Analysis DM Reports that somehow report on task information, how is this done?

I had a quick look at the documentation but I assume that this type of thing is not discussed there.

Thank you,


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Report This Post
Platinum Member
posted Hide Post
I've never used the impact analysis reports so I can't say how they work. I would be surprised to hear they work off the logs since the logs only contain information on what has been run since the last time the log was recreated, and there could be many more .fex and .etg files. However, I can tell you this - the log files have an associated etllog.mas. And from that, you can report on it. For me, the .MAS is located in ibi/srv##/home/catalog. Take a look at it, display out some of the data and see if maybe it has what you're looking for.

Regards,

Jessica Bottone

This message has been edited. Last edited by: Jessica Bottone,


Data Migrator 5.3, 7.1, 7.6
WebFOCUS 7.1, 7.6, 7.7
SQL Server, Oracle, DB2
Windows
 
Posts: 126 | Registered: January 18, 2007Report This Post
Virtuoso
posted Hide Post
I'm not 100% positive, but I believe the Impact Analysis reports works much like they do in MRE, where it actually opens and parses the procedures in the specified path, looking for the desired field or file. It does take a little while - the more procedures you have the longer it takes. I've never looked, but it would be interesting to find the procedures that are actually run for Impact Analysis and see what they contain.


Regards,

Darin



In FOCUS since 1991
WF Server: 7.7.04 on Linux and Z/OS, ReportCaster, Self-Service, MRE, Java, Flex
Data: DB2/UDB, Adabas, SQL Server Output: HTML,PDF,EXL2K/07, PS, AHTML, Flex
WF Client: 77 on Linux w/Tomcat
 
Posts: 2298 | Location: Salt Lake City, Utah | Registered: February 02, 2007Report This Post
Silver Member
posted Hide Post
Darin and Francis,
Did you sign up to the server with Securities OFF or Securities OPSYS? You could have check your user’s roles; and keep in mind that you are using different releases: 53 and 71.
Speaking of Impact Analysis, I was wondering what exactly you need to do. Are you trying to retrieve the “REQ_NAME” field from the etllog table? If there is something specific you need to do, please ask with details. I’ll try to help.

Thanks.
Marina
 
Posts: 37 | Registered: February 13, 2007Report This Post
Platinum Member
posted Hide Post
Frances, I was digging through the ibi tech site today and found this (see below). I remembered this post and recalled that you wished there were other choices for the ETL log than just completely purging and rebuilding. Well, apparently there is. I recall that your version is 5.3 but maybe this will work for you too. I found ETLDELLS.FEX in the srv##/home/catalog directory. Instead of purging everything older than 7 days ago, I just hardcoded a value for DDATE, ran it, and it did delete everything older than that date. Pretty cool!

***************************
How do you schedule an ETLLOG cleaning in 7.12?

Case: 13322022

Product: Data Migrator Release: 7.1.0 07.01

Primary OS: Windows Date: 05/12/21

Case information refers to the Product, Release, and OS for which this question was asked. The solution may apply to other Products, Releases, and Operating Systems.

Symptom: How do you schedule an ETLLOG cleaning in 7.12?

Problem: How to automatically delete rows from the server log and statistics table?

Solution: It may be useful for a server to automate deleting rows from the server log and statistics tables. For example, you can delete all rows more than a week old on a daily basis.

1. Create a stored procedure with the following lines:

-SET &DDATE=AYMD(&YYMD,-7,'I8');
-SET &DDATE = EDIT (&DDATE,'9999/99/99') ;
EX ETLDELLS DDATE=&DDATE, DTIME=00:00:00, SUSTART=Y, ETLSTART=Y

2. Create a new process flow.

3. Drag the procedure into the flow and connect it to the Start object.

4. Drag a Schedule icon into the workspace.

5. Double-click the Schedule icon and set a schedule for running the flow.

6. Save the flow.


Data Migrator 5.3, 7.1, 7.6
WebFOCUS 7.1, 7.6, 7.7
SQL Server, Oracle, DB2
Windows
 
Posts: 126 | Registered: January 18, 2007Report This Post
Expert
posted Hide Post
Jessica,

Thanks very much - this is very valuable information. Now I can set up a scheduled ETL Log clean-up.

Regards,

Francis.


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Report This Post
Virtuoso
posted Hide Post
A very valuable discussion...

Our logfiles are growing very fast (I do not know why, no time to find out) and what we do once a month is recreating them.
This scheduled cleaning process much better.

Where is that ETLDELLS process stored Jessica?




Frank

prod: WF 7.6.10 platform Windows,
databases: msSQL2000, msSQL2005, RMS, Oracle, Sybase,IE7
test: WF 7.6.10 on the same platform and databases,IE7

 
Posts: 2387 | Location: Amsterdam, the Netherlands | Registered: December 03, 2006Report This Post
Expert
posted Hide Post
Frank, on our server it's located here: D:\ibi\srv53\home\catalog\etldells.fex


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Report This Post
Virtuoso
posted Hide Post
Thanks, I'll take a look tomorrow. It will be ...\srv71\... I suppose.




Frank

prod: WF 7.6.10 platform Windows,
databases: msSQL2000, msSQL2005, RMS, Oracle, Sybase,IE7
test: WF 7.6.10 on the same platform and databases,IE7

 
Posts: 2387 | Location: Amsterdam, the Netherlands | Registered: December 03, 2006Report This Post
Expert
posted Hide Post
Frank,

I also found the program in D:\ibi\dm\ibi\srv53\home\catalog which is the location of our DataMigrator server.


Francis


Give me code, or give me retirement. In FOCUS since 1991

Production: WF 7.7.05M, Dev Studio, BID, MRE, WebSphere, DB2 / Test: WF 8.1.05M, App Studio, BI Portal, Report Caster, jQuery, HighCharts, Apache Tomcat, MS SQL Server
 
Posts: 10577 | Location: Toronto, Ontario, Canada | Registered: April 27, 2005Report This Post
Platinum Member
posted Hide Post
For me, it's stored in C:\ibi\srv71\home\catalog


Data Migrator 5.3, 7.1, 7.6
WebFOCUS 7.1, 7.6, 7.7
SQL Server, Oracle, DB2
Windows
 
Posts: 126 | Registered: January 18, 2007Report This Post
Virtuoso
posted Hide Post
I have found this fex Jessica, I created the cleaning procedure, but when I run it....well the logfile is cleaned. All the records are gone, so it does start but does not use the variables.




Frank

prod: WF 7.6.10 platform Windows,
databases: msSQL2000, msSQL2005, RMS, Oracle, Sybase,IE7
test: WF 7.6.10 on the same platform and databases,IE7

 
Posts: 2387 | Location: Amsterdam, the Netherlands | Registered: December 03, 2006Report This Post
Platinum Member
posted Hide Post
Wow. Do you happen to know what range of dates you had in your log at the time, and what did you set the date variable to?

I can tell you this: I'm currently on release 7.1.5, my log file contained information from 9/10/2008 to current, and I set my code to look like this:

 
-SET &ECHO=ALL;
SET WIDTH = 132
SET PANEL = 99999
SET ASNAMES =ON
 
-*- -SET &DDATE=AYMD(&YYMD,-7,'I8');
-*- -SET &DDATE = EDIT (&DDATE,'9999/99/99') ;
-*- EX ETLDELLS DDATE=&DDATE, DTIME=00:00:00, SUSTART=Y, ETLSTART=Y
 
EX ETLDELLS DDATE=2008/09/11, DTIME=00:00:00, SUSTART=Y, ETLSTART=Y
 


And everything before 9/11/2008, which was only 9/10/2008, was deleted. All else remained. I checked before I ran ETLDELLS.FEX and after.


Data Migrator 5.3, 7.1, 7.6
WebFOCUS 7.1, 7.6, 7.7
SQL Server, Oracle, DB2
Windows
 
Posts: 126 | Registered: January 18, 2007Report This Post
  Powered by Social Strata  

Read-Only Read-Only Topic

Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [SOLVED] Data Migrator Logs - Where are they stored?

Copyright © 1996-2020 Information Builders