Focal Point Banner


As of December 1, 2020, Focal Point is retired and repurposed as a reference repository. We value the wealth of knowledge that's been shared here over the years. You'll continue to have access to this treasure trove of knowledge, for search purposes only.

Join the TIBCO Community
TIBCO Community is a collaborative space for users to share knowledge and support one another in making the best use of TIBCO products and services. There are several TIBCO WebFOCUS resources in the community.

  • From the Home page, select Predict: WebFOCUS to view articles, questions, and trending articles.
  • Select Products from the top navigation bar, scroll, and then select the TIBCO WebFOCUS product page to view product overview, articles, and discussions.
  • Request access to the private WebFOCUS User Group (login required) to network with fellow members.

Former myibi community members should have received an email on 8/3/22 to activate their user accounts to join the community. Check your Spam folder for the email. Please get in touch with us at community@tibco.com for further assistance. Reference the community FAQ to learn more about the community.


Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    [SOLVED] Reading just first line of a fixed length file

Read-Only Read-Only Topic
Go
Search
Notify
Tools
[SOLVED] Reading just first line of a fixed length file
 Login/Join
 
Member
posted
Hi all,

I am using data flows in Data Migrator and I need to read several large fixed length text files. The first line of each file contains a sequence number. In order to process all files in the correct order I need to read these sequence numbers for all the files.

At the moment I read the whole file and filter on the record type and some other attributes. Obviously this is a very inefficient process, but I cannot find any other way. I assume others have encountered the same situation and found solutions for this.

So my question is:

How can I efficiently (time-wise and resouce-wise) read the first line of a fixed length text file?


Etienne

This message has been edited. Last edited by: FP Mod Chuck,


WF 8.2
DMC
Gen Number: 833
Release: 81M 32bit
 
Posts: 10 | Registered: September 21, 2018Report This Post
Platinum Member
posted Hide Post
Hi Etienne,
If you look at the properties of the SQL object, you can indicate "rows to retrieve". If you set this to 1 it will probably do what you want.
regards, Martin.


WebFocus 8206M, iWay DataMigrator, Windows, DB2 Windows V10.5, MS SQL Server, Azure SQL, Hyperstage, ReportCaster
 
Posts: 168 | Registered: March 29, 2013Report This Post
Guru
posted Hide Post
Or you you could take a completely different approach and use dialog manager to read a record from a disk file.

Say you have file in your files application directory called file.dat where the first six characters contain the sequence number.

From the DMC select an application directory and New, Stored Procedure. In the open window type

-DEFAULT &SEQNUM = '      '
FILEDEF INPUT DISK files/file.dat  
-RUN
-READ INPUT &SEQNUM.A6.
-TYPE &SEQNUM 


You'll want to do more than just type the value; you could for example read in the last sequence number processed from a different file, and compare the two values.

When you are done click the Save icon to save the procedure.

In a Process flow then drag the stored procedure between Start and a Data Flow. Delete the existing arrow connecting them, and add connections from Start to the Procedure, and from the Procedure to the Data Flow.


N/A
 
Posts: 397 | Location: New York City | Registered: May 03, 2007Report This Post
Member
posted Hide Post
@Martin, thanks for your suggestion.
I forgot to mention I already tried this.

The problem is that I need to combine some attributes on the first line with some table. What DM is doing is first performing the join with all records in the file and after that take the first line. This is caused by the fact that you can put a join before the final SQL object but not after the final SQL object and you can set "rows to retrieve" on the final SQL object only.


WF 8.2
DMC
Gen Number: 833
Release: 81M 32bit
 
Posts: 10 | Registered: September 21, 2018Report This Post
Member
posted Hide Post
@Clif, thank you for your suggestion.

We are trying to do as much as possible using the flows and coding as little as possible. However it looks like we have to do it this way.

Does your code example for the Stored Procedure read just a single line?


WF 8.2
DMC
Gen Number: 833
Release: 81M 32bit
 
Posts: 10 | Registered: September 21, 2018Report This Post
Platinum Member
posted Hide Post
Etienne,
If you want to remain with flows and no coding, it might be a possibility to first make a simple flow which only includes the fixed length file, set "rows to retrieve" to 1, make new target (preferably in same database as tables you have to join with later on), and then in next flow join with this new table. And use process flow to get those flows to run after each other.
Martin.


WebFocus 8206M, iWay DataMigrator, Windows, DB2 Windows V10.5, MS SQL Server, Azure SQL, Hyperstage, ReportCaster
 
Posts: 168 | Registered: March 29, 2013Report This Post
Member
posted Hide Post
@Martin, thanks. I agree, it looks like that is the only option when you want to stick with data flows. It seems a bit silly though to do it like that.

The way our organization organized its (software) processes will result in lots of small flows and small jobs.

So we have to weigh the stored procedure alternative as described by Clif versus what you just described. I will first try if performance of the full join is acceptable. If not I will discuss with my collegues which option to choose.


WF 8.2
DMC
Gen Number: 833
Release: 81M 32bit
 
Posts: 10 | Registered: September 21, 2018Report This Post
Guru
posted Hide Post
As written it just reads the first line.

If you needed to you could keep the file open with NOCLOSE and read subsequent lines in a loop.


N/A
 
Posts: 397 | Location: New York City | Registered: May 03, 2007Report This Post
  Powered by Social Strata  

Read-Only Read-Only Topic

Focal Point    Focal Point Forums  Hop To Forum Categories  iWay Software Product Forum on Focal Point    [SOLVED] Reading just first line of a fixed length file

Copyright © 1996-2020 Information Builders