Focal Point Banner


As of December 1, 2020, Focal Point is retired and repurposed as a reference repository. We value the wealth of knowledge that's been shared here over the years. You'll continue to have access to this treasure trove of knowledge, for search purposes only.

Join the TIBCO Community
TIBCO Community is a collaborative space for users to share knowledge and support one another in making the best use of TIBCO products and services. There are several TIBCO WebFOCUS resources in the community.

  • From the Home page, select Predict: WebFOCUS to view articles, questions, and trending articles.
  • Select Products from the top navigation bar, scroll, and then select the TIBCO WebFOCUS product page to view product overview, articles, and discussions.
  • Request access to the private WebFOCUS User Group (login required) to network with fellow members.

Former myibi community members should have received an email on 8/3/22 to activate their user accounts to join the community. Check your Spam folder for the email. Please get in touch with us at community@tibco.com for further assistance. Reference the community FAQ to learn more about the community.


Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [CLOSED] Help with reading multiple txt-files from folder (EDID4-format)

Read-Only Read-Only Topic
Go
Search
Notify
Tools
[CLOSED] Help with reading multiple txt-files from folder (EDID4-format)
 Login/Join
 
Jean-Pierre
Member
posted
Hello everyone,

I am dealing with about 70.000 text-files (.txt) containing data in the EDID4-format. Every text-file can contain several thousand rows. If I assume that every file has 5.000 rows this leads to a volume of about 350 million rows and about 42 GB in text-files. I would like to process these files in WebFOCUS but don't know exactly how to read a folder with that mass of data sources. Is it possible? Is this a standard use case?

So what I'm thinking about:

1. Create one masterfile that handles the EDID4-format. (done)
2. Loading the txt-files to the server.
3. Reading all names of the txt-files to WebFOCUS. --> How?
4a. Looping a report that assigns the masterfile to every text-file
4b. Reading the content of every text-file
4c. Aggregate data
5. Save the aggregated data to another source. FOCUS-File(s)? Database?

I would appreciate any ideas, tips and alternatives on this "challenge".

Thank you!

JP

This message has been edited. Last edited by: Kerry,


-----
WebFOCUS 7.6.4 on Windows Server 2003 with Oracle DB / MS SQL Server
 
Posts: 21 | Location: Essen, Germany | Registered: January 30, 2009Report This Post
<JG>
posted
Try this as a basis
 
APP MAP DATASTORE C:\TEMP\
APP PRENDPATH DATASTORE
-RUN
APP QUERY DATASTORE HOLD
-RUN
TABLE FILE FOCAPPQ
PRINT FILENAME
WHERE UPCASE(70, FILENAME, 'A70')  CONTAINS '.TXT';
ON TABLE SAVE AS FILELIST
END
-RUN
FILEDEF RAWDATA DISK c:\TEMP\RAWDATA.FTM (APPEND
-SET &LOOPER=&LINES;
-REPEAT ENDREPEAT &LOOPER TIMES
-READ FILELIST NOCLOSE &FILENAME.A70.
APP FI INFILE DISK BASEAPP/&FILENAME
-RUN
TABLE FILE INFILE
PRINT WHATEVER
ON TABLE HOLD AS RAWDATA FORMAT ALPHA
END
-RUN
-ENDREPEAT
-*
TABLE FILE RAWDATA
SUM WHATEVER
BY WHATEVER
ON TABLE HOLD FORMAT FOCUS
END
 
Report This Post
Waz
Expert
posted Hide Post
I did a quick search for EDID4, and found that is seems to be SAP related.

Have you asked IBI if WebFOCUS can read these files ?


Waz...

Prod:WebFOCUS 7.6.10/8.1.04Upgrade:WebFOCUS 8.2.07OS:LinuxOutputs:HTML, PDF, Excel, PPT
In Focus since 1984
Pity the lost knowledge of an old programmer!

 
Posts: 6347 | Location: 33°49'23.0"S, 151°11'41.0"E | Registered: October 31, 2006Report This Post
Jean-Pierre
Member
posted Hide Post
Thank you JG for the code, I will check this out!

@ Waz

I did not ask IB, since EDID4 is the SAP IDOC format afaik. We receive the data as pure text-files and are about to read these as text-files. Afaik this format is supported by the SAP/R3-Adapter, but the customer is not going to purchase this adapter for this case.


-----
WebFOCUS 7.6.4 on Windows Server 2003 with Oracle DB / MS SQL Server
 
Posts: 21 | Location: Essen, Germany | Registered: January 30, 2009Report This Post
Waz
Expert
posted Hide Post
quote:
customer is not going to purchase this adapter


I can completely understand your situation, there are many thing I would like, but cannot justify.


Waz...

Prod:WebFOCUS 7.6.10/8.1.04Upgrade:WebFOCUS 8.2.07OS:LinuxOutputs:HTML, PDF, Excel, PPT
In Focus since 1984
Pity the lost knowledge of an old programmer!

 
Posts: 6347 | Location: 33°49'23.0"S, 151°11'41.0"E | Registered: October 31, 2006Report This Post
  Powered by Social Strata  

Read-Only Read-Only Topic

Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [CLOSED] Help with reading multiple txt-files from folder (EDID4-format)

Copyright © 1996-2020 Information Builders