We have 2 virtual machines running Linux under VM on a z90 mainframe. On machine has the WebFOCUS reporting server and it runs one version of Suse Linux. The other machine contains the production Oracle database. This machine was upgraded to a newer version of Suse Linux and Oracle 10g this past weekend. Since this upgrade, we have several long running batch jobs that run for a while and then just stop with no errors. It is almost as if there was a -EXIT in the code. The strange thing is that they stop at the same place. Has anyone ever encountered something like this or do you have any suggestions on how to find out what is going on?

Where is it stopping? Are you running shell commands? Can you upload code?

Sayed

The job has been running just fine for a year. I'm 99.9% positive, it has nothing to do with the code. This started on Monday, the day after the Oracle/Linux upgrade went live. There was no change to the WebFOCUS installation (we were already running the Oracle 10g client against the old Oracle 9i database). I think I am looking for some kind of system setting but my Linux guys and DBAs can't think of anything that would cause this. Below is the part of the message file. As you can see, I use -TYPE commands to document progress.

 =======================================
 ===   Starting 200706 at 00.05.02   ===
 =======================================
 
   If month is being rerun, delete month
 
0 TRANSACTIONS:         TOTAL =     1  ACCEPTED=     1  REJECTED=     0
  SEGMENTS:             INPUT =     0  UPDATED =     0  DELETED = 12375
.
.
.
   If one exists, replace assignment county with County of Jurisdiction
 
0 NUMBER OF RECORDS IN TABLE=    13819  LINES=  13819
  
0 NUMBER OF RECORDS IN TABLE=    13819  LINES=  11329
.
.
.
 =======================================
 ===   Starting 200707 at 00.15.31   ===
 =======================================
 
   If month is being rerun, delete month
 
0 TRANSACTIONS:         TOTAL =     1  ACCEPTED=     1  REJECTED=     0
  SEGMENTS:             INPUT =     0  UPDATED =     0  DELETED = 12478
.
.
.
   If one exists, replace assignment county with County of Jurisdiction
 
0 NUMBER OF RECORDS IN TABLE=    13633  LINES=  13633
  
0 NUMBER OF RECORDS IN TABLE=    13633  LINES=  11284
.
.
.

This repeats until...

 =======================================
 ===   Starting 200801 at 01.20.55   ===
 =======================================
 
   If month is being rerun, delete month
 
0 TRANSACTIONS:         TOTAL =     1  ACCEPTED=     1  REJECTED=     0
  SEGMENTS:             INPUT =     0  UPDATED =     0  DELETED = 11736
.
.
. 
   If one exists, replace assignment county with County of Jurisdiction
 

At this point the job just stops. It needs to finish January and run for 4 more months of data. The previous night, the job stopped at this exact point in the program but while processing Dec 2007. I have at least one other job the is showing simliar behavior. Both are long running (over 1.5 hrs) batch jobs.

Did I say that I hate wierd problems?

BTW the verticle dots are places where the job is running, I just did not copy the message file output.

Could it be that the upgrade came with a default resource limit? Maybe something as simple as computer time. You said it's stopping in the same place. If it's some sort of resource limit, then it would make sense that it's stopping in the same place.

Did you try just running January? Maybe some bad data...

Jessica, I have already the systems group that question and they said no. I even asked if running 2 different versions of Suse Linux might cause this and they said no.

Danny, yes, I restarted the job in Jan and it ran to completion. Even if there was bad data, seems I would get an Oracle or WebFOCUS error. There's nothing.

The jobs run too long to turn on traces and the Reporting Server log files show nothing, at least nothing I can see.

Wait. I was re-reading your posts. You say at the beginning that it always stops at the same place. Then you show a stop in January but that previously it stopped in December.
It really looks as if the system bumps you off because you are "overtime".
If you run from December onwards, what happens?

Danny, if I start the job in Jan 08 instead of Jun 07, it finishes. I really think there is something fishy going on. I have looked at the RC logs and found the following in every job that just up and stops:

Warning: /opt/ibi/713/ibi/srv71/source: No such file or directory.
(no debugging symbols found)...Using host libthread_db library "/lib64/tls/libthread_db.so.1".
Core was generated by `/opt/ibi/713/ibi/srv71/home/bin/tscom3.out -traceoff .
Program terminated with signal 11, Segmentation fault.
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libntdump.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libntdump.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libqopsys.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libqopsys.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libmbx.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libmbx.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libr1fns.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libr1fns.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libnlsp.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libnlsp.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/tls/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/tls/libm.so.6
Reading symbols from /lib64/tls/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libtokh.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libtokh.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libcbh.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libcbh.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libemgsrv.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libemgsrv.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libntdm.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libntdm.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libkkdo.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libkkdo.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libnlssys.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libnlssys.so
Reading symbols from /lib/ld64.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/ld64.so.1
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libfocutl.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libfocutl.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libcomph.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libcomph.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libntctx.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libntctx.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libqcvtin.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libqcvtin.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libfocini.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libfocini.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libcorfoc.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libcorfoc.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libibiuth.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libibiuth.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libmaspar.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libmaspar.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libptree.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libptree.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libtabutl.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libtabutl.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libgntint.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libgntint.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libgrmmas.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libgrmmas.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libhffunc.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libhffunc.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libio3270.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libio3270.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libmfp.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libmfp.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libncache.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libncache.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libnthr.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libnthr.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libparser.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libparser.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libqmfint.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libqmfint.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libutl.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libutl.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libxfcode.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libxfcode.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libxsrb.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libxsrb.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libgrmdba.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libgrmdba.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libgrmdef.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libgrmdef.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libufaapi.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libufaapi.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libxfutil.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libxfutil.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libioh.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libioh.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libnwh.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libnwh.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libpth.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libpth.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libr1sec.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libr1sec.so
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libngfl.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libngfl.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/librth.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/librth.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libintcbs.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libintcbs.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libntmas.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libntmas.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libntsn.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libntsn.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libcmh.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libcmh.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libmfdp.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libmfdp.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libntdb.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libntdb.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libimgdir.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libimgdir.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libdmh.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libdmh.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libdqsqlf.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libdqsqlf.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libor8foc.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libor8foc.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libqstrac.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libqstrac.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libor8ps10.so...(no debugging symbols found)...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libor8ps10.so
Reading symbols from /opt/ibi/10gClient/lib/libclntsh.so.10.1...done.
Loaded symbols for /opt/ibi/10gClient/lib/libclntsh.so.10.1
Reading symbols from /opt/ibi/10gClient/lib/libnnz10.so...done.
Loaded symbols for /opt/ibi/10gClient/lib/libnnz10.so
Reading symbols from /lib64/tls/libpthread.so.0...done.
Loaded symbols for /lib64/tls/libpthread.so.0
Reading symbols from /lib64/libnsl.so.1...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libtabp.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libtabp.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libfocscr.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libfocscr.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libfocapi.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libfocapi.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libsndh.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libsndh.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libr1ine2.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libr1ine2.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libntmspr.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libntmspr.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libsqlds.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libsqlds.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libcdtab.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libcdtab.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libvsamx.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libvsamx.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libvvset.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libvvset.so
Reading symbols from /opt/ibi/713/ibi/srv71/home/bin/libfoc.so...done.
Loaded symbols for /opt/ibi/713/ibi/srv71/home/bin/libfoc.so
#0 0x0000000000000000 in ?? ()
(gdb) #0 0x0000000000000000 in ?? ()
#1 0x0000020001eff7f2 in ttcdrv ()
from /opt/ibi/10gClient/lib/libclntsh.so.10.1
#2 0x0000020001d7c85c in nioqwa ()
from /opt/ibi/10gClient/lib/libclntsh.so.10.1
#3 0x0000020001b8d382 in upirtrc ()
from /opt/ibi/10gClient/lib/libclntsh.so.10.1
#4 0x0000020001b1342c in kpurcsc ()
from /opt/ibi/10gClient/lib/libclntsh.so.10.1
#5 0x0000020001ab8f0e in kpuexecv8 ()
from /opt/ibi/10gClient/lib/libclntsh.so.10.1
#6 0x0000020001abbcaa in kpuexec ()
from /opt/ibi/10gClient/lib/libclntsh.so.10.1
#7 0x0000020001b4be28 in OCIStmtExecute ()
from /opt/ibi/10gClient/lib/libclntsh.so.10.1
#8 0x000002000186ac66 in jjexec ()
from /opt/ibi/713/ibi/srv71/home/bin/libor8foc.so
#9 0x000002000185a22c in jjsql ()
from /opt/ibi/713/ibi/srv71/home/bin/libor8foc.so
#10 0x0000020001874f00 in or8foc ()
from /opt/ibi/713/ibi/srv71/home/bin/libor8foc.so
#11 0x0000020002b97ba4 in dsqopn ()
from /opt/ibi/713/ibi/srv71/home/bin/libsqlds.so
#12 0x0000020002b98ee0 in dget1 ()
from /opt/ibi/713/ibi/srv71/home/bin/libsqlds.so
#13 0x0000020002b9ce82 in dgetm ()
from /opt/ibi/713/ibi/srv71/home/bin/libsqlds.so
#14 0x0000020002b9c178 in dgetm ()
from /opt/ibi/713/ibi/srv71/home/bin/libsqlds.so
#15 0x0000020002b9acb4 in dgetm ()
from /opt/ibi/713/ibi/srv71/home/bin/libsqlds.so
#16 0x0000020002b8491c in sqlds ()
from /opt/ibi/713/ibi/srv71/home/bin/libsqlds.so
#17 0x0000020002b841e4 in sqlds ()
from /opt/ibi/713/ibi/srv71/home/bin/libsqlds.so
#18 0x0000020000bb1510 in gntint ()
from /opt/ibi/713/ibi/srv71/home/bin/libgntint.so
#19 0x0000020000bb0484 in gntint ()
from /opt/ibi/713/ibi/srv71/home/bin/libgntint.so
#20 0x0000020000bad3ee in gntint ()
from /opt/ibi/713/ibi/srv71/home/bin/libgntint.so
#21 0x00000200008c3d62 in getpth ()
from /opt/ibi/713/ibi/srv71/home/bin/libcorfoc.so
#22 0x00000200008c0cca in record ()
from /opt/ibi/713/ibi/srv71/home/bin/libcorfoc.so
#23 0x00000200007e23e2 in ptbccsa ()
from /opt/ibi/713/ibi/srv71/home/bin/libcorfoc.so
#24 0x00000200007e4542 in gosort ()
from /opt/ibi/713/ibi/srv71/home/bin/libcorfoc.so
#25 0x00000200006661ae in tabctl ()
from /opt/ibi/713/ibi/srv71/home/bin/libcorfoc.so
#26 0x0000020002a1df64 in ts3table ()
from /opt/ibi/713/ibi/srv71/home/bin/libtabp.so
#27 0x0000020001309cf6 in rttab ()
from /opt/ibi/713/ibi/srv71/home/bin/librth.so
#28 0x000002000131601a in rtgo ()
from /opt/ibi/713/ibi/srv71/home/bin/librth.so
#29 0x00000200011f58c4 in nwsrvsig ()
from /opt/ibi/713/ibi/srv71/home/bin/libnwh.so
#30 0x00000200011fe6f6 in nwloop ()
from /opt/ibi/713/ibi/srv71/home/bin/libnwh.so
#31 0x00000200011fd574 in nwgo ()
from /opt/ibi/713/ibi/srv71/home/bin/libnwh.so
#32 0x0000000080003012 in ?? ()
#33 0x0000000080001ed6 in ?? ()
#34 0x00000200002678d4 in __libc_start_main () from /lib64/tls/libc.so.6
#35 0x0000000080001dc2 in ?? ()
(gdb) *** ibisnap.log from /opt/ibi/apps/temp/cpsl1, exec=/opt/ibi/713/ibi/srv71/home/bin/tscom3.out
Task finished.

The warning is correct, there is no such folder as /opt/ibi/713/ibi/srv71/source, never has been. I don't understand why this folder is being searched for. The jobs that are scheduled through RC and finish like they should, even this job prior to last weekend, don't have this warning or all the messages that follow. I have sent this to my systems groups and IBI hoping they can tell me what is going on. Tonight I'm going to try the job as deferred through the server console.

Just thought I'd update the with the resolution. For the version of Linux being used on the Linux guest where Oracle is, WebFOCUS needed the 10.2 version of the Oracle Client and we were using 10.1. Since the update, there hasn't been one job that stopped. Everything is back to normal.