Focal Point Banner
Community Center Education Summit Technical Support User Groups
Let's Get Social!

Facebook Twitter LinkedIn YouTube
Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [CLOSED] Can we TABLE agent statistics?
Go
New
Search
Notify
Tools
Reply
  
[CLOSED] Can we TABLE agent statistics?
 Login/Join
 
Virtuoso
posted
We're doing a lot of custom ETL's these days and we keep running into questions like:
- "How many agents do we have available for jobs right now?", or
- "How much memory do we still have available on our (remote) server?"

Obviously, we could, as persons, obtain that information from the webconsole. The problem is that many of those processes are scheduled automatically, often when we aren't even around (nothing beats night-time for heavy data-processing).

The intent here is to build some smarts into our ETL processes, so that they don't start more jobs than we have agents to accept them and that we don't run our servers out of virtual memory (FOC1165) by starting more jobs when we're already low - which doesn't entirely prevent that issue of course, but it should reduce occurrences.

Is there something like that? I imagine IBI would use that internally to populate the information in the webconsole...

I did search the documentation, but that didn't turn up anything useful.

[Edit: Odd to find this topic closed after returning from my holidays]

This message has been edited. Last edited by: FP Mod Chuck,


WebFOCUS 8.1.03, Windows 7-64/2008-64, IBM DB2/400, Oracle 11g & RDB, MS SQL-Server 2005, SAP, PostgreSQL 11, Output: HTML, PDF, Excel 2010
: Member of User Group Benelux :
 
Posts: 1652 | Location: Enschede, Netherlands | Registered: August 12, 2010Reply With QuoteReport This Post
Virtuoso
posted Hide Post
Wep

The Enterprise Usage Monitor may be the answer but I don't have it installed to tell if it does everything you want. It is relatively new and was developed by Mark Nesson. Outside of that I do know that you can set up agent queuing and set thresholds for the number of agents allowed to be active at any time which sounds like what you want as well.


Thank you for using Focal Point!

Chuck Wolff - Focal Point Moderator
WebFOCUS 7x and 8x, Windows, Linux All output Formats
 
Posts: 1694 | Location: Customer Support | Registered: April 12, 2005Reply With QuoteReport This Post
Virtuoso
posted Hide Post
The real issue you will have is you now have to have an external process to control the submission of the ETL jobs and their dependencies.

I built a process much like this for a customer quite a few years ago. We used iWay Service Manager to implement a correlated submission process. We had over 1000 jobs that had to run on any given night. We also had 6 ETL servers to submit to. The process would direct the job to the server we felt had the best capacity. The key for us was that the ETL jobs had to update the correlation tables with the status at various points. We relied on the that status, as well as job history, to determine the best server.

That being said, IB is currently implanting Clustering/Failover for ETL. I believe that it will submit sub-flows to the cluster, so the job will always be directed to the server with the best "current" capacity. You might want to look into that.


"There is no limit to what you can achieve ... if you don’t care who gets the credit." Roger Abbott
 
Posts: 1101 | Location: Toronto, Ontario | Registered: May 26, 2004Reply With QuoteReport This Post
Member
posted Hide Post
Hey, Wep.

One feature I've played around with in the past is using the "-showagents" option with edastart. It's not the prettiest output, but it will give a list of all the agents and the corresponding web console info.


WebFOCUS 8.1.0.5
 
Posts: 2 | Location: Raleigh, NC | Registered: March 25, 2013Reply With QuoteReport This Post
Virtuoso
posted Hide Post
@Chuck: We already enabled queueing for these ETL jobs.
One of our problems is that some of those ETL jobs leave no trace of even an initial connection.

That seems to imply that either IIS (we're in the process of moving to Linux) just dropped the connection w/o even so much as logging it did that (which I totally suspect it does), or that our ETL client (Apache Nifi) is not always initiating the connection.

We need to add logging of connection failures (and other errors) on the Nifi-side, but the external hiry who implemented that blamed WebFOCUS and flat out refused to "add logging for BS that wasn't Nifi's fault"...

But even then, that doesn't help us; we'd just know what went wrong. If we want to fix that, we either need to resend the job when we detect a failure (but when? how often should we try? etc.), or we shouldn't send jobs when the server is already busy. In either case, we need information to base our decisions on, hence the question.

The other issue is that agens on our remote server in India sometimes hang on to (much) more allocated memory than they need, starving other agents that get jobs that need that memory. That causes jobs to get aborted on the remote server, that can't simply be retried as they still use that same agent that's just going to run out of memory again on the same batch.


WebFOCUS 8.1.03, Windows 7-64/2008-64, IBM DB2/400, Oracle 11g & RDB, MS SQL-Server 2005, SAP, PostgreSQL 11, Output: HTML, PDF, Excel 2010
: Member of User Group Benelux :
 
Posts: 1652 | Location: Enschede, Netherlands | Registered: August 12, 2010Reply With QuoteReport This Post
Virtuoso
posted Hide Post
@dhagen: We have an external tool that is initiating the ETL's and we need to throttle that somehow. That seems to be pretty much what you describe.

We do have only one ETL server and it does our reporting as well. We are dealing with a manageable amount of data, we just need to be able to manage how much we process in parallel.

Frankly, I have no idea what iWay Service Manager is or does. I'm pretty sure though that it'll cost money in an amount that our management is just going to say "no" to. The guys in India, and especially any layer of management we need to go through, does not (want to) understand the scope of the problem nor the qualities that IBI's products bring us over competitive products that seem cheaper on paper (we recently got MS PauperBI for self-service reporting, for example). Frankly, they frown upon our usage of WebFOCUS (I have yet to hear a valid reason for that) and they want to get rid of it, except that our entire global R&D department will start screaming murder if they do...
It's all politics, but that's what we have to deal with.

@dougshartzer: That looks like the start of a possible solution. I'll look into that. Do you happen to know whether that reports results of the already running instance?


WebFOCUS 8.1.03, Windows 7-64/2008-64, IBM DB2/400, Oracle 11g & RDB, MS SQL-Server 2005, SAP, PostgreSQL 11, Output: HTML, PDF, Excel 2010
: Member of User Group Benelux :
 
Posts: 1652 | Location: Enschede, Netherlands | Registered: August 12, 2010Reply With QuoteReport This Post
  Powered by Social Strata  
 

Focal Point    Focal Point Forums  Hop To Forum Categories  WebFOCUS/FOCUS Forum on Focal Point     [CLOSED] Can we TABLE agent statistics?

Copyright © 1996-2018 Information Builders, leaders in enterprise business intelligence.