We're doing a lot of custom ETL's these days and we keep running into questions like:
- "How many agents do we have available for jobs right now?", or
- "How much memory do we still have available on our (remote) server?"

Obviously, we could, as persons, obtain that information from the webconsole. The problem is that many of those processes are scheduled automatically, often when we aren't even around (nothing beats night-time for heavy data-processing).

The intent here is to build some smarts into our ETL processes, so that they don't start more jobs than we have agents to accept them and that we don't run our servers out of virtual memory (FOC1165) by starting more jobs when we're already low - which doesn't entirely prevent that issue of course, but it should reduce occurrences.

Is there something like that? I imagine IBI would use that internally to populate the information in the webconsole...

I did search the documentation, but that didn't turn up anything useful.

[Edit: Odd to find this topic closed after returning from my holidays]

This message has been edited. Last edited by: FP Mod Chuck,

Wep

The Enterprise Usage Monitor may be the answer but I don't have it installed to tell if it does everything you want. It is relatively new and was developed by Mark Nesson. Outside of that I do know that you can set up agent queuing and set thresholds for the number of agents allowed to be active at any time which sounds like what you want as well.

The real issue you will have is you now have to have an external process to control the submission of the ETL jobs and their dependencies.

I built a process much like this for a customer quite a few years ago. We used iWay Service Manager to implement a correlated submission process. We had over 1000 jobs that had to run on any given night. We also had 6 ETL servers to submit to. The process would direct the job to the server we felt had the best capacity. The key for us was that the ETL jobs had to update the correlation tables with the status at various points. We relied on the that status, as well as job history, to determine the best server.

That being said, IB is currently implanting Clustering/Failover for ETL. I believe that it will submit sub-flows to the cluster, so the job will always be directed to the server with the best "current" capacity. You might want to look into that.

Hey, Wep.

One feature I've played around with in the past is using the "-showagents" option with edastart. It's not the prettiest output, but it will give a list of all the agents and the corresponding web console info.

@Chuck: We already enabled queueing for these ETL jobs.
One of our problems is that some of those ETL jobs leave no trace of even an initial connection.

That seems to imply that either IIS (we're in the process of moving to Linux) just dropped the connection w/o even so much as logging it did that (which I totally suspect it does), or that our ETL client (Apache Nifi) is not always initiating the connection.

We need to add logging of connection failures (and other errors) on the Nifi-side, but the external hiry who implemented that blamed WebFOCUS and flat out refused to "add logging for BS that wasn't Nifi's fault"...

But even then, that doesn't help us; we'd just know what went wrong. If we want to fix that, we either need to resend the job when we detect a failure (but when? how often should we try? etc.), or we shouldn't send jobs when the server is already busy. In either case, we need information to base our decisions on, hence the question.

The other issue is that agens on our remote server in India sometimes hang on to (much) more allocated memory than they need, starving other agents that get jobs that need that memory. That causes jobs to get aborted on the remote server, that can't simply be retried as they still use that same agent that's just going to run out of memory again on the same batch.

@dhagen: We have an external tool that is initiating the ETL's and we need to throttle that somehow. That seems to be pretty much what you describe.

We do have only one ETL server and it does our reporting as well. We are dealing with a manageable amount of data, we just need to be able to manage how much we process in parallel.

Frankly, I have no idea what iWay Service Manager is or does. I'm pretty sure though that it'll cost money in an amount that our management is just going to say "no" to. The guys in India, and especially any layer of management we need to go through, does not (want to) understand the scope of the problem nor the qualities that IBI's products bring us over competitive products that seem cheaper on paper (we recently got MS PauperBI for self-service reporting, for example). Frankly, they frown upon our usage of WebFOCUS (I have yet to hear a valid reason for that) and they want to get rid of it, except that our entire global R&D department will start screaming murder if they do...
It's all politics, but that's what we have to deal with.

@dougshartzer: That looks like the start of a possible solution. I'll look into that. Do you happen to know whether that reports results of the already running instance?