More on workload skewing

Based on a true story, the names have been omitted to protect the innocent, guilty, and the vaguely interested.

Anyone who has talked to me is aware of my interest in workload skewing that can be a natural part a shared queue environment.  I have been involved in a number of situations, recommending mitigation techniques to a number of customers.  But as with anything else, there is always more to learn about the causes, effects and interaction between the CF and MQ.

As a bit of background, some customers have observed what the development lab have termed “over notification” when using shared queues.  Simply there are multiple application instances that have an MQGET with a wait active on a particular queue, when the Coupling Facility code detects a transition from empty to non-empty – a message has been put to a previously empty queue – and notifies MQ of the transition, MQ in turn notifies all the application instances that are waiting.  All of the applications attempt to get messages, which if there is only one message means that one instance gets the message but there is CPU associated with all those other instances trying to get a message.

Over notification works to the advantage of some workloads, where the arrival of one message is typically the start of a deluge of traffic.  This is particularly true when the queue is not likely to return to a zero depth for a long period of time, if the CF transition is never re-driven those ‘extra’ instances will never be notified that there are messages to be processed.

Reducing over notification can be an advantage when there are CPU constraints, there is less CPU used when there are fewer notifications going out and there is less contention between instances that can consume a limited number of messages.  The MQ change team has developed a tuning option that helps control over notification.

It gets interesting when conditions change.  In this particular instance, the customer had used the tuning option to control over notification in their production environment because one application was consuming a significant amount of CPU when the primary shared queue was basically being trickle fed most of the time.

However, when starting to test a different application, they did not implement the over notification tuning option.  They initially could not drive a lot of workload, were running into CPU waits, and when examining the SMF data we found that they had a significant level of latching on type 24 latches – which is the primary symptom of churn on the ‘get wait’ chain often seen with the over notification.   The tuning option was turned on, and greater throughput was achieved.  We saw a significant reduction in the type 24 latching.

Additional capacity was added to the test environment to mimic their production environment and the situation changed.  As more load was added, the LPAR that had the faster service times from the coupling facility was consuming 90% of the work, and we were not seeing activity on the second LPAR no matter how high the workload was driven,  In addition, the inbound queue depths were building up, and messages were remaining on the queue for longer periods.  At the same time there were a lot of application instances that did not appear to be active.

The development labs in Poughkeepsie and Hursley has a conversation about this and found that because of the suppression of notification on the part of MQ and the lack of the coupling facility transition (the queue never went back to 0 under the constant load) notification was causing the bottleneck.  Turning off the tuning option reduce the skewing to a more expected 60/40 split based on the difference in service times, at the cost of some CPU because the reintroduced over notification was creating some of the original get-waiting churn and contention for the messages.

We anticipate that the over notification ‘churn’ can be controlled in the test environment when a disciplined study of the number of active application instances can be done.  Perhaps that will help them reach equilibrium between CPU consumption and message throughput.

Workload  matters.

 

A suggestion for a control queue bufferpool and pageset

In the past few months I have heard of a few incidents caused by the sudden over use of a bufferpool and pageset that resulted in some significant slowdowns, application outages, or queue manager shutdowns.  The incidents are similar in that there was an unexpected surge in messages arriving on a queue, which ultimately caused the pageset to go into expansion an eventually to fill.  The problem was that the bufferpool and pageset were also used for some system operational queues, like the SYSTEM.COMMAND.INPUT queue.

The SYSTEM.COMMAND.INPUT queue defaults to using the SYSVOLAT storage class, pageset 3.  This pageset is also used by the REMOTE storage class, which is the default STGCLASS value for the SYSTEM.CLUSTER.TRANSMIT.QUEUE and for other transmission queues.  While we have long recommended that the SYSTEM.CLUSTER.TRANSMIT.QUEUE be placed in a dedicated storage class, using a devoted bufferpool and pageset, that recommendation is often much easier to make than to implement.  Especially before MQ V8.

The problem arises when the pageset fills and commands to the queue manager cannot be put on the command input queue.  And it can get really awful from that point on.

We’ve also recently seen this when a customer used the REMOTE storage class for their dead letter queue.  The queue manager had not had a problem for the 15 years of its production life, until there was a surge of messages placed on the DLQ because of an application problem.  Again, the underlying pageset filled up and no commands could be issued.

The recommendation is simple, isolate the SYSTEM.COMMAND.INPUT and a couple of other control queues to a separate storage class that points to an unique pageset and bufferpool to avoid the potential for a problem with an application, a channel, or the cluster to impact the ability to control the queue managers.

Recommended queues for the new storage class:
SYSTEM.COMMAND.INPUT
SYSTEM.CHANNEL.INITQ
SYSTEM.CLUSTER.COMMAND.QUEUE

Last two IMQ08 sessions for 2016

Hi folks!

First, we have two sessions with seats available in 2016.

Chicago – November 29 thru December 1
San Francisco – December 13 thru 15

Please note that while the workshop number is still IMQ08 so far, the labs are running against V 9 code and are gradually updating labs, etc. as we have discovered or uncovered some interesting changes in some places!

If you would like to register, please contact your IBM representative or Chris Griego (cgriego@us.ibm.com), Shalawn King (shalawn@us.ibm.com), my teammate Mitch (mitchj@us.ibm.com) or me (elkinsc@us.ibm.com).

We are starting to plan our 2017 adventures, I will post as we begin scheduling.

Lyn

IBM Technical University in Baltimore

Greetings all,

For those of you in the Baltimore/Washington DC area, there is an IBM Technical University event being held at the Hilton. This covers many z-centric topics, including 3 sessions on MQ.

The site for the event is:

https://www-304.ibm.com/services/learning/ites.wss/zz-en?pageType=page&c=J603289R25742I89

This is NOT a marketing event, but a technical event, you can see the complete agenda here:

https://ibmtechu.com/cgi-bin/cms/printA3.cgi?myevent=Baltimore2016

There is a fee for the event, but if your company is sending more than one person discounts are available. Please contact me (elkinsc@us.ibm.com) if your company is sending more than one person and you need the correct discount code. Or contact your local IBM account representative for assistance.

Thanks,
Lyn