Med: daemons: Don't add repeated I_PE_CALC messages to the fsa queue.

Let's say you have a two node cluster, node1 and node2.  For purposes of
testing, it's easiest if you use fence_dummy instead of a real fencing
agent as this will fake fencing happening but without rebooting the node
so you can see all the log files.

Assume the DC is node1.  Now do the following on node2:

- pcs node standby node1
- pcs resource defaults update resource-stickiness=1
- for i in $(seq 1 300); do echo $i; pcs resource create dummy$i ocf:heartbeat:Dummy --group dummy-group; done
- pcs node unstandby node1

It will take a long time to create that many resources.  After node1
comes out of standby, it'll take a minute or two but eventually you'll
see that node1 was fenced.  On node1, you'll see a lot of transition
abort messages happen.  Each of these transition aborts causes an
I_PE_CALC message to be generated and added to the fsa queue.  In my
testing, I've seen the queue grow to ~ 600 messages, all of which are
exactly the same thing.

These messages are fed into controld's glib event loop at
G_PRIORITY_HIGH, while the presence of regular IPC messages trigger at
G_PRIORITY_DEFAULT.  Thus, the fsa messages take priority.  It takes a
while for controld to process all these high priority messages, during
which time it is unable to read anything out of its IPC backlog.

based continues to attempt to send IPC events to controld but is unable
to do so, so the backlog continues to grow.  Eventually, the backlog
reaches that 500 message threshold without anything having been read by
controld, which triggers the eviction process.

There doesn't seem to be any reason for all these I_PE_CALC messages to
be generated.  They're all exactly the same, they don't appear to be
tagged with any unique data tying them to a specific query, and their
presence just slows everything down.

Thus, the fix here is very simple: if the latest message in the queue is
an I_PE_CALC message, just don't add another one.  We could also make
sure there's only ever one I_PE_CALC message in the queue, but there
could potentially be valid reasons for there to be multiple interleaved
with other message types.  I am erring on the side of caution with this
minimal fix.

Related: RHEL-76276

Author: clumens

daemons/controld/controld_te_utils.c

@@ -497,6 +497,14 @@ abort_transition_graph(int abort_priority, enum pcmk__graph_next abort_action,
             controld_stop_transition_timer();
             controld_start_transition_timer();
         } else {
+            fsa_data_t *message = g_queue_peek_tail(controld_globals.fsa_message_queue);
+
+            if (message != NULL && message->fsa_input == I_PE_CALC) {
+                do_crm_log(level, "Last item in fsa queue is I_PE_CALC, not "
+                           "adding another");
+                return;
+            }
+
             register_fsa_input(C_FSA_INTERNAL, I_PE_CALC, NULL);
         }
         return;