Optimize `isBusy` logic by incorporating mempool transactions and enabling configurable `MAX_TRX_SIZE`

[xxo1shine]

Summary

TRON nodes currently determine whether they are in a busy state based only on the size of the transaction processing thread pool queues, without considering the number of transactions already cached in the mempool.
By incorporating both the thread pool queues and the mempool transaction count into the evaluation, the node can more accurately reflect its current load. This enables better control over transaction intake and improves the stability of resource management.
In addition, it is recommended that MAX_TRX_SIZE be configurable, allowing flexible adjustment based on the node's deployment environment.

Root Cause

The current isBusy() logic is as follows:

public boolean isBusy() {
   return queue.size() + smartContractQueue.size() > MAX_TRX_SIZE;
}

This logic only accounts for:

• The standard transaction processing queue (queue)
• The smart contract processing queue (smartContractQueue)

It does not include transactions already stored in the mempool.
In the actual processing flow, once a transaction is executed successfully, it enters the mempool and waits to be packed into a block. Therefore, the overall node load is reflected not only by processing queues but also by the size of cached transactions.
The following mempool-related queues are currently not included in the load calculation:

• pendingTransactions
• rePushTransactions
• pushTransactionQueue

As a result, the load assessment is incomplete.

Reproduction

• Start a node and connect it to the P2P network
• Continuously send transactions to the node so they accumulate in the mempool
• Control the thread pool processing speed to keep queue and smartContractQueue at relatively low levels
• Observe that isBusy() still reports the node as not busy
• The node continues accepting new transactions while the total cached transaction volume keeps increasing

Impact

• Incomplete load evaluation: mempool transactions are not considered
• Inaccurate resource estimation: the total transaction load on the node is underestimated
• Suboptimal intake control: the node may continue accepting new transactions even when the mempool is already large

Suggested Fix

Include mempool-related transaction counts in the isBusy() calculation and make MAX_TRX_SIZE configurable.
Example:

public boolean isBusy() {
   return queue.size()
       + smartContractQueue.size()
       + manager.getPendingSize()
       + manager.getRePushTransactions().size()
       + manager.getPushTransactionQueue().size()
       > MAX_TRX_SIZE; // configurable
}

By introducing these additional metrics and making MAX_TRX_SIZE configurable, the node can achieve a more comprehensive understanding of its load. This enables more accurate transaction intake control and allows operators to tune thresholds based on their deployment environment.

[lxcmyf]

Interesting point. I have a couple of questions here:

• If we want to reflect actual mempool pressure, should pushTransactionQueue really be counted the same way as pendingTransactions / rePushTransactions? It feels a bit more like an ingress queue than packable mempool state.
• There is already maxTransactionPendingSize on the manager side. Is there a reason to introduce another configurable MAX_TRX_SIZE knob instead of aligning with the existing limit?
• Also, getPendingSize() currently includes poppedTransactions too. Do you think that should be part of the busy signal, or should isBusy() only track transactions that are still actively waiting to be processed / packed?

The general direction makes sense, I am just not sure yet what the cleanest boundary is for "busy" here.

[lvs0075]

Seconding @lxcmyf's concerns with concrete code evidence. Manager.pushTransactionQueue.add(trx) paired with the finally removal shows pushTransactionQueue only holds transactions currently in sig-verification + transactionLock acquisition. The same trx is effectively counted by both TransactionsMsgHandler.queue and pushTransactionQueue, so summing them double-counts.

Similarly, Manager.getPendingSize() includes poppedTransactions, which is populated only at Manager inside eraseBlock() during fork reorg — a one-off reorg artifact, not a backpressure signal. I'd scope the mempool term in isBusy() strictly to pendingTransactions + rePushTransactions, matching isTooManyPending().

[xxo1shine]

Thanks for the thoughtful review, @lxcmyf!

On pushTransactionQueue as ingress:
You're right that it's more of an ingress buffer than packable mempool state. That said, a large ingress queue still means the node is already under load — transactions are piling up faster than they can be consumed. Including it captures end-to-end pressure rather than just the mid/late stages of the pipeline. If we want a stricter definition of "mempool pressure," we could drop it, but we'd risk missing the early stage of overload.

On MAX_TRX_SIZE vs maxTransactionPendingSize:
maxTransactionPendingSize is a limit on pendingTransactions alone. The composite sum across all queues operates at a different scale, so reusing that threshold directly would either be too tight or too loose. A dedicated MAX_TRX_SIZE (configurable) gives operators a single knob tuned for the aggregate load signal, independent of individual queue caps.

On poppedTransactions in the busy signal:
getPendingSize() currently includes poppedTransactions, which are transactions temporarily displaced during block forging or chain switching. These will be re-pushed, so they represent real in-flight work. Excluding them would require a separate accessor. For simplicity we kept the existing getPendingSize() semantics, but if the consensus is that popped transactions shouldn't contribute to the busy signal, we can expose a narrower method.

[xxo1shine]

@lvs0075, thanks for the in-depth analysis.

Regarding poppedTransactions: your observation is correct — it is only populated during fork reorg (eraseBlock()) and is a one-off reorg artifact rather than a sustained backpressure signal.

Regarding the double-counting concern with pushTransactionQueue: after verification, transactions in pushTransactionQueue (including both API-broadcast and P2P-received transactions) will not simultaneously exist in queue — the two are non-overlapping, so there is no double-counting. Therefore isBusy() retains the logic proposed in the issue.

[lvs0075]

@lvs0075, thanks for the in-depth analysis.

Regarding poppedTransactions: your observation is correct — it is only populated during fork reorg (eraseBlock()) and is a one-off reorg artifact rather than a sustained backpressure signal.

Regarding the double-counting concern with pushTransactionQueue: after verification, transactions in pushTransactionQueue (including both API-broadcast and P2P-received transactions) will not simultaneously exist in queue — the two are non-overlapping, so there is no double-counting. Therefore isBusy() retains the logic proposed in the issue.

They certainly cannot exist simultaneously; the queue will be cleared and handed over to the thread for processing, and only then will the transaction enter the pushTransactionQueue queue.

[xxo1shine]

@lvs0075, thanks for the in-depth analysis.
Regarding poppedTransactions: your observation is correct — it is only populated during fork reorg (eraseBlock()) and is a one-off reorg artifact rather than a sustained backpressure signal.
Regarding the double-counting concern with pushTransactionQueue: after verification, transactions in pushTransactionQueue (including both API-broadcast and P2P-received transactions) will not simultaneously exist in queue — the two are non-overlapping, so there is no double-counting. Therefore isBusy() retains the logic proposed in the issue.

They certainly cannot exist simultaneously; the queue will be cleared and handed over to the thread for processing, and only then will the transaction enter the pushTransactionQueue queue.

@lvs0075 Thanks for the clarification and confirmation.

I’ll proceed with implementing the proposed isBusy() enhancement as described in the issue, including:

• incorporating pendingTransactions, rePushTransactions, and pushTransactionQueue into the load evaluation
• adding MAX_TRX_SIZE as a configurable parameter to allow tuning based on deployment environments

I will also add basic validation to ensure the behavior remains consistent under high-throughput and reorg scenarios.

I’ll prepare a PR for review once the implementation is ready.

[lxcmyf]

Thanks for the detailed explanation.

I think the main remaining concern is less about overlap and more about the semantic boundary of isBusy().

TransactionsMsgHandler.isBusy() is currently checked from InventoryMsgHandler as an admission gate for incoming TRX inventory, so today it is effectively protecting the ingress / validation stage. In that context, queue / smartContractQueue and even pushTransactionQueue are still "pre-mempool" pressure, while pendingTransactions / rePushTransactions are already accepted transactions waiting to be packed.

So even if queue and pushTransactionQueue are non-overlapping, combining all of them under one threshold still mixes different pipeline stages into a single busy signal. That makes the threshold harder to reason about operationally, especially once we also introduce another configurable knob.

The same point applies to getPendingSize(): as we already discussed, it currently includes poppedTransactions, which is not really a steady-state backpressure signal.

My preference would be either:

1. keep isBusy() scoped to ingress / validation pressure only; or
2. if we want an aggregate end-to-end pressure signal, introduce that explicitly with a clearer name and documented relationship to maxTransactionPendingSize, instead of folding mempool and pre-mempool stages into the current isBusy() path.

That would make the boundary much clearer and make operator tuning easier too.

[xxo1shine]

@lxcmyf Thanks for the clarification — I think I now understand your concern around the semantic boundary of isBusy().

If we treat isBusy() strictly as an ingress/validation gate, i.e. the mechanism used by InventoryMsgHandler to decide whether to continue accepting incoming TRX inventory messages, then I agree your interpretation is consistent with the current design intent. In that sense, it primarily reflects pre-mempool / admission pressure (queue, smartContractQueue, pushTransactionQueue).

However, my original concern is slightly different and more focused on overall system resource pressure, especially memory growth.

In the current architecture, even if ingress queues remain small (so isBusy() returns false), transactions can still accumulate significantly in:

• pendingTransactions
• rePushTransactions

These structures are already inside the mempool / post-admission stage, and they directly contribute to JVM heap usage. In such cases, the node may continue accepting new transactions from peers while overall memory usage keeps increasing, because isBusy() does not reflect this part of the system load.

From a system stability perspective, this creates a gap between:

• ingress pressure (what isBusy() currently captures), and
• actual resource pressure (especially memory consumption in mempool)

So I agree that your suggestion is correct in terms of keeping the boundary clean, but I think it only addresses one side of the problem (ingress control), while the original issue is about preventing memory bloat under sustained load.

One possible direction could be to explicitly separate these concerns:

• Keep isBusy() as an ingress / validation-level signal (as you suggested), used by InventoryMsgHandler
• Introduce a separate, explicit metric for overall node load (including mempool size), which can be used for broader admission control or monitoring

This would keep the current semantics clean, while still allowing us to address the memory pressure problem directly.

Curious to hear your thoughts on this separation.

[waynercheung]

After tracing the current code paths, I think the discussion is really about two different backpressure domains that we should avoid collapsing into one signal:

1. ingress / validation pressure: TransactionsMsgHandler.queue, smartContractQueue, and possibly Manager.pushTransactionQueue
2. post-admission / mempool pressure: pendingTransactions + rePushTransactions

TransactionsMsgHandler.isBusy() is currently only consulted by InventoryMsgHandler before accepting TRX inventory, so today it behaves as an ingress admission gate, not a general node-load metric.

A few code observations that may help:

• pushTransactionQueue is transient work inside Manager.pushTransaction(). It is not packable mempool state, but it does represent in-flight validation / transactionLock pressure that queue.size() does not capture. So I can justify counting it for ingress pressure; I would not describe it as mempool pressure.
• getPendingSize() includes poppedTransactions, and poppedTransactions is populated in eraseBlock() during reorg. I would avoid reusing getPendingSize() for steady-state busy/admission logic.
• maxTransactionPendingSize already exists, but it is not currently a peer-ingress guard. It is checked in Wallet.broadcastTransaction(), while the P2P path goes through InventoryMsgHandler -> TransactionsMsgHandler -> TronNetDelegate.pushTransaction() -> Manager.pushTransaction() without consulting isTooManyPending(). So the original concern about mempool growth under sustained P2P load is real.

My preference would be:

• keep isBusy() scoped to ingress / validation pressure; if needed, add pushTransactionQueue.size() there
• add a separate isMempoolOverloaded() / getMempoolPressure() based on pendingTransactions + rePushTransactions only
• decide explicitly whether that second signal is monitoring-only or also an admission gate for TRX inventory

One additional caveat: these are still soft thresholds, not hard caps, because the executor queue and pending/repush queues are effectively unbounded today. If the goal is heap protection, we should be explicit about that in the design.