MPP response memory is not properly accounted to SQL tracker during MPPDataPacket.Unmarshal

[yi888long]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Create a table with many varchar columns and one wide mediumtext column:

CREATE DATABASE IF NOT EXISTS testdb;
USE testdb;

CREATE TABLE `a_table` (
  `a1` bigint NOT NULL,
  `a2` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
  `a3` varchar(30) COLLATE utf8_general_ci DEFAULT NULL,
  `a4` varchar(20) COLLATE utf8_general_ci DEFAULT NULL,
  `a5` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
  `a6` varchar(100) COLLATE utf8_general_ci DEFAULT NULL,
  `a7` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
  `a8` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
  `a9` varchar(30) COLLATE utf8_general_ci NOT NULL,
  `a10` int NOT NULL,
  `a11` datetime DEFAULT NULL,
  `a12` date DEFAULT NULL,
  `a13` int DEFAULT NULL,
  `a14` varchar(60) COLLATE utf8_general_ci DEFAULT NULL,
  `a15` varchar(60) COLLATE utf8_general_ci DEFAULT NULL,
  `a16` mediumtext COLLATE utf8_general_ci DEFAULT NULL COMMENT 'wide payload',
  `a17` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
  `a18` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
  `a19` varchar(20) COLLATE utf8_general_ci DEFAULT NULL,
  `a20` varchar(5) COLLATE utf8_general_ci DEFAULT NULL,
  `a21` varchar(50) COLLATE utf8_general_ci DEFAULT NULL,
  `a22` varchar(128) COLLATE utf8_general_ci DEFAULT NULL,
  `a23` varchar(255) COLLATE utf8_general_ci DEFAULT NULL,
  `a24` int NOT NULL DEFAULT '0',
  PRIMARY KEY (`a1`) /*T![clustered_index] NONCLUSTERED */,
  UNIQUE KEY `uk_a9` (`a9`,`a10`,`a15`,`a4`,`a23`),
  KEY `idx_a14` (`a14`),
  KEY `idx_a2` (`a2`),
  KEY `idx_a3` (`a3`),
  KEY `idx_a4` (`a4`),
  KEY `idx_a11` (`a11`),
  KEY `idx_a12` (`a12`),
  KEY `idx_a19_a20` (`a19`,`a20`),
  KEY `idx_a22` (`a22`),
  KEY `idx_a9` (`a9`),
  KEY `idx_a23` (`a23`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;


Load data where:
many rows match a9 LIKE '4%'
column a16 is relatively large, for example about 2 KiB per row
the total matched a16 payload is hundreds of MiB or larger
In my local reproduction, I used 131072 rows and about 256 MiB of total a16 payload.
Add a TiFlash replica and wait until it becomes available:


ALTER TABLE testdb.a_table SET TIFLASH REPLICA 1;

SELECT *
FROM information_schema.tiflash_replica
WHERE TABLE_SCHEMA = 'testdb'
  AND TABLE_NAME = 'a_table';


Run the following SQL concurrently, for example in 4 concurrent sessions:

WITH cte1 AS (
  SELECT a15, a3, a4, a22, a12,
         a13, a11, a9, a23, a5,
         a6, a17, a7, a14, a10,
         a8, a2, a21, a19, a16
  FROM (
    SELECT a15, a3, a4, a22, a12,
           a13, a11, a9,
           IFNULL(a23, '') AS a23, a5,
           a6, a7, a17, a14, a10,
           a8, a2, a21, a19, a16,
           ROW_NUMBER() OVER (
             PARTITION BY a9, a4, a15, a23
             ORDER BY a11 DESC
           ) AS rn
    FROM testdb.a_table
    WHERE a9 LIKE CONCAT('4', '%')
  ) sub
),
cte2 AS (
  SELECT a9, a23, a4, MIN(a13) AS a13
  FROM cte1
  GROUP BY a4, a9, a23
  HAVING MIN(a13) = NULL
)
SELECT COUNT(*) AS total
FROM cte1;

The execution plan uses TiFlash MPP and has the following shape:

HashAgg
└─CTEFullScan
  CTE_0
  └─TableReader
    └─ExchangeSender
      └─Projection
        └─Window(row_number)
          └─Sort
            └─ExchangeReceiver
              └─ExchangeSender(HashPartition)
                └─Projection
                  └─TableFullScan pushed down to TiFlash

While the SQL is running, check TiDB process memory, SQL memory tracker, and heap profile.
For example:

Collect the TiDB heap profile from the status port:


### 2. What did you expect to see? (Required)

Collect the TiDB heap profile from the status port:

### 3. What did you see instead (Required)

TiDB process memory can grow very high during concurrent MPP execution and can reach tidb_server_memory_limit.
However, TiDB SQL memory tracking does not show a SQL consuming comparable memory. In my reproduction:
TiDB RSS peak was about 3.31 GiB
the largest single SQL memory shown by processlist was only about 617.8 MiB
after the queries finished, SQL tracker memory returned to 0
heap profile showed most live heap under MPP packet unmarshalling:
668.63MB  github.com/pingcap/kvproto/pkg/mpp.(*MPPDataPacket).Unmarshal
697.40MB  cumulative in gRPC / TiKV MPP receive path
This suggests that MPP response memory is allocated before it is charged to the SQL tracker.
The suspected code path is:

google.golang.org/grpc.recv()
  -> codec.Unmarshal()
    -> github.com/pingcap/kvproto/pkg/mpp.(*MPPDataPacket).Unmarshal()
      -> MPPDataPacket.Data allocation/copy

TiDB only charges memory later in localMppCoordinator.sendToRespCh. Also, sendToRespCh releases the charge too early by using defer Consume(-respSize) after sending the response to respChan.
Downstream, selectResult.fetchResp calls tipb.SelectResponse.Unmarshal(resultSubset.GetData()), which can allocate more memory before charging the decoded SelectResponse.
Relevant code in v8.5.2:
pkg/executor/internal/mpp/local_mpp_coordinator.go
  sendToRespCh(): charges resp.MemSize(), then defers uncharge
  receiveResults(): calls stream.Recv()
  handleMPPStreamResponse(): wraps MPPDataPacket as mppResponse

pkg/distsql/select_result.go
  fetchResp(): calls resultSubset.GetData(), then SelectResponse.Unmarshal(), then charges memory

github.com/pingcap/kvproto/pkg/mpp/mpp.pb.go
  MPPDataPacket.Unmarshal(): m.Data = append(m.Data[:0], ...)
This makes the global memory controller observe high Go heap usage, while MemUsageTop1Tracker may not identify the real SQL memory consumer.

### 4. What is your TiDB version? (Required)

<!-- Paste the output of SELECT tidb_version() -->

Release Version: v8.5.2
Git Commit Hash: f43a13324440f92209e2a9f04c0bbe9cf763978d

[kennedy8312]

/sig execution
/severity major

[kennedy8312]

tidb-triage-bot AI rationale

• severity/major: Concurrent TiFlash MPP queries can drive TiDB process memory to the server limit because response memory is not accounted to the SQL tracker.
• sig/execution: The issue is in MPP execution response handling and memory accounting paths such as local_mpp_coordinator and select_result.