如何解决MySQL - 最近加入时的性能问题
我有两个表，markets（27 条记录）和 histories（约 175 万条记录，每个市场约 67K）。
我需要获取每个市场的最新 histories 记录。
我尝试的解决方案有效，但速度非常慢。
表 DDL
CREATE TABLE `markets` (
`id` int unsigned NOT NULL AUTO_INCREMENT,`base_asset_id` int unsigned NOT NULL,`quote_asset_id` int unsigned NOT NULL,`deleted_at` timestamp NULL DEFAULT NULL,PRIMARY KEY (`id`),UNIQUE KEY `markets_base_asset_id_quote_asset_id_unique` (`base_asset_id`,`quote_asset_id`),KEY `markets_base_asset_id_index` (`base_asset_id`),KEY `markets_quote_asset_id_index` (`quote_asset_id`),CONSTRAINT `markets_base_asset_id_foreign` FOREIGN KEY (`base_asset_id`) REFERENCES `assets` (`id`),CONSTRAINT `markets_quote_asset_id_foreign` FOREIGN KEY (`quote_asset_id`) REFERENCES `assets` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=28 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
CREATE TABLE `histories` (
`id` bigint unsigned NOT NULL AUTO_INCREMENT,`market_id` int unsigned NOT NULL,`timeframe` enum('1_m','5_m','15_m','30_m','1_H','4_H','6_H','12_H','1_D','1_W','1_M') CHARACTER SET ascii COLLATE ascii_bin NOT NULL,`time` int unsigned NOT NULL,`is_source` tinyint(1) NOT NULL DEFAULT '0',`is_final` tinyint(1) NOT NULL DEFAULT '0',`open` decimal(36,18) NOT NULL,`high` decimal(36,`low` decimal(36,`close` decimal(36,18) DEFAULT NULL,`volume` decimal(36,`ohlc_avg` decimal(36,`hl_avg` decimal(36,UNIQUE KEY `histories_market_id_timeframe_time_unique` (`market_id`,`timeframe`,`time`),KEY `histories_market_id_index` (`market_id`),KEY `histories_timeframe_index` (`timeframe`),KEY `histories_time_index` (`time`) USING BTREE,CONSTRAINT `histories_market_id_foreign` FOREIGN KEY (`market_id`) REFERENCES `markets` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2334503 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
我尝试了什么
1 - 不相关的子查询
我开始使用这个解决方案，因为我在其他时候使用过它，需要 ~7.5 秒：
SELECT
m.*,h.time,h.close
FROM
markets m
LEFT JOIN (
SELECT
market_id,MAX(`time`) AS `time`
FROM
histories h
WHERE
h.is_final = 1
GROUP BY
market_id
) latest_history
ON latest_history.market_id = m.id
LEFT JOIN histories h
ON h.market_id = m.id
and h.`time` = latest_history.time;
EXPLAIN 结果：
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| 1 | PRIMARY | m | NULL | ALL | NULL | NULL | NULL | NULL | 27 | 100.00 | NULL |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 4 | db_name.m.id | 1745 | 100.00 | Using index |
| 1 | PRIMARY | h | NULL | ref | histories_market_id_timeframe_time_unique,histories_market_id_index,histories_time_index | histories_time_index | 4 | latest_history.time | 26 | 100.00 | Using where |
| 2 | DERIVED | h | NULL | index | histories_market_id_timeframe_time_unique,histories_market_id_index | histories_market_id_index | 4 | NULL | 1744647 | 10.00 | Using where |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
2 - 使用 WITH
我尝试使用 WITH 运行子查询，但没有任何改进，仍然 ~7.5s：
WITH latest_history AS (
SELECT
market_id,MAX(h.`time`) AS `time`
FROM
histories h
WHERE
h.is_final = 1
GROUP BY
market_id
)
SELECT
m.*,h.close
FROM
markets m
LEFT JOIN latest_history
ON latest_history.market_id = m.id
LEFT JOIN histories h
ON h.market_id = m.id
AND h.`time` = latest_history.time;
EXPLAIN 结果（与上一个相同）
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
| 1 | PRIMARY | m | NULL | ALL | NULL | NULL | NULL | NULL | 27 | 100.00 | NULL |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 4 | db_name.m.id | 1745 | 100.00 | Using index |
| 1 | PRIMARY | h | NULL | ref | histories_market_id_timeframe_time_unique,histories_market_id_index | histories_market_id_index | 4 | NULL | 1744647 | 10.00 | Using where |
+----+-------------+------------+------------+-------+------------------------------------------------------------------------------------------+---------------------------+---------+---------------------+---------+----------+-------------+
3 - 使用 WITH 和窗口函数
然后我从 5.7 升级到 8.0.22 以尝试另一种建议的方法，这需要更多时间：~11s
WITH latest_history AS (
SELECT
h.*,ROW_NUMBER() OVER (PARTITION BY market_id ORDER BY id DESC) AS rn
FROM
histories AS h
where
h.is_final = 1
)
SELECT
m.*,latest_history.time,latest_history.close
FROM
markets m
LEFT JOIN latest_history
ON latest_history.market_id = m.id
AND latest_history.rn = 1;
EXPLAIN 结果：
+----+-------------+------------+------------+------+---------------+-------------+---------+--------------------+---------+----------+-----------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------+------------+------+---------------+-------------+---------+--------------------+---------+----------+-----------------------------+
| 1 | PRIMARY | m | NULL | ALL | NULL | NULL | NULL | NULL | 27 | 100.00 | NULL |
| 1 | PRIMARY | <derived2> | NULL | ref | <auto_key0> | <auto_key0> | 12 | db_name.m.id,const | 1744 | 100.00 | NULL |
| 2 | DERIVED | h | NULL | ALL | NULL | NULL | NULL | NULL | 1744647 | 10.00 | Using where; Using filesort |
+----+-------------+------------+------------+------+---------------+-------------+---------+--------------------+---------+----------+-----------------------------+
其他信息
然后我看到解决方案1和2中需要的单独子查询（MAX和GROUP BY）需要~ 7.5s！
所以我认为 histories 结构/索引应该有一些根本性的错误，而不是我加入 markets 的方式。
明确地说，这就是我所指的：
SELECT
market_id,MAX(h.`time`) AS `time`
FROM
histories h
WHERE
h.is_final = 1
GROUP BY
market_id;
EXPLAIN 结果：
+----+-------------+-------+------------+-------+---------------------------------------------------------------------+---------------------------+---------+------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------------------------------------------------------------+---------------------------+---------+------+---------+----------+-------------+
| 1 | SIMPLE | h | NULL | index | histories_market_id_timeframe_time_unique,histories_market_id_index | histories_market_id_index | 4 | NULL | 1744647 | 10.00 | Using where |
+----+-------------+-------+------------+-------+---------------------------------------------------------------------+---------------------------+---------+------+---------+----------+-------------+
time 是一个表示 Unix 时间戳的 int，可以使用 id 但它不会提高性能。
当然，我尝试查看有关此组性能问题的其他问题/帖子，但我仍然无法解决它。
解决方法
一种替代方法是使用子查询进行过滤：
select m.*,h.time,h.close
from markets m
left join histories h
on h.market_id = m.id
and h.time = (
select max(h1.time) from histories h1 where h1.market_id = m.id and h1.is_final = 1
)
为了提高性能，您需要在 histories(market_id,is_final,time desc) 上建立索引。
由于您只需要 histories 表中的两列，您可能还需要考虑使用两个子查询：
select m.*,(select h.time from history h where h.market_id = m.id and h.is_final = 1 order by h.time desc limit 1) as time,(select h.close from history h where h.market_id = m.id and h.is_final = 1 order by h.time desc limit 1) as close
from markets m
相同的索引将有助于查询 - 我们甚至可以在索引末尾添加 close，因此：histories(market_id,time desc,close)。
最后：不是最新版本的 MySQL，您可以尝试横向连接：
select m.*,h.*
from markets m
left join lateral (
select h.time,h.close
from history h
where h.market_id = m.id and h.is_final = 1
order by h.time desc limit 1
) h on true
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。