mysql左连接多表的地方很慢

阅读其他左连接慢文章，尝试了很多方法，但仍然没有答案。mysql左连接多表的地方很慢

t_pbl_list是ip子网黑名单。 800K排。

CREATE TABLE `t_pbl_list` (
    `_mask` tinyint(3) unsigned NOT NULL, #subnet 
    `_ipNet` int(10) unsigned NOT NULL, 
    `_attr` tinyint(3) unsigned NOT NULL, # bit0: exception 
    PRIMARY KEY (`_mask`,`_ipNet`), 
    KEY `ix_pbl_list` (`_mask`,`_ipNet`,`_attr`) 
) ENGINE=MEMORY DEFAULT CHARSET=latin1

t_pbl_result是IP结果列表3.7M行..

CREATE TABLE `t_pbl_result` (
    `_ip` int(10) unsigned NOT NULL, 
    `_utSlotAdd` mediumint(8) unsigned NOT NULL, 
    PRIMARY KEY (`_ip`) 
) ENGINE=MyISAM DEFAULT CHARSET=latin1

我想_ip从t_pbl_result这不是t_pbl_list选择。

我不知道为什么这个sql需要50秒的内存。 200米为Myisam运行。

1, SIMPLE, A, index, , PRIMARY, 4, , 3687929, Using index 
1, SIMPLE, B32, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B31, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B30, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B29, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B28, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B27, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B26, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B25, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B24, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B23, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B22, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B21, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B20, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B19, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B18, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B17, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B16, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B15, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B14, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B13, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B12, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B11, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B10, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B09, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, 
1, SIMPLE, B08, eq_ref, PRIMARY,ix_pbl_list, PRIMARY, 5, const,func, 1, Using where

SQL：在无条件

SELECT 
A._ip 
FROM t_pbl_result AS A 
LEFT JOIN t_pbl_list AS B32 ON B32._mask = 32 AND B32._ipNet = A._ip >> 0 
LEFT JOIN t_pbl_list AS B31 ON B31._mask = 31 AND B31._ipNet = A._ip >> 01 
LEFT JOIN t_pbl_list AS B30 ON B30._mask = 30 AND B30._ipNet = A._ip >> 02 
LEFT JOIN t_pbl_list AS B29 ON B29._mask = 29 AND B29._ipNet = A._ip >> 03 
LEFT JOIN t_pbl_list AS B28 ON B28._mask = 28 AND B28._ipNet = A._ip >> 04 
LEFT JOIN t_pbl_list AS B27 ON B27._mask = 27 AND B27._ipNet = A._ip >> 05 
LEFT JOIN t_pbl_list AS B26 ON B26._mask = 26 AND B26._ipNet = A._ip >> 06 
LEFT JOIN t_pbl_list AS B25 ON B25._mask = 25 AND B25._ipNet = A._ip >> 07 
LEFT JOIN t_pbl_list AS B24 ON B24._mask = 24 AND B24._ipNet = A._ip >> 08 
LEFT JOIN t_pbl_list AS B23 ON B23._mask = 23 AND B23._ipNet = A._ip >> 09 
LEFT JOIN t_pbl_list AS B22 ON B22._mask = 22 AND B22._ipNet = A._ip >> 10 
LEFT JOIN t_pbl_list AS B21 ON B21._mask = 21 AND B21._ipNet = A._ip >> 11 
LEFT JOIN t_pbl_list AS B20 ON B20._mask = 20 AND B20._ipNet = A._ip >> 12 
LEFT JOIN t_pbl_list AS B19 ON B19._mask = 19 AND B19._ipNet = A._ip >> 13 
LEFT JOIN t_pbl_list AS B18 ON B18._mask = 18 AND B18._ipNet = A._ip >> 14 
LEFT JOIN t_pbl_list AS B17 ON B17._mask = 17 AND B17._ipNet = A._ip >> 15 
LEFT JOIN t_pbl_list AS B16 ON B16._mask = 16 AND B16._ipNet = A._ip >> 16 
LEFT JOIN t_pbl_list AS B15 ON B15._mask = 15 AND B15._ipNet = A._ip >> 17 
LEFT JOIN t_pbl_list AS B14 ON B14._mask = 14 AND B14._ipNet = A._ip >> 18 
LEFT JOIN t_pbl_list AS B13 ON B13._mask = 13 AND B13._ipNet = A._ip >> 19 
LEFT JOIN t_pbl_list AS B12 ON B12._mask = 12 AND B12._ipNet = A._ip >> 20 
LEFT JOIN t_pbl_list AS B11 ON B11._mask = 11 AND B11._ipNet = A._ip >> 21 
LEFT JOIN t_pbl_list AS B10 ON B10._mask = 10 AND B10._ipNet = A._ip >> 22 
LEFT JOIN t_pbl_list AS B09 ON B09._mask = 09 AND B09._ipNet = A._ip >> 23 
LEFT JOIN t_pbl_list AS B08 ON B08._mask = 08 AND B08._ipNet = A._ip >> 24 
WHERE 
(  # not in any subnet. 
     B32._mask IS NULL AND B31._mask IS NULL AND B30._mask IS NULL AND B29._mask IS NULL AND B28._mask IS NULL AND B27._mask IS NULL 
    AND B26._mask IS NULL AND B25._mask IS NULL AND B24._mask IS NULL AND B23._mask IS NULL AND B22._mask IS NULL AND B21._mask IS NULL 
    AND B20._mask IS NULL AND B19._mask IS NULL AND B18._mask IS NULL AND B17._mask IS NULL AND B16._mask IS NULL AND B15._mask IS NULL 
    AND B14._mask IS NULL AND B13._mask IS NULL AND B12._mask IS NULL AND B11._mask IS NULL AND B10._mask IS NULL AND B09._mask IS NULL 
    AND B08._mask IS NULL 
) 
OR 
(  # any subnet has exception attr. 
     B32._attr & 1 <=> 1 OR B31._attr & 1 <=> 1 OR B30._attr & 1 <=> 1 OR B29._attr & 1 <=> 1 OR B28._attr & 1 <=> 1 OR B27._attr & 1 <=> 1 
    OR B26._attr & 1 <=> 1 OR B25._attr & 1 <=> 1 OR B24._attr & 1 <=> 1 OR B23._attr & 1 <=> 1 OR B22._attr & 1 <=> 1 OR B21._attr & 1 <=> 1 
    OR B20._attr & 1 <=> 1 OR B19._attr & 1 <=> 1 OR B18._attr & 1 <=> 1 OR B17._attr & 1 <=> 1 OR B16._attr & 1 <=> 1 OR B15._attr & 1 <=> 1 
    OR B14._attr & 1 <=> 1 OR B13._attr & 1 <=> 1 OR B12._attr & 1 <=> 1 OR B11._attr & 1 <=> 1 OR B10._attr & 1 <=> 1 OR B09._attr & 1 <=> 1 
    OR B08._attr & 1 <=> 1 
) 
;

尝试查询。只需放置_ip，将_attr合并到其他表中。也很慢。

试过在使用子查询的地方加入。也很慢。

试过的使用不存在于哪里而是加入。也很慢。

试过删除>>操作功能。也很慢。

我认为内连接可以快得多，但我需要在同一个表中连接所有表。

来源

2017-08-27 wa56

您对Memory和myisam的选择看起来非常慎重，但我很好奇你为什么选择这些。 – Strawberry

需要200s在myisaim上运行这两个表的SQL。在列表table.I内存需要50s我不知道太多的MySQL。只是觉得't_pbl_list'不大，所以可能是记忆。 't_pbl_result'很重要，很大，即使有什么不对，也不会损失，所以myisam。（innodb更好的交易，但我认为总是添加，删除，读取myisam可能会更快） – wa56

我认为你很聪明地避免了需要检查IP是否在范围内的范围扫描。不知道是否有一个从根本上更快的方法。但你可以只使用甚至是掩码（即使是在掩码中分割所有奇怪的掩码），它也会减少连接数和时间，但可能需要更多的工作来管理列表。你也可以考虑一个不同的过程：你可以将当前的允许/阻止状态存储在't_pbl_result'中，并使用触发器使该列保持最新状态。这当然取决于你如何使用这些列表（例如，如果你经常编辑范围）。 – Solarflare

我想从t_pbl_result中选择不在t_pbl_list中的_ip。

也许我错过了一些东西。你有没有一个原因，你不只参加一次？

SELECT  _ip 
FROM  (
       SELECT  res._ip 
          , lst._ipNet 
       FROM  t_pbl_result res 
       LEFT JOIN t_pbl_list lst ON 
          res._ip = lst._ipNet 
      ) t1 
WHERE  _ipNet IS NULL

来源

2017-08-27 04:29:28 HodgePodge

谢谢@HodgePodge，但是rst._ip就像192.168.0.5和lst._ipNet，像192.168.0.0和lst._mask = 24。所以我认为我需要尝试每个掩码。 – wa56

@郑王这确实增加了复杂性。此查询是否可以正常运行：SELECT _ip FROM（SELECT A._ip，B32._ipNet FROM t_pbl_result AS A LEFT JOIN t_pbl_list AS B32 ON B32._mask = 32 AND B32._ipNet = A._ip >> 0）t1 WHERE _ipNet不是NULL';如果没关系，只需单独运行每个子语句，然后在最后加入_ip列表即可。 – HodgePodge

thx，这是有道理的。但是它只需要4秒就可以完成B32，工会24次....还有一个例外情况是_attr＆1 = 1. – wa56

50秒？那么，做3.7M * 24 JOINs需要时间。

t_pbl_list中的ip-ranges是否重叠？如果不是，那么可以在单个测试中执行每个测试的优化，而不是24个测试。仍然有3.7M测试。请参阅http://mysql.rjweb.org/doc.php/ipranges

来源

2017-08-27 18:11:10

mysql左连接多表的地方很慢

回答

相关问题