Skip to content

Commit 378cdc5

Browse files
Dmitry LenevDmitry Lenev
Dmitry Lenev
authored and
Dmitry Lenev
committed
Patch that refactors global read lock implementation and fixes
bug #57006 "Deadlock between HANDLER and FLUSH TABLES WITH READ LOCK" and bug #54673 "It takes too long to get readlock for 'FLUSH TABLES WITH READ LOCK'". The first bug manifested itself as a deadlock which occurred when a connection, which had some table open through HANDLER statement, tried to update some data through DML statement while another connection tried to execute FLUSH TABLES WITH READ LOCK concurrently. What happened was that FTWRL in the second connection managed to perform first step of GRL acquisition and thus blocked all upcoming DML. After that it started to wait for table open through HANDLER statement to be flushed. When the first connection tried to execute DML it has started to wait for GRL/the second connection creating deadlock. The second bug manifested itself as starvation of FLUSH TABLES WITH READ LOCK statements in cases when there was a constant stream of concurrent DML statements (in two or more connections). This has happened because requests for protection against GRL which were acquired by DML statements were ignoring presence of pending GRL and thus the latter was starved. This patch solves both these problems by re-implementing GRL using metadata locks. Similar to the old implementation acquisition of GRL in new implementation is two-step. During the first step we block all concurrent DML and DDL statements by acquiring global S metadata lock (each DML and DDL statement acquires global IX lock for its duration). During the second step we block commits by acquiring global S lock in COMMIT namespace (commit code acquires global IX lock in this namespace). Note that unlike in old implementation acquisition of protection against GRL in DML and DDL is semi-automatic. We assume that any statement which should be blocked by GRL will either open and acquires write-lock on tables or acquires metadata locks on objects it is going to modify. For any such statement global IX metadata lock is automatically acquired for its duration. The first problem is solved because waits for GRL become visible to deadlock detector in metadata locking subsystem and thus deadlocks like one in the first bug become impossible. The second problem is solved because global S locks which are used for GRL implementation are given preference over IX locks which are acquired by concurrent DML (and we can switch to fair scheduling in future if needed). Important change: FTWRL/GRL no longer blocks DML and DDL on temporary tables. Before this patch behavior was not consistent in this respect: in some cases DML/DDL statements on temporary tables were blocked while in others they were not. Since the main use cases for FTWRL are various forms of backups and temporary tables are not preserved during backups we have opted for consistently allowing DML/DDL on temporary tables during FTWRL/GRL. Important change: This patch changes thread state names which are used when DML/DDL of FTWRL is waiting for global read lock. It is now either "Waiting for global read lock" or "Waiting for commit lock" depending on the stage on which FTWRL is. Incompatible change: To solve deadlock in events code which was exposed by this patch we have to replace LOCK_event_metadata mutex with metadata locks on events. As result we have to prohibit DDL on events under LOCK TABLES. This patch also adds extensive test coverage for interaction of DML/DDL and FTWRL. Performance of new and old global read lock implementations in sysbench tests were compared. There were no significant difference between new and old implementations.
1 parent aee8fce commit 378cdc5

75 files changed

Lines changed: 5493 additions & 1244 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎mysql-test/include/check_ftwrl_compatible.inc‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
#
2+
# SUMMARY
3+
# Check that a statement is compatible with FLUSH TABLES WITH READ LOCK.
4+
#
5+
# PARAMETERS
6+
# $con_aux1 Name of the 1st aux connection to be used by this script.
7+
# $con_aux2 Name of the 2nd aux connection to be used by this script.
8+
# $statement The statement to be checked.
9+
# $cleanup_stmt The statement to be run in order to revert effects of
10+
# the statement to be checked.
11+
# $skip_3rd_chk Skip the 3rd stage of checking. The purpose of the third
12+
# stage is to check that metadata locks taken by this
13+
# statement are compatible with metadata locks taken
14+
# by FTWRL.
15+
#
16+
# EXAMPLE
17+
# flush_read_lock.test
18+
#
19+
--disable_result_log
20+
--disable_query_log
21+
22+
# Reset DEBUG_SYNC facility for safety.
23+
set debug_sync= "RESET";
24+
25+
#
26+
# First, check that the statement can be run under FTWRL.
27+
#
28+
flush tables with read lock;
29+
--disable_abort_on_error
30+
--eval $statement
31+
--enable_abort_on_error
32+
let $err= $mysql_errno;
33+
if (!$err)
34+
{
35+
--echo Success: Was able to run '$statement' under FTWRL.
36+
unlock tables;
37+
if (`SELECT "$cleanup_stmt" <> ""`)
38+
{
39+
--eval $cleanup_stmt;
40+
}
41+
}
42+
if ($err)
43+
{
44+
--echo Error: Wasn't able to run '$statement' under FTWRL!
45+
unlock tables;
46+
}
47+
48+
#
49+
# Then check that this statement won't be blocked by FTWRL
50+
# that is active in another connection.
51+
#
52+
connection $con_aux1;
53+
flush tables with read lock;
54+
55+
connection default;
56+
--send_eval $statement;
57+
58+
connection $con_aux1;
59+
60+
--enable_result_log
61+
--enable_query_log
62+
let $wait_condition=
63+
select count(*) = 0 from information_schema.processlist
64+
where info = "$statement";
65+
--source include/wait_condition.inc
66+
--disable_result_log
67+
--disable_query_log
68+
69+
if ($success)
70+
{
71+
--echo Success: Was able to run '$statement' with FTWRL active in another connection.
72+
73+
connection default;
74+
# Apparently statement was successfully executed and so
75+
# was not blocked by FTWRL.
76+
# To be safe against wait_condition.inc succeeding due to
77+
# races let us first reap the statement being checked to
78+
# ensure that it has been successfully executed.
79+
--reap
80+
81+
connection $con_aux1;
82+
unlock tables;
83+
84+
connection default;
85+
}
86+
if (!$success)
87+
{
88+
--echo Error: Wasn't able to run '$statement' with FTWRL active in another connection!
89+
unlock tables;
90+
connection default;
91+
--reap
92+
}
93+
94+
if (`SELECT "$cleanup_stmt" <> ""`)
95+
{
96+
--eval $cleanup_stmt;
97+
}
98+
99+
if (`SELECT "$skip_3rd_check" = ""`)
100+
{
101+
#
102+
# Finally, let us check that FTWRL will succeed if this statement
103+
# is active but has already closed its tables.
104+
#
105+
connection default;
106+
set debug_sync='execute_command_after_close_tables SIGNAL parked WAIT_FOR go';
107+
--send_eval $statement;
108+
109+
connection $con_aux1;
110+
set debug_sync="now WAIT_FOR parked";
111+
--send flush tables with read lock
112+
113+
connection $con_aux2;
114+
--enable_result_log
115+
--enable_query_log
116+
let $wait_condition=
117+
select count(*) = 0 from information_schema.processlist
118+
where info = "flush tables with read lock";
119+
--source include/wait_condition.inc
120+
--disable_result_log
121+
--disable_query_log
122+
123+
if ($success)
124+
{
125+
--echo Success: Was able to run FTWRL while '$statement' was active in another connection.
126+
connection $con_aux1;
127+
# Apparently FTWRL was successfully executed and so was not blocked by
128+
# the statement being checked. To be safe against wait_condition.inc
129+
# succeeding due to races let us first reap the FTWRL to ensure that it
130+
# has been successfully executed.
131+
--reap
132+
unlock tables;
133+
set debug_sync="now SIGNAL go";
134+
connection default;
135+
--reap
136+
}
137+
if (!$success)
138+
{
139+
--echo Error: Wasn't able to run FTWRL while '$statement' was active in another connection!
140+
set debug_sync="now SIGNAL go";
141+
connection default;
142+
--reap
143+
connection $con_aux1;
144+
--reap
145+
unlock tables;
146+
connection default;
147+
}
148+
149+
set debug_sync= "RESET";
150+
if (`SELECT "$cleanup_stmt" <> ""`)
151+
{
152+
--eval $cleanup_stmt;
153+
}
154+
155+
}
156+
157+
--enable_result_log
158+
--enable_query_log

‎mysql-test/include/check_ftwrl_incompatible.inc‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
#
2+
# SUMMARY
3+
# Check that a statement is incompatible with FLUSH TABLES WITH READ LOCK.
4+
#
5+
# PARAMETERS
6+
# $con_aux1 Name of the 1st aux connection to be used by this script.
7+
# $con_aux2 Name of the 2nd aux connection to be used by this script.
8+
# $statement The statement to be checked.
9+
# $cleanup_stmt1 The 1st statement to be run in order to revert effects
10+
# of statement to be checked.
11+
# $cleanup_stmt2 The 2nd statement to be run in order to revert effects
12+
# of statement to be checked.
13+
# $skip_3rd_chk Skip the 3rd stage of checking. The purpose of the third
14+
# stage is to check that metadata locks taken by this
15+
# statement are incompatible with metadata locks taken
16+
# by FTWRL.
17+
#
18+
# EXAMPLE
19+
# flush_read_lock.test
20+
#
21+
--disable_result_log
22+
--disable_query_log
23+
24+
# Reset DEBUG_SYNC facility for safety.
25+
set debug_sync= "RESET";
26+
27+
#
28+
# First, check that the statement cannot be run under FTWRL.
29+
#
30+
flush tables with read lock;
31+
--disable_abort_on_error
32+
--eval $statement
33+
--enable_abort_on_error
34+
let $err= $mysql_errno;
35+
if ($err)
36+
{
37+
--echo Success: Was not able to run '$statement' under FTWRL.
38+
unlock tables;
39+
}
40+
if (!$err)
41+
{
42+
--echo Error: Was able to run '$statement' under FTWRL!
43+
unlock tables;
44+
if (`SELECT "$cleanup_stmt1" <> ""`)
45+
{
46+
--eval $cleanup_stmt1;
47+
}
48+
if (`SELECT "$cleanup_stmt2" <> ""`)
49+
{
50+
--eval $cleanup_stmt2;
51+
}
52+
}
53+
54+
55+
#
56+
# Then check that this statement is blocked by FTWRL
57+
# that is active in another connection.
58+
#
59+
connection $con_aux1;
60+
flush tables with read lock;
61+
62+
connection default;
63+
--send_eval $statement;
64+
65+
connection $con_aux1;
66+
67+
--enable_result_log
68+
--enable_query_log
69+
let $wait_condition=
70+
select count(*) = 1 from information_schema.processlist
71+
where (state = "Waiting for global read lock" or
72+
state = "Waiting for commit lock") and
73+
info = "$statement";
74+
--source include/wait_condition.inc
75+
--disable_result_log
76+
--disable_query_log
77+
78+
if ($success)
79+
{
80+
--echo Success: '$statement' is blocked by FTWRL active in another connection.
81+
}
82+
if (!$success)
83+
{
84+
--echo Error: '$statement' wasn't blocked by FTWRL active in another connection!
85+
}
86+
unlock tables;
87+
88+
connection default;
89+
--reap
90+
91+
if (`SELECT "$cleanup_stmt1" <> ""`)
92+
{
93+
--eval $cleanup_stmt1;
94+
}
95+
if (`SELECT "$cleanup_stmt2" <> ""`)
96+
{
97+
--eval $cleanup_stmt2;
98+
}
99+
100+
if (`SELECT "$skip_3rd_check" = ""`)
101+
{
102+
#
103+
# Finally, let us check that FTWRL will not succeed if this
104+
# statement is active but has already closed its tables.
105+
#
106+
connection default;
107+
--eval set debug_sync='execute_command_after_close_tables SIGNAL parked WAIT_FOR go';
108+
--send_eval $statement;
109+
110+
connection $con_aux1;
111+
set debug_sync="now WAIT_FOR parked";
112+
--send flush tables with read lock
113+
114+
connection $con_aux2;
115+
--enable_result_log
116+
--enable_query_log
117+
let $wait_condition=
118+
select count(*) = 1 from information_schema.processlist
119+
where (state = "Waiting for global read lock" or
120+
state = "Waiting for commit lock") and
121+
info = "flush tables with read lock";
122+
--source include/wait_condition.inc
123+
--disable_result_log
124+
--disable_query_log
125+
126+
if ($success)
127+
{
128+
--echo Success: FTWRL is blocked when '$statement' is active in another connection.
129+
}
130+
if (!$success)
131+
{
132+
--echo Error: FTWRL isn't blocked when '$statement' is active in another connection!
133+
}
134+
set debug_sync="now SIGNAL go";
135+
connection default;
136+
--reap
137+
connection $con_aux1;
138+
--reap
139+
unlock tables;
140+
connection default;
141+
142+
set debug_sync= "RESET";
143+
144+
if (`SELECT "$cleanup_stmt1" <> ""`)
145+
{
146+
--eval $cleanup_stmt1;
147+
}
148+
if (`SELECT "$cleanup_stmt2" <> ""`)
149+
{
150+
--eval $cleanup_stmt2;
151+
}
152+
}
153+
154+
--enable_result_log
155+
--enable_query_log

‎mysql-test/include/handler.inc‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1545,8 +1545,6 @@ lock table not_exists_write read;
15451545
--echo # We still have the read lock.
15461546
--error ER_CANT_UPDATE_WITH_READLOCK
15471547
drop table t1;
1548-
handler t1 read next;
1549-
handler t1 close;
15501548
handler t1 open;
15511549
select a from t2;
15521550
handler t1 read next;

‎mysql-test/include/wait_show_condition.inc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ if (`SELECT '$wait_for_all' = '1'`)
101101

102102
if (!$found)
103103
{
104-
echo # Timeout in include/wait_show_condition.inc for $wait_condition;
104+
echo # Timeout in include/wait_show_condition.inc for $condition;
105105
echo # show_statement : $show_statement;
106106
echo # field : $field;
107107
echo # condition : $condition;

‎mysql-test/r/delayed.result‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -418,6 +418,18 @@ COMMIT;
418418
UNLOCK TABLES;
419419
# Connection con1
420420
# Reaping: INSERT DELAYED INTO t1 VALUES (5)
421+
# Connection default
422+
# Test 5: LOCK TABLES + INSERT DELAYED in one connection.
423+
# This test has triggered some asserts in metadata locking
424+
# subsystem at some point in time..
425+
LOCK TABLE t1 WRITE;
426+
INSERT DELAYED INTO t2 VALUES (7);
427+
UNLOCK TABLES;
428+
SET AUTOCOMMIT= 0;
429+
LOCK TABLE t1 WRITE;
430+
INSERT DELAYED INTO t2 VALUES (8);
431+
UNLOCK TABLES;
432+
SET AUTOCOMMIT= 1;
421433
# Connection con2
422434
# Connection con1
423435
# Connection default

0 commit comments

Comments
 (0)