STATUS OF ROLLBACKED TRANSACTION" and bug #17054007 - "TRANSACTION
IS NOT FULLY ROLLED BACK IN CASE OF INNODB DEADLOCK".
The problem in the first bug report was that although deadlock involving
metadata locks was reported using the same error code and message as InnoDB
deadlock it didn't rollback transaction like the latter. This caused
confusion to users as in some cases after ER_LOCK_DEADLOCK transaction
could have been restarted immediately and in some cases rollback was
required.
The problem in the second bug report was that although InnoDB deadlock
caused transaction rollback in all storage engines it didn't cause release
of metadata locks. So concurrent DDL on the tables used in transaction was
blocked until implicit or explicit COMMIT or ROLLBACK was issued in the
connection which got InnoDB deadlock.
The former issue has stemmed from the fact that when support for detection
and reporting metadata locks deadlocks was added we erroneously assumed
that InnoDB doesn't rollback transaction on deadlock but only last statement
(while this is what happens on InnoDB lock timeout actually) and so didn't
implement rollback of transactions on MDL deadlocks.
The latter issue was caused by the fact that rollback of transaction due
to deadlock is carried out by setting THD::transaction_rollback_request
flag at the point where deadlock is detected and performing rollback
inside of trans_rollback_stmt() call when this flag is set. And
trans_rollback_stmt() is not aware of MDL locks, so no MDL locks are
released.
This patch solves these two problems in the following way:
- In case when MDL deadlock is detect transaction rollback is requested
by setting THD::transaction_rollback_request flag.
- Code performing rollback of transaction if THD::transaction_rollback_request
is moved out from trans_rollback_stmt(). Now we handle rollback request
on the same level as we call trans_rollback_stmt() and release statement/
transaction MDL locks.
includes:
* remove some remnants of "Bug#14521864: MYSQL 5.1 TO 5.5 BUGS PARTITIONING"
* introduce LOCK_share, now LOCK_ha_data is strictly for engines
* rea_create_table() always creates .par file (even in "frm-only" mode)
* fix a 5.6 bug, temp file leak on dummy ALTER TABLE
Merged all ddl_logging code.
Merged sql_partition.cc
innodb_mysql_lock2.test and partition_cache.test now works.
Changed interface to strconvert() to make it easier to use with not \0 terminated strings.
sql/sql_partition.cc:
Full merge with 5.6
sql/sql_table.cc:
Merged all ddl_logging code
sql/strfunc.cc:
Added from_length argument to strconvert() to make it possible to use without end terminated strings.
sql/strfunc.h:
Added from_length argument to strconvert() to make it possible to use without end terminated strings.
with query cache, charset ucs2 and collation ucs2_unicode_ci
@ mysql-test/r/ctype_ucs2_query_cache.result
@ mysql-test/t/ctype_ucs2_query_cache-master.opt
@ mysql-test/t/ctype_ucs2_query_cache.test
Adding tests
@ sql/sql_cache.cc
Fixing not to use default_character_set->state_map,
which can point to a non-ASCII character set (utc2, utf16, utf32)
and thus have state_map undefined.
mysys/errors.c:
revert upstream's fix. use a much simpler one
mysys/my_write.c:
revert upstream's fix. use a simpler one
sql/item_xmlfunc.cc:
useless, but ok
sql/mysqld.cc:
simplify upstream's fix
storage/heap/hp_delete.c:
remove upstream's fix.
we'll use a much less expensive approach.
Base code and idea from a patch from by plinux at Taobao.
The idea is that we mark all memory that are thread specific with MY_THREAD_SPECIFIC.
Memory counting is done per thread in the my_malloc_size_cb_func callback function from my_malloc().
There are plenty of new asserts to ensure that for a debug server the counting is correct.
Information_schema.processlist gets two new columns: MEMORY_USED and EXAMINED_ROWS.
- The later is there mainly to show how query is progressing.
The following changes in interfaces was needed to get this to work:
- init_alloc_root() amd init_sql_alloc() has extra option so that one can mark memory with MY_THREAD_SPECIFIC
- One now have to use alloc_root_set_min_malloc() to set min memory to be allocated by alloc_root()
- my_init_dynamic_array() has extra option so that one can mark memory with MY_THREAD_SPECIFIC
- my_net_init() has extra option so that one can mark memory with MY_THREAD_SPECIFIC
- Added flag for hash_init() so that one can mark hash table to be thread specific.
- Added flags to init_tree() so that one can mark tree to be thread specific.
- Removed with_delete option to init_tree(). Now one should instead use MY_TREE_WITH_DELETE_FLAG.
- Added flag to Warning_info::Warning_info() if the structure should be fully initialized.
- String elements can now be marked as thread specific.
- Internal HEAP tables are now marking it's memory as MY_THREAD_SPECIFIC.
- Changed type of myf from int to ulong, as this is always a set of bit flags.
Other things:
- Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
- We now also show EXAMINED_ROWS in SHOW PROCESSLIST
- Added new variable 'memory_used'
- Fixed bug where kill_threads_for_user() was using the wrong mem_root to allocate memory.
- Removed calls to the obsoleted function init_dynamic_array()
- Use set_current_thd() instead of my_pthread_setspecific_ptr(THR_THD,...)
client/completion_hash.cc:
Updated call to init_alloc_root()
client/mysql.cc:
Updated call to init_alloc_root()
client/mysqlbinlog.cc:
init_dynamic_array() -> my_init_dynamic_array()
Updated call to init_alloc_root()
client/mysqlcheck.c:
Updated call to my_init_dynamic_array()
client/mysqldump.c:
Updated call to init_alloc_root()
client/mysqltest.cc:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
Fixed compiler warnings
extra/comp_err.c:
Updated call to my_init_dynamic_array()
extra/resolve_stack_dump.c:
Updated call to my_init_dynamic_array()
include/hash.h:
Added HASH_THREAD_SPECIFIC
include/heap.h:
Added flag is internal temporary table.
include/my_dir.h:
Safety fix: Ensure that MY_DONT_SORT and MY_WANT_STAT don't interfer with other mysys flags
include/my_global.h:
Changed type of myf from int to ulong, as this is always a set of bit flags.
include/my_sys.h:
Added MY_THREAD_SPECIFIC and MY_THREAD_MOVE
Added malloc_flags to DYNAMIC_ARRAY
Added extra mysys flag argument to my_init_dynamic_array()
Removed deprecated functions init_dynamic_array() and my_init_dynamic_array.._ci
Updated paramaters for init_alloc_root()
include/my_tree.h:
Added my_flags to allow one to use MY_THREAD_SPECIFIC with hash tables.
Removed with_delete. One should now instead use MY_TREE_WITH_DELETE_FLAG
Updated parameters to init_tree()
include/myisamchk.h:
Added malloc_flags to allow one to use MY_THREAD_SPECIFIC for checks.
include/mysql.h:
Added MYSQL_THREAD_SPECIFIC_MALLOC
Used 'unused1' to mark memory as thread specific.
include/mysql.h.pp:
Updated file
include/mysql_com.h:
Used 'unused1' to mark memory as thread specific.
Updated parameters for my_net_init()
libmysql/libmysql.c:
Updated call to init_alloc_root() to mark memory thread specific.
libmysqld/emb_qcache.cc:
Updated call to init_alloc_root()
libmysqld/lib_sql.cc:
Updated call to init_alloc_root()
mysql-test/r/create.result:
Updated results
mysql-test/r/user_var.result:
Updated results
mysql-test/suite/funcs_1/datadict/processlist_priv.inc:
Update to handle new format of SHOW PROCESSLIST
mysql-test/suite/funcs_1/datadict/processlist_val.inc:
Update to handle new format of SHOW PROCESSLIST
mysql-test/suite/funcs_1/r/is_columns_is.result:
Update to handle new format of SHOW PROCESSLIST
mysql-test/suite/funcs_1/r/processlist_priv_no_prot.result:
Updated results
mysql-test/suite/funcs_1/r/processlist_val_no_prot.result:
Updated results
mysql-test/t/show_explain.test:
Fixed usage of debug variable so that one can run test with --debug
mysql-test/t/user_var.test:
Added test of memory_usage variable.
mysys/array.c:
Added extra my_flags option to init_dynamic_array() and init_dynamic_array2() so that one can mark memory with MY_THREAD_SPECIFIC
All allocated memory is marked with the given my_flags.
Removed obsolete function init_dynamic_array()
mysys/default.c:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
mysys/hash.c:
Updated call to my_init_dynamic_array_ci().
Allocated memory is marked with MY_THREAD_SPECIFIC if HASH_THREAD_SPECIFIC is used.
mysys/ma_dyncol.c:
init_dynamic_array() -> my_init_dynamic_array()
Added #if to get rid of compiler warnings
mysys/mf_tempdir.c:
Updated call to my_init_dynamic_array()
mysys/my_alloc.c:
Added extra parameter to init_alloc_root() so that one can mark memory with MY_THREAD_SPECIFIC
Extend MEM_ROOT with a flag if memory is thread specific.
This is stored in block_size, to keep the size of the MEM_ROOT object identical as before.
Allocated memory is marked with MY_THREAD_SPECIFIC if used with init_alloc_root()
mysys/my_chmod.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_chsize.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_copy.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_create.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_delete.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_error.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_fopen.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_fstream.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_getwd.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_lib.c:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
Updated DBUG_PRINT because of change of myf type
mysys/my_lock.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_malloc.c:
Store at start of each allocated memory block the size of the block and if the block is thread specific.
Call malloc_size_cb_func, if set, with the memory allocated/freed.
Updated DBUG_PRINT because of change of myf type
mysys/my_open.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_pread.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_read.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_redel.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_rename.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_seek.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_sync.c:
Updated DBUG_PRINT because of change of myf type
mysys/my_thr_init.c:
Ensure that one can call my_thread_dbug_id() even if thread is not properly initialized.
mysys/my_write.c:
Updated DBUG_PRINT because of change of myf type
mysys/mysys_priv.h:
Updated parameters to sf_malloc and sf_realloc()
mysys/safemalloc.c:
Added checking that for memory marked with MY_THREAD_SPECIFIC that it's the same thread that is allocation and freeing the memory.
Added sf_malloc_dbug_id() to allow MariaDB to specify which THD is handling the memory.
Added my_flags arguments to sf_malloc() and sf_realloc() to be able to mark memory with MY_THREAD_SPECIFIC.
Added sf_report_leaked_memory() to get list of memory not freed by a thread.
mysys/tree.c:
Added flags to init_tree() so that one can mark tree to be thread specific.
Removed with_delete option to init_tree(). Now one should instead use MY_TREE_WITH_DELETE_FLAG.
Updated call to init_alloc_root()
All allocated memory is marked with the given malloc flags
mysys/waiting_threads.c:
Updated call to my_init_dynamic_array()
sql-common/client.c:
Updated call to init_alloc_root() and my_net_init() to mark memory thread specific.
Updated call to my_init_dynamic_array().
Added MYSQL_THREAD_SPECIFIC_MALLOC so that client can mark memory as MY_THREAD_SPECIFIC.
sql-common/client_plugin.c:
Updated call to init_alloc_root()
sql/debug_sync.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/event_scheduler.cc:
Removed calls to net_end() as this is now done in ~THD()
Call set_current_thd() to ensure that memory is assigned to right thread.
sql/events.cc:
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/filesort.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/filesort_utils.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/ha_ndbcluster.cc:
Updated call to init_alloc_root()
Updated call to my_net_init()
Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
sql/ha_ndbcluster_binlog.cc:
Updated call to my_net_init()
Updated call to init_sql_alloc()
Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
sql/ha_partition.cc:
Updated call to init_alloc_root()
sql/handler.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
Added missing call to my_dir_end()
sql/item_func.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/item_subselect.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/item_sum.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/log.cc:
More DBUG
Updated call to init_alloc_root()
sql/mdl.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/mysqld.cc:
Added total_memory_used
Updated call to init_alloc_root()
Move mysql_cond_broadcast() before my_thread_end()
Added mariadb_dbug_id() to count memory per THD instead of per thread.
Added my_malloc_size_cb_func() callback function for my_malloc() to count memory.
Move initialization of mysqld_server_started and mysqld_server_initialized earlier.
Updated call to my_init_dynamic_array().
Updated call to my_net_init().
Call my_pthread_setspecific_ptr(THR_THD,...) to ensure that memory is assigned to right thread.
Added status variable 'memory_used'.
Updated call to init_alloc_root()
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/mysqld.h:
Added set_current_thd()
sql/net_serv.cc:
Added new parameter to my_net_init() so that one can mark memory with MY_THREAD_SPECIFIC.
Store in net->thread_specific_malloc if memory is thread specific.
Mark memory to be thread specific if requested.
sql/opt_range.cc:
Updated call to my_init_dynamic_array()
Updated call to init_sql_alloc()
Added MY_THREAD_SPECIFIC to allocated memory.
sql/opt_subselect.cc:
Updated call to init_sql_alloc() to mark memory thread specific.
sql/protocol.cc:
Fixed compiler warning
sql/records.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/rpl_filter.cc:
Updated call to my_init_dynamic_array()
sql/rpl_handler.cc:
Updated call to my_init_dynamic_array2()
sql/rpl_handler.h:
Updated call to init_sql_alloc()
sql/rpl_mi.cc:
Updated call to my_init_dynamic_array()
sql/rpl_tblmap.cc:
Updated call to init_alloc_root()
sql/rpl_utility.cc:
Updated call to my_init_dynamic_array()
sql/slave.cc:
Initialize things properly before calling functions that allocate memory.
Removed calls to net_end() as this is now done in ~THD()
sql/sp_head.cc:
Updated call to init_sql_alloc()
Updated call to my_init_dynamic_array()
Added parameter to warning_info() that it should be fully initialized.
sql/sp_pcontext.cc:
Updated call to my_init_dynamic_array()
sql/sql_acl.cc:
Updated call to init_sql_alloc()
Updated call to my_init_dynamic_array()
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_admin.cc:
Added parameter to warning_info() that it should be fully initialized.
sql/sql_analyse.h:
Updated call to init_tree() to mark memory thread specific.
sql/sql_array.h:
Updated call to my_init_dynamic_array() to mark memory thread specific.
sql/sql_audit.cc:
Updated call to my_init_dynamic_array()
sql/sql_base.cc:
Updated call to init_sql_alloc()
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_cache.cc:
Updated comment
sql/sql_class.cc:
Added parameter to warning_info() that not initialize it until THD is fully created.
Updated call to init_sql_alloc()
Mark THD::user_vars has to be thread specific.
Updated call to my_init_dynamic_array()
Ensure that memory allocated by THD is assigned to the THD.
More DBUG
Always acll net_end() in ~THD()
Assert that all memory signed to this THD is really deleted at ~THD.
Fixed set_status_var_init() to not reset memory_used.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_class.h:
Added MY_THREAD_SPECIFIC to allocated memory.
Added malloc_size to THD to record allocated memory per THD.
sql/sql_delete.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/sql_error.cc:
Added 'initialize' parameter to Warning_info() to say if should allocate memory for it's structures.
This is used by THD::THD() to not allocate memory until THD is ready.
Added Warning_info::free_memory()
sql/sql_error.h:
Updated Warning_info() class.
sql/sql_handler.cc:
Updated call to init_alloc_root() to mark memory thread specific.
sql/sql_insert.cc:
More DBUG
sql/sql_join_cache.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/sql_lex.cc:
Updated call to my_init_dynamic_array()
sql/sql_lex.h:
Updated call to my_init_dynamic_array()
sql/sql_load.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/sql_parse.cc:
Removed calls to net_end() and thd->cleanup() as these are now done in ~THD()
Ensure that examined_row_count() is reset before query.
Fixed bug where kill_threads_for_user() was using the wrong mem_root to allocate memory.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
Don't restore thd->status_var.memory_used when restoring thd->status_var
sql/sql_plugin.cc:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
Don't allocate THD on the stack, as this causes problems with valgrind when doing thd memory counting.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_prepare.cc:
Added parameter to warning_info() that it should be fully initialized.
Updated call to init_sql_alloc() to mark memory thread specific.
sql/sql_reload.cc:
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_select.cc:
Updated call to my_init_dynamic_array() and init_sql_alloc() to mark memory thread specific.
Added MY_THREAD_SPECIFIC to allocated memory.
More DBUG
sql/sql_servers.cc:
Updated call to init_sql_alloc() to mark memory some memory thread specific.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_show.cc:
Updated call to my_init_dynamic_array()
Mark my_dir() memory thread specific.
Use my_pthread_setspecific_ptr(THR_THD,...) to mark that allocated memory should be allocated to calling thread.
More DBUG.
Added malloc_size and examined_row_count to SHOW PROCESSLIST.
Added MY_THREAD_SPECIFIC to allocated memory.
Updated call to init_sql_alloc()
Added parameter to warning_info() that it should be fully initialized.
sql/sql_statistics.cc:
Fixed compiler warning
sql/sql_string.cc:
String elements can now be marked as thread specific.
sql/sql_string.h:
String elements can now be marked as thread specific.
sql/sql_table.cc:
Updated call to init_sql_alloc() and my_malloc() to mark memory thread specific
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
Fixed compiler warning
sql/sql_test.cc:
Updated call to my_init_dynamic_array() to mark memory thread specific.
sql/sql_trigger.cc:
Updated call to init_sql_alloc()
sql/sql_udf.cc:
Updated call to init_sql_alloc()
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/sql_update.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
sql/table.cc:
Updated call to init_sql_alloc().
Mark memory used by temporary tables, that are not for slave threads, as MY_THREAD_SPECIFIC
Updated call to init_sql_alloc()
sql/thr_malloc.cc:
Added my_flags argument to init_sql_alloc() to be able to mark memory as MY_THREAD_SPECIFIC.
sql/thr_malloc.h:
Updated prototype for init_sql_alloc()
sql/tztime.cc:
Updated call to init_sql_alloc()
Updated call to init_alloc_root() to mark memory thread specific.
my_pthread_setspecific_ptr(THR_THD,...) -> set_current_thd()
sql/uniques.cc:
Updated calls to init_tree(), my_init_dynamic_array() and my_malloc() to mark memory thread specific.
sql/unireg.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
storage/csv/ha_tina.cc:
Updated call to init_alloc_root()
storage/federated/ha_federated.cc:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
Ensure that memory allocated by fedarated is registered for the system, not for the thread.
storage/federatedx/federatedx_io_mysql.cc:
Updated call to my_init_dynamic_array()
storage/federatedx/ha_federatedx.cc:
Updated call to init_alloc_root()
Updated call to my_init_dynamic_array()
storage/heap/ha_heap.cc:
Added MY_THREAD_SPECIFIC to allocated memory.
storage/heap/heapdef.h:
Added parameter to hp_get_new_block() to be able to do thread specific memory tagging.
storage/heap/hp_block.c:
Added parameter to hp_get_new_block() to be able to do thread specific memory tagging.
storage/heap/hp_create.c:
- Internal HEAP tables are now marking it's memory as MY_THREAD_SPECIFIC.
- Use MY_TREE_WITH_DELETE instead of removed option 'with_delete'.
storage/heap/hp_open.c:
Internal HEAP tables are now marking it's memory as MY_THREAD_SPECIFIC.
storage/heap/hp_write.c:
Added new parameter to hp_get_new_block()
storage/maria/ma_bitmap.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_blockrec.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_check.c:
Updated call to init_alloc_root()
storage/maria/ma_ft_boolean_search.c:
Updated calls to init_tree() and init_alloc_root()
storage/maria/ma_ft_nlq_search.c:
Updated call to init_tree()
storage/maria/ma_ft_parser.c:
Updated call to init_tree()
Updated call to init_alloc_root()
storage/maria/ma_loghandler.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_open.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_sort.c:
Updated call to my_init_dynamic_array()
storage/maria/ma_write.c:
Updated calls to my_init_dynamic_array() and init_tree()
storage/maria/maria_pack.c:
Updated call to init_tree()
storage/maria/unittest/sequence_storage.c:
Updated call to my_init_dynamic_array()
storage/myisam/ft_boolean_search.c:
Updated call to init_tree()
Updated call to init_alloc_root()
storage/myisam/ft_nlq_search.c:
Updated call to init_tree()
storage/myisam/ft_parser.c:
Updated call to init_tree()
Updated call to init_alloc_root()
storage/myisam/ft_stopwords.c:
Updated call to init_tree()
storage/myisam/mi_check.c:
Updated call to init_alloc_root()
storage/myisam/mi_write.c:
Updated call to my_init_dynamic_array()
Updated call to init_tree()
storage/myisam/myisamlog.c:
Updated call to init_tree()
storage/myisam/myisampack.c:
Updated call to init_tree()
storage/myisam/sort.c:
Updated call to my_init_dynamic_array()
storage/myisammrg/ha_myisammrg.cc:
Updated call to init_sql_alloc()
storage/perfschema/pfs_check.cc:
Rest current_thd
storage/perfschema/pfs_instr.cc:
Removed DBUG_ENTER/DBUG_VOID_RETURN as at this point my_thread_var is not allocated anymore, which can cause problems.
support-files/compiler_warnings.supp:
Disable compiler warning from offsetof macro.
IN QUERY CACHE CODE
DESCRIPTION:
MySQL Server crashes sporadically when Query Caching is on and
the server has high contention among clients.
ANALYSIS :
Scenario 1:
In Query_cache::move_by_type() when handling RESULT or its related blocks,
Write Lock is acquired on its parent Query block. However the next and prev
pointers are cached in local variables before lock acquisition. In an extremely
high contention scenario there exists a possibility that
Query_cache::append_result_data() is operating on the same query block
and as a consequence might append a new Result block to the end of Result
blocks Linked List of the Query. This would manipulate the next, prev pointers
of the Block being processed in move_by_type(), however the local pointers
still point to previous nodes there by causing Data Corruption leading to crash.
Scenario 2:
In Windows SDK "BOOL" is typedefed as "int" and BOOLEAN is typedefed as
"usigned char". The function pointer definition "srw_bool_func" mistakenly uses
BOOL instead of BOOLEAN thereby virtually making the function
my_TryAcquireSRWLockExclusive() always succeed because only the LSB of EAX
has the actual result of the call, however due to type mismatch all bytes of EAX
are used for evaluation. Again during high contention scenarios in
Query_cache::free_old_query() calls try_lock_writing() on a Query, this call
always succeeds and the query is freed, even though it is used by some other
thread, in this case Query_cache::send_result_to_client() was using it and the
code causes a crash because it accessed free or reallocated memory.
FIX :
Scenario 1:
The next, prev pointers are now accessed only after Lock acquisition in
Query_cache::move_by_type().
Scenario 2:
In the definition of "srw_bool_func" BOOL has been replaced with "BOOLEAN"
IN QUERY CACHE CODE
DESCRIPTION:
MySQL Server crashes sporadically when Query Caching is on and
the server has high contention among clients.
ANALYSIS :
Scenario 1:
In Query_cache::move_by_type() when handling RESULT or its related blocks,
Write Lock is acquired on its parent Query block. However the next and prev
pointers are cached in local variables before lock acquisition. In an extremely
high contention scenario there exists a possibility that
Query_cache::append_result_data() is operating on the same query block
and as a consequence might append a new Result block to the end of Result
blocks Linked List of the Query. This would manipulate the next, prev pointers
of the Block being processed in move_by_type(), however the local pointers
still point to previous nodes there by causing Data Corruption leading to crash.
FIX :
Scenario 1:
The next, prev pointers are now accessed only after Lock acquisition in
Query_cache::move_by_type().
ROBUST AGAINST BUGS IN CALLERS".
Both MDL subsystems and Table Definition Cache code assume
that callers ensure that names of objects passed to them are
not longer than NAME_LEN bytes. Unfortunately due to bugs in
callers this assumption might be broken in some cases. As
result we get nasty bugs causing buffer overruns when we
construct MDL key or TDC key from object names.
This patch makes TDC code more robust against such bugs by
ensuring that we always checking size of result buffer when
constructing TDC keys. This doesn't free its callers from
ensuring that both db and table names are shorter than
NAME_LEN bytes. But at least this steps prevents buffer
overruns in case of bug in caller, replacing them with less
harmful behavior.
This is 5.1-only version of patch.
This patch introduces new version of create_table_def_key()
helper function which constructs TDC key without risk of
result buffer overrun. Places in code that construct TDC keys
were changed to use this function.
Also changed rm_temporary_table() and open_new_frm() functions
to avoid use of "unsafe" strmov() and strxmov() functions and
use safer strnxmov() instead.
ROBUST AGAINST BUGS IN CALLERS".
Both MDL subsystems and Table Definition Cache code assume
that callers ensure that names of objects passed to them are
not longer than NAME_LEN bytes. Unfortunately due to bugs in
callers this assumption might be broken in some cases. As
result we get nasty bugs causing buffer overruns when we
construct MDL key or TDC key from object names.
This patch makes MDL and TDC code more robust against such
bugs by ensuring that we always checking size of result
buffer when constructing MDL and TDC keys. This doesn't
free its callers from ensuring that both db and table names
are shorter than NAME_LEN bytes. But at least these steps
prevents buffer overruns in case of bug in caller, replacing
them with less harmful behavior.
This is 5.5-only version of patch.
Changed code of MDL_key::mdl_key_init() to take into account
size of buffer for the key.
Introduced new version of create_table_def_key() helper function
which constructs TDC key without risk of result buffer overrun.
Places in code that construct TDC keys were changed to use this
function.
Also changed rm_temporary_table() and open_new_frm() functions
to avoid use of "unsafe" strmov() and strxmov() functions and
use safer strnxmov() instead.
and small collateral changes
mysql-test/lib/My/Test.pm:
somehow with "print" we get truncated writes sometimes
mysql-test/suite/perfschema/r/digest_table_full.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/dml_handler.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/information_schema.result:
host table is not ported over yet
mysql-test/suite/perfschema/r/nesting.result:
this differs, because we don't rewrite general log queries, and multi-statement
packets are logged as a one entry. this result file is identical to what mysql-5.6.5
produces with the --log-raw option.
mysql-test/suite/perfschema/r/relaylog.result:
MariaDB modifies the binlog index file directly, while MySQL 5.6 has a feature "crash-safe binlog index" and modifies a special "crash-safe" shadow copy of the index file and then moves it over. That's why this test shows "NONE" index file writes in MySQL and "MANY" in MariaDB.
mysql-test/suite/perfschema/r/server_init.result:
MariaDB initializes the "manager" resources from the "manager" thread, and starts this thread only when --flush-time is not 0. MySQL 5.6 initializes "manager" resources unconditionally on server startup.
mysql-test/suite/perfschema/r/stage_mdl_global.result:
this differs, because MariaDB disables query cache when query_cache_size=0. MySQL does not
do that, and this causes useless mutex locks and waits.
mysql-test/suite/perfschema/r/statement_digest.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_consumers.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/perfschema/r/statement_digest_long_query.result:
md5 hashes of statement digests differ, because yacc token codes are different in mariadb
mysql-test/suite/rpl/r/rpl_mixed_drop_create_temp_table.result:
will be updated to match 5.6 when alfranio.correia@oracle.com-20110512172919-c1b5kmum4h52g0ni and anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y are merged
mysql-test/suite/rpl/r/rpl_non_direct_mixed_mixing_engines.result:
will be updated to match 5.6 when anders.song@greatopensource.com-20110105052107-zoab0bsf5a6xxk2y is merged
The crash happend when combining query cache, prepared statements and using a read only cursor.
sql/sql_cache.cc:
Fixed unlikely error when one adjust query cache size in middle of operation
sql/sql_cursor.cc:
Disable query cache when using cursors. This fixed lp:1039277
tests/mysql_client_test.c:
Test case for lp:1039277
Now partition engine adds underlying tables to the QC and ask underlying tables engine permittion to cache the query and return result of the query.
Incorrect QC cleanup in case of table registration failure fixe.
Unified interface for myisammrg & partitioned engnes for QC.
mysql-test/suite/innodb/t/group_commit_crash.test:
remove autoincrement to avoid rbr being used for insert ... select
mysql-test/suite/innodb/t/group_commit_crash_no_optimize_thread.test:
remove autoincrement to avoid rbr being used for insert ... select
mysys/my_addr_resolve.c:
a pointer to a buffer is returned to the caller -> the buffer cannot be on the stack
mysys/stacktrace.c:
my_vsnprintf() is ok here, in 5.5
sql/share/errmsg-utf8.txt:
correct the error message, as query_cache_type variable is not read-ony anymore
sql/sql_cache.cc:
the caller should verify that query cache resize
is possible, before trying it
sql/sys_vars.cc:
* test if qc resize is possible in the sysvar on_check() funntion,
not in the on_update() function.
* use the error message that better describes the problem
- Changed storage to be 2 bytes instead of sizeof(size_t) (simple optimization)
- Fixed bug when using query_cache_strip_comments and query that started with '('
- Fixed DBUG_PRINT() that used wrong (not initialized) variables.
mysql-test/mysql-test-run.pl:
Added some space to make output more readable.
mysql-test/r/query_cache.result:
Updated test results
mysql-test/t/query_cache.test:
Added test with query_cache_strip_comments
sql/mysql_priv.h:
Added QUERY_CACHE_DB_LENGTH_SIZE
sql/sql_cache.cc:
Fixed bug when using query_cache_strip_comments and query that started with '('
Store db length in 2 characters instead of size_t.
Get db length from correct position (earlier we had an error when query started with ' ')
Fixed DBUG_PRINT() that used wrong (not initialized) variables.
* rename all debugging related command-line options
and variables to start from "debug-", and made them all
OFF by default.
* replace "MySQL" with "MariaDB" in error messages
* "Cast ... converted ... integer to it's ... complement"
is now a note, not a warning
* @@query_cache_strip_comments now has a session scope,
not global.
sql/sql_insert.cc:
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
******
CREATE ... IF NOT EXISTS may do nothing, but
it is still not a failure. don't forget to my_ok it.
sql/sql_table.cc:
small cleanup
******
small cleanup
A buffer large enough to hold the query _plus_ some additional
data is allocated before parsing is started. The additional data
is used by the query cache, and consists of the name of the current
database and a set of flags.
When a packet containing multiple SQL statements is sent to the
server and one of the statements changes the current database
(a "USE <db>" statement), and the name of the new current database
is longer than of the previous, there is not enough space in the
buffer for the new name, and we write out over the buffer boundary.
The fix adds an extra field to store the number of bytes
allocated to the database name in the buffer. If the current
database name changes, and the new name is longer than the
previous one, we refuse to cache the query.
- If USER is given, all threads for that user is signaled
- If SOFT is used then the KILL will not be sent to the handler. This can be used to not interrupt critical things in the handler like 'REPAIR'.
Internally added more kill signals. This gives us more information of why a query/connection was killed.
- KILL_SERVER is used when server is going down. In this case the users gets ER_SHUTDOWN as the reason connection was killed.
- Changed signals to number in correct order, which makes it easier to test how the signal should affect the code.
- New error message ER_CONNECTION_KILLED if connection was killed by 'KILL CONNECTION'. Before we got error ER_SHUTDOWN.
Changed names of not used parameters KILL_QUERY & KILL_CONNCTION to mysql_kill() to not conflict with defines in the server
include/mysql.h.pp:
Updated file
include/mysql_com.h:
Changed names of not used parameters KILL_QUERY & KILL_CONNCTION to mysql_kill() to not conflict with defines in the server
mysql-test/r/kill.result:
Added test of KILL USER
mysql-test/suite/rpl/r/rpl_stm_000001.result:
Updated error code
mysql-test/suite/rpl/t/rpl_stm_000001.test:
Updated error codes
mysql-test/t/flush_read_lock_kill.test:
Updated error codes
mysql-test/t/kill.test:
Added test of KILL USER
plugin/handler_socket/handlersocket/database.cpp:
Removed THD:: from KILL
sql/debug_sync.cc:
Removed THD:: from KILL
sql/event_scheduler.cc:
Removed THD:: from KILL
sql/filesort.cc:
Removed THD:: from KILL
sql/ha_ndbcluster_binlog.cc:
Removed THD:: from KILL
sql/handler.cc:
Removed THD:: from KILL
Simplify code.
sql/lex.h:
Added new keywords HARD | SOFT
sql/log.cc:
Removed THD:: from KILL
Added testing of new error ER_CONNECTION_KILLED
sql/log_event.cc:
Removed THD:: from KILL
Added testing of new error ER_CONNECTION_KILLED
sql/mysql_priv.h:
Added new prototypes
sql/mysqld.cc:
Removed THD:: from KILL
Use KILL_SERVER_HARD signal on shutdown.
sql/scheduler.cc:
Removed THD:: from KILL
Simplify test if connection should be killed
sql/share/errmsg.txt:
New error message ER_CONNECTION_KILLED
sql/slave.cc:
Removed THD:: from KILL
sql/sp_head.cc:
Removed THD:: from KILL
sql/sql_base.cc:
Removed THD:: from KILL
sql/sql_cache.cc:
Removed THD:: from KILL
sql/sql_class.cc:
Removed THD:: from KILL
Added killed_errno()
Only signal kill to storage engine if HARD bit is set.
sql/sql_class.h:
Move KILL options out from THD to make them easier to use in sql_yacc.yy
sql/sql_connect.cc:
Removed THD:: from KILL
sql/sql_delete.cc:
Removed THD:: from KILL
sql/sql_error.cc:
Removed THD:: from KILL
sql/sql_insert.cc:
Removed THD:: from KILL
Simplifed testing if thread is killed.
sql/sql_lex.h:
Added kill options to st_lex
sql/sql_load.cc:
Removed THD:: from KILL
sql/sql_parse.cc:
Added kill options to st_lex
Simplifed and optimzed testing of thd->killed at end of query
Added support for KILL USER
Extended sql_kill() to allow use of more kill signals.
sql/sql_repl.cc:
Removed THD:: from KILL
sql/sql_show.cc:
Removed THD:: from KILL
Simplied testing if query/connection was killed
sql/sql_table.cc:
Removed THD:: from KILL
sql/sql_update.cc:
Removed THD:: from KILL
sql/sql_yacc.yy:
Added support for new KILL syntax: KILL [HARD|SOFT] [CONNECTION|QUERY] [ID | USER user_name]
storage/archive/ha_archive.cc:
Simplify compilation
storage/maria/ha_maria.cc:
Removed THD:: from KILL
Fixed crash when setting query_cache_type to 0.
client/Makefile.am:
Added zlib include (needed by checksum.c)
sql/set_var.cc:
Updated call to disable_query_cache()
sql/sql_cache.cc:
Don't give warning if we start mysqld with --query_cache_type=0 --query_cache-size=0
Fixed crash when setting query_cache_type to 0 (we shouldn't call query_cache.disable_query_cache() when there is no current_thd)
sql/sql_cache.h:
Added THD to disable_query_cache()
client/mysqltest.cc:
Column names.
mysql-test/r/grant_cache_no_prot.result:
fix of text.
mysql-test/r/grant_cache_ps_prot.result:
Fix of test.
mysql-test/r/query_cache.result:
Switching on and off query cache.
mysql-test/t/query_cache.test:
Switching on and off query cache.
mysys/charset.c:
Fix of parser.
sql/handler.cc:
thd added to parameters.
sql/log_event.cc:
thd added to parameters.
sql/log_event_old.cc:
thd added to parameters.
sql/mysql_priv.h:
Fixed functions definitions.
sql/mysqld.cc:
Comments stripping.
sql/set_var.cc:
Switching on and off query cache.
sql/set_var.h:
Switching on and off query cache.
sql/share/errmsg.txt:
New errors.
sql/sql_cache.cc:
Switching query cache on and off, removing comments.
sql/sql_cache.h:
thd added to parameters.
sql/sql_class.h:
Comments stripping.
sql/sql_db.cc:
thd added to parameters.
sql/sql_lex.cc:
lex fixed.
sql/sql_parse.cc:
thd added to parameters.
Resolved all conflicts, bad merges and fixed a few minor bugs in the code.
Commented out the queries from multi_update, view, subselect_sj, func_str,
derived_view, view_grant that failed either with crashes in ps-protocol or
with wrong results.
The failures are clear indications of some bugs in the code and these bugs
are to be fixed.
Only wait for a single debug signal at a time as the signal state
is global. Also, do not activate the query cache debug sync points
if the thread has no associated THD session.
mysql-test/t/query_cache_debug.test:
Only wait for a single debug signal at a time as the signal state
is global.
sql/sql_cache.cc:
Do not execute the debug sync point if the thread has no associated
THD session. This scenario happens for federated threads.
The problem was that threads waiting on the query cache lock
are not easily seen due to the lack of a state indicating that
the thread is waiting on the said lock. This made it difficult
for users to quickly spot (for example, via SHOW PROCESSLIST)
a query cache contention problem.
The solution is to update the thread state when the query cache
lock needs to be acquired. Whenever the lock is to be acquired,
the thread state is updated to "Waiting for query cache lock"
and is reset once the lock is granted or the wait is interrupted.
The intention is to make query cache related hangs more evident.
To further investigate query cache related locking problems, one
may use PERFORMANCE_SCHEMA to track the overhead associated with
the locking bits and determine which particular lock is being a
contention point.
sql/sql_cache.cc:
Set and reset the thread state whenever a attempt to lock the
query cache is made.
Use DEBUG_SYNC instead of the now unnecessary wait_for_kill hack.
thd->in_sub_stmt
In a precursor patch for Bug#52044
(revid:bzr/kostja@stripped), a
number of reorganizations of code was made. In addition some
assertions were added to ensure the correct transactional state.
The reorganization had a small glitch so statements that was
active in the query cache was not followed by a
statement commit/rollback (this code was removed). A section
in the trans_commit_stmt/trans_rollback_stmt code is to
clear the thd->transaction.stmt list of affected storage
engines. When a new statement is initiated, an assert
introduced by the 523044 patch checks if this list is cleared.
When the query cache is accessed, this list may be populated,
and since it's not committed it will not be cleared.
This fix adds explicit statement commit or rollback for
statements that is contained in the query cache.
Fix some bugs where we stored values other than 0 or 1 in my_bool
Fixed some compiler warnings
client/mysql.cc:
Changed interrupted_query from my_bool to int, as we stored 2 in it.
client/mysqladmin.cc:
Changed return variable type to same type as function value type
client/mysqltest.cc:
Changed 'found' to int as we store other values than 0 or 1 into it
Changed type for parameter of set_reconnect() to match usage.
extra/libevent/evbuffer.c:
Added __attribute__((unused))
extra/libevent/event.c:
Added __attribute__((unused))
extra/libevent/signal.c:
Added __attribute__((unused))
sql/event_data_objects.h:
my_bool -> bool
sql/event_db_repository.cc:
my_bool -> bool
sql/event_db_repository.h:
my_bool -> bool
sql/event_parse_data.h:
my_bool -> bool
sql/events.cc:
my_bool -> bool
sql/events.h:
my_bool -> bool
sql/field.cc:
my_bool -> bool
sql/field.h:
my_bool -> bool
sql/hash_filo.h:
my_bool -> bool
sql/item.cc:
my_bool -> bool
sql/item.h:
my_bool -> bool
sql/item_cmpfunc.h:
my_bool -> bool
Changed result_for_null_param from my_bool to int as we stored -1 in it.
sql/item_func.cc:
my_bool -> bool
Modified udf wrapper functions so that the UDF functions would continue to use my_bool. (To keep compatibility with UDF:s)
sql/item_func.h:
my_bool -> bool
sql/item_subselect.h:
my_bool -> bool
sql/item_sum.cc:
Modified udf wrapper functions so that the UDF functions would continue to use my_bool. (To keep compatibility with UDF:s)
sql/parse_file.h:
my_bool -> bool
sql/rpl_mi.h:
my_bool -> bool
sql/sp_rcontext.h:
my_bool -> bool
sql/sql_analyse.h:
my_bool -> bool
sql/sql_base.cc:
Change some assignments so that we don't initialize bool variables with int's.
sql/sql_bitmap.h:
my_bool -> bool
sql/sql_cache.cc:
my_bool -> bool
sql/sql_cache.h:
my_bool -> bool
sql/sql_class.h:
my_bool -> bool
sql/sql_insert.cc:
Change some assignments so that we don't initialize bool variables with int's.
sql/sql_prepare.cc:
my_bool -> bool
sql/table.h:
my_bool -> bool
storage/maria/ma_check.c:
Removed duplicate assignment
strings/decimal.c:
Fixed wrong variable usage.
Don't do complex arithmetic on bool when simple works.
compression protocol.
The loss of connection was caused by a malformed packet
sent by the server in case when query cache was in use.
When storing data in the query cache, the query cache
memory allocation algorithm had a tendency to reduce
the amount of memory block necessary to store a result
set, up to finally storing the entire result set in a single
block. With a significant result set, this memory block
could turn out to be quite large - 30, 40 MB and on.
When such a result set was sent to the client, the entire
memory block was compressed and written to network as a
single network packet. However, the length of the
network packet is limited by 0xFFFFFF (16MB), since
the packet format only allows 3 bytes for packet length.
As a result, a malformed, overly large packet
with truncated length would be sent to the client
and break the client/server protocol.
The solution is, when sending result sets from the query
cache, to ensure that the data is chopped into
network packets of size <= 16MB, so that there
is no corruption of packet length. This solution,
however, has a shortcoming: since the result set
is still stored in the query cache as a single block,
at the time of sending, we've lost boundaries of individual
logical packets (one logical packet = one row of the result
set) and thus can end up sending a truncated logical
packet in a compressed network packet.
As a result, on the client we may require more memory than
max_allowed_packet to keep, both, the truncated
last logical packet, and the compressed next packet.
This never (or in practice never) happens without compression,
since without compression it's very unlikely that
a) a truncated logical packet would remain on the client
when it's time to read the next packet
b) a subsequent logical packet that is being read would be
so large that size-of-new-packet + size-of-old-packet-tail >
max_allowed_packet.
To remedy this issue, we send data in 1MB sized packets,
that's below the current client default of 16MB for
max_allowed_packet, but large enough to ensure there is no
unnecessary overhead from too many syscalls per result set.
sql/net_serv.cc:
net_realloc() modified: consider already used memory
when compare packet buffer length
sql/sql_cache.cc:
modified Query_cache::send_result_to_client: send result to client
in chunks limited by 1 megabyte.
- Changed to still use bcmp() in certain cases becasue
- Faster for short unaligneed strings than memcmp()
- Bettern when using valgrind
- Changed to use my_sprintf() instead of sprintf() to get higher portability for old systems
- Changed code to use MariaDB version of select->skip_record()
- Removed -%::SCCS/s.% from Makefile.am:s to remove automake warnings
to write into a closed socket
sql/protocol.cc:
Protocol::flush modified: set thd->main_da.can_overwrite_status= TRUE
before call to net_flush() in order to prevent crash on assert in case
of socket write failure, reset it to FALSE when net_flush() returned;
Protocol::send_fields modified: return from method with error if call to
my_net_write(), proto.write() or write_eof_packet() failed.
sql/sql_cache.cc:
Query_cache::send_result_to_client modified: call to
thd->main_da.disable_status() only if write to socket
was successful.
sql/sql_cursor.cc:
Materialized_cursor::fetch modified: leave method if call to
result->send_data() failed.
sql/sql_prepare.cc:
send_prep_stmt() modified: call to thd->main_da.disable_status()
only if thd->protocol_text.send_fields() completed successfully.
Essentially, the problem is that safemalloc is excruciatingly
slow as it checks all allocated blocks for overrun at each
memory management primitive, yielding a almost exponential
slowdown for the memory management functions (malloc, realloc,
free). The overrun check basically consists of verifying some
bytes of a block for certain magic keys, which catches some
simple forms of overrun. Another minor problem is violation
of aliasing rules and that its own internal list of blocks
is prone to corruption.
Another issue with safemalloc is rather the maintenance cost
as the tool has a significant impact on the server code.
Given the magnitude of memory debuggers available nowadays,
especially those that are provided with the platform malloc
implementation, maintenance of a in-house and largely obsolete
memory debugger becomes a burden that is not worth the effort
due to its slowness and lack of support for detecting more
common forms of heap corruption.
Since there are third-party tools that can provide the same
functionality at a lower or comparable performance cost, the
solution is to simply remove safemalloc. Third-party tools
can provide the same functionality at a lower or comparable
performance cost.
The removal of safemalloc also allows a simplification of the
malloc wrappers, removing quite a bit of kludge: redefinition
of my_malloc, my_free and the removal of the unused second
argument of my_free. Since free() always check whether the
supplied pointer is null, redudant checks are also removed.
Also, this patch adds unit testing for my_malloc and moves
my_realloc implementation into the same file as the other
memory allocation primitives.
client/mysqldump.c:
Pass my_free directly as its signature is compatible with the
callback type -- which wasn't the case for free_table_ent.
Adding my_global.h first in all files using
NO_EMBEDDED_ACCESS_CHECKS.
Correcting a merge problem resulting from a changed definition
of check_some_access compared to the original patches.
This patch:
- Moves all definitions from the mysql_priv.h file into
header files for the component where the variable is
defined
- Creates header files if the component lacks one
- Eliminates all include directives from mysql_priv.h
- Eliminates all circular include cycles
- Rename time.cc to sql_time.cc
- Rename mysql_priv.h to sql_priv.h
