image.png
Mutation功能是MergeTree表引擎最新推出一大功能，实现完备度上还有一下两点需要去优化：
1.mutation没有实时可见能力。这里的实时可见并不是指在存储上立即原地更新，而是给用户提供一种途径可以立即看到数据订正后的最终视图确保订正无误。类比在使用CollapsingMergeTree、SummingMergeTree等高级MergeTree引擎时，数据还没有完全merge到一个Data Part之前，存储层并没有一个数据的最终视图。但是用户可以通过Final查询模式，在计算引擎层实时聚合出数据的最终视图。这个原理对mutation实时可见也同样适用，在实时查询中通过FilterBlockInputStream和ExpressionBlockInputStream完成用户的mutation操作，给用户提供一个最终视图。
2.mutation和merge相互独立执行。看完本文前面的分析，大家应该也注意到了目前Data Part的merge和mutation是相互独立执行的，Data Part在同一时刻只能是在merge或者mutation操作中。对于MergeTree这种存储彻底Immutable的设计，数据频繁merge、mutation会引入巨大的IO负载。实时上merge和mutation操作是可以合并到一起去考虑的，这样可以省去数据一次读写盘的开销。对数据写入压力很大又有频繁mutation的场景，会有很大帮助。
image.png
对于第2点，这里我们不禁又回想起clickhouse官方文档对于参数background_pool_size的说明：
image.png
这里提到了额外的两个参数:
number_of_free_entries_in_pool_to_execute_mutation
number_of_free_entries_in_pool_to_lower_max_size_of_merge
M(UInt64, number_of_free_entries_in_pool_to_lower_max_size_of_merge, 8, "When there is less than specified number of free entries in pool (or replicated queue), start to lower maximum size of merge to process (or to put in queue). This is to allow small merges to process - not filling the pool with long running merges.", 0) \
M(UInt64, number_of_free_entries_in_pool_to_execute_mutation, 10, "When there is less than specified number of free entries in pool, do not execute part mutations. This is to leave free threads for regular merges and avoid \"Too many parts\"", 0) \
这两个参数怎么讲？和background_pool_size有什么关联，其实很简单，刚才提到因为后台的merge和mutation是一个线程池来调度的，所以参数number_of_free_entries_in_pool_to_execute_mutation的大概意思，是预留出足够的线程数量去做mutation，如果线程buffer不够，则不执行，这个会尽可能规避too many parts的现象。（侧面说明目前merge工作不繁重，这个值调到合适的水准，会让系统后台尽量优先做merge工作）
std::shared_ptr<StorageMergeTree::MergeMutateSelectedEntry> StorageMergeTree::selectPartsToMutate(const StorageMetadataPtr & metadata_snapshot, String */* disable_reason */, TableLockHolder & /* table_lock_holder */)
{
...
for (const auto & part : getDataPartsVector())
{
if (currently_merging_mutating_parts.count(part))
continue;
auto mutations_begin_it = current_mutations_by_version.upper_bound(part->info.getDataVersion());
if (mutations_begin_it == mutations_end_it)
continue;
//这个函数做了判断
size_t max_source_part_size = merger_mutator.getMaxSourcePartSizeForMutation();
if (max_source_part_size < part->getBytesOnDisk())
{
LOG_DEBUG(log, "Current max source part size for mutation is {} but part size {}. Will not mutate part {}. "
"Max size depends not only on available space, but also on settings "
"'number_of_free_entries_in_pool_to_execute_mutation' and 'background_pool_size'",
max_source_part_size, part->getBytesOnDisk(), part->name);
continue;
}
...
tagger = std::make_unique<CurrentlyMergingPartsTagger>(future_part, MergeTreeDataMergerMutator::estimateNeededDiskSpace({part}), *this, metadata_snapshot, true);
return std::make_shared<MergeMutateSelectedEntry>(future_part, std::move(tagger), commands);
}
return {};
}
UInt64 MergeTreeDataMergerMutator::getMaxSourcePartSizeForMutation() const
{
const auto data_settings = data.getSettings();
size_t busy_threads_in_pool = CurrentMetrics::values[CurrentMetrics::BackgroundPoolTask].load(std::memory_order_relaxed);
/// DataPart can be store only at one disk. Get maximum reservable free space at all disks.
UInt64 disk_space = data.getStoragePolicy()->getMaxUnreservedFreeSpace();
/// Allow mutations only if there are enough threads, leave free threads for merges else
if (busy_threads_in_pool <= 1
|| background_pool_size - busy_threads_in_pool >= data_settings->number_of_free_entries_in_pool_to_execute_mutation)
return static_cast<UInt64>(disk_space / DISK_USAGE_COEFFICIENT_TO_RESERVE);
return 0;
}
彩蛋
在本文的开头提到：
保存Mutation命令到存储文件中，唤醒一个异步处理merge和mutation的工作线程；
我们实操看看效果：
xiejinke.local :) ALTER TABLE SignReplacingMergeTreeTest update name='王码子' where id = 15;
ALTER TABLE SignReplacingMergeTreeTest
UPDATE name = '王码子' WHERE id = 15
Query id: 292c6b52-e03d-40e7-8c74-a5750e9b0b54
Ok.
0 rows in set. Elapsed: 20.909 sec.
xiejinke.local :) ALTER TABLE SignReplacingMergeTreeTest update name='王码子333' where id = 15;
ALTER TABLE ReplacingMergeTreeTest
UPDATE name = '王码子333' WHERE id = 15
Query id: c16987b5-8273-44a5-9fd2-5ac68c60a20b
Ok.
0 rows in set. Elapsed: 49.775 sec.
image.png
来看看文件：
image.png
image.png
参考文章：
阿里云：ClickHouse内核分析-MergeTree的Merge和Mutation机制
https://developer.aliyun.com/article/762090?spm=a2c6h.12873581.0.0.29cc802f1GeMHc&groupCode=clickhouse
background_pool_size官方解释：
https://clickhouse.tech/docs/en/operations/settings/settings/#background_pool_size
