MariaDB Galera Cluster: A Critical Analysis of Safety and Consistency

TL;DR: MariaDB Galera Cluster, despite claims of strong consistency, exhibits serious safety issues including lost writes, stale reads, and lost updates. Users should be aware of these limitations and configure appropriately.

Executive Summary

MariaDB Galera Cluster promises synchronous replication and strong consistency, but our analysis reveals significant safety concerns that contradict these claims. The system frequently loses committed transactions during coordinated failures, occasionally loses data even with proper configuration, and routinely violates stronger consistency models like Snapshot Isolation and Repeatable Read—even in healthy clusters.

These findings are particularly concerning given MariaDB’s documentation suggesting Galera provides “instant” replication with “no lost transactions.” Our testing demonstrates this is simply not true in practice.

Background: What is MariaDB Galera Cluster?

MariaDB Galera Cluster is an active-active replication system that allows reads and writes on every node. Built on the Galera replication library, it uses a certification-based replication approach where transactions are executed optimistically and then certified across nodes based on primary key conflicts.

The system is designed to provide synchronous replication, meaning transactions should be committed across all nodes before acknowledgment. However, this claim is misleading—Galera continues to operate when a minority of nodes fails, which is impossible if every transaction truly required commitment on all nodes.

Key Findings

1. Write Loss During Coordinated Failures (MDEV-38974)

When all nodes crashed simultaneously, MariaDB Galera Cluster regularly lost committed transactions. For example, during a one-minute test, the cluster lost nine values appended to three different rows.

This occurs because Galera’s recommended configuration sets innodb_flush_log_at_trx_commit = 0, which means transaction logs may not be flushed to disk before acknowledgment. While this improves performance, it creates vulnerability during coordinated failures (power outages, cooling failures, etc.).

Recommendation: Set innodb_flush_log_at_trx_commit = 1 to reduce write loss probability.

2. Write Loss During Process Crashes and Partitions (MDEV-38976)

Even with innodb_flush_log_at_trx_commit = 1, the system occasionally lost committed transactions when process crashes and network partitions occurred together. In one instance, approximately nineteen seconds of writes across four objects were lost.

This issue appears infrequently but demonstrates that Galera’s safety guarantees are weaker than claimed.

3. Lost Update Violations (MDEV-38977)

Under normal operation, Galera allowed Lost Update anomalies where one transaction’s writes could be completely overwritten by another transaction without detection. This violates both Snapshot Isolation and Repeatable Read consistency models.

For example, a transaction could read a value, another transaction could modify it, and the first transaction could overwrite those changes without awareness—exactly the scenario Snapshot Isolation is designed to prevent.

4. Stale Reads (MDEV-38999)

Even in healthy clusters without faults, Galera occasionally allowed stale reads where a committed transaction’s writes were not visible to a subsequent transaction that began afterward. This directly contradicts claims of “instant” replication and “no replica lag.”

Safety Analysis

The core issue is that MariaDB Galera Cluster provides a consistency model weaker than Read Uncommitted, despite claims of providing something “between Serializable and Repeatable Read.” The system’s actual behavior suggests it may not even meet the requirements of basic Read Committed isolation.

Critical Safety Issues:

• Committed transactions can be lost during coordinated failures
• Committed transactions can be lost during process crashes + network partitions
• Lost Update violations occur regularly in healthy clusters
• Stale reads occur regularly in healthy clusters
• Documentation is misleading about consistency guarantees

Configuration Recommendations

1. Set innodb_flush_log_at_trx_commit = 1 – This dramatically reduces write loss during coordinated failures, though it doesn’t eliminate it entirely.

2. Understand the limitations – Even with proper configuration, expect occasional write loss during complex failure scenarios.

3. Avoid read-modify-write patterns – Since Lost Update violations occur, patterns common in ORMs and application code may produce incorrect results.

4. Don’t assume committed transactions are visible – Always verify that your application can handle stale reads.

Documentation Issues

MariaDB’s documentation contains several problematic claims:

• “Data is consistent across all nodes at all times” – False, as demonstrated by stale reads
• “No lost transactions” – False, as demonstrated by multiple write loss scenarios
• “Instantly replicated to all other nodes” – False, as demonstrated by stale reads
• “Transactions work as expected” – False, as demonstrated by Lost Update violations

The documentation also buries the only mention of isolation levels in a section about converting to Galera, making it difficult for users to understand what consistency guarantees they’re actually getting.

Consistency Model Analysis

Based on our testing, MariaDB Galera Cluster provides a consistency model weaker than Read Uncommitted, despite claims of providing something stronger than Repeatable Read. The system’s behavior suggests it may not even meet the requirements of basic Read Committed isolation.

This is particularly concerning given that Galera is marketed as providing “synchronous” replication, which typically implies stronger consistency guarantees.

Future Work

Our testing focused on primary key operations using string concatenation. Additional areas for investigation include:

• Blind writes to registers – Likely to exhibit Lost Update similar to our findings
• Banking workloads – Money could be created or destroyed out of thin air
• Predicate operations – How Galera handles range queries and complex WHERE clauses
• Slow networks – Impact on consistency under high latency
• Clock skew – How Galera handles timing issues across nodes
• Disk faults – How the system behaves with storage failures

Conclusion

MariaDB Galera Cluster does not provide the consistency and safety guarantees claimed in its documentation. Users should be aware of the following:

• Write loss can occur during coordinated failures (mitigated by proper configuration)
• Write loss can occur during complex failure scenarios even with proper configuration
• Lost Update violations occur regularly in healthy clusters
• Stale reads occur regularly in healthy clusters
• The system’s actual consistency model is weaker than Read Uncommitted

While Galera Cluster may still be useful for applications that can tolerate these limitations, users should not rely on its claimed consistency guarantees. The documentation should be revised to accurately reflect the system’s actual behavior and limitations.

Viral Tags & Phrases

• “MariaDB Galera Cluster loses committed transactions”
• “Synchronous replication that isn’t actually synchronous”
• “Database safety issues that could cost you money”
• “When your database lies about consistency”
• “The hidden dangers of Galera Cluster”
• “Database failures that create money out of thin air”
• “Why your transactions might disappear”
• “The consistency model that doesn’t exist”
• “Database documentation that misleads users”
• “When ‘instant replication’ means ‘sometimes it works'”

Key Takeaways

1. Don’t trust the documentation – Galera’s claimed consistency guarantees are not met in practice
2. Configure properly – Set innodb_flush_log_at_trx_commit = 1 to reduce write loss
3. Design defensively – Assume Lost Update and stale reads can occur
4. Test thoroughly – Don’t assume Galera provides the consistency you need
5. Consider alternatives – If you need true strong consistency, look elsewhere

This analysis represents our findings from a brief exploration of Galera Cluster. As with all Jepsen analyses, we can prove the presence of bugs but not their absence. Users should conduct their own testing to verify these findings in their specific environments.

,