The Day Two Databases Became Two Truths
Two databases during coexistence are normal. Two writable copies of the same business fact are an architectural incident waiting for a calendar date.
Why Sync Does Not Decide Authority
Teams treat dual databases as a replication problem. If Oracle, SQL Server, DB2, or PostgreSQL can sync rows, the migration is safe. Replication copies data. It does not decide which side is allowed to change it.
For dual-database coexistence, a senior review should ask which coexistence decision is being made, which evidence proves it, and what signal would force the team to pause.
Inventory Drift in the Coexistence Window
A retailer introduced a modern inventory service while store POS inventory stayed on SQL Server. E-commerce reserved stock in PostgreSQL. Store returns still updated SQL Server. A sync job reconciled every fifteen minutes. Peak season exposed the design: the web channel sold units that stores had already adjusted. Last-write-wins became last-customer-loses. The fix was one writer per SKU-location phase, reservation commands through the authoritative domain, and CDC-fed read models.
The Writer Table Comes First
Where Teams Misread the System
Sync is not ownership
A nightly job that copies customers from Oracle to Postgres does not answer: who may change address tomorrow morning?
If legacy CRM still allows edits and the new profile service also allows edits, sync jobs fight application writes. Last writer wins. Auditors ask which store is official. Operations exports both and merges in Excel.
A common failure pattern: the nightly job copies rows correctly, but daytime CSR edits land in legacy while self-service profile updates land in Postgres. The job overwrites one side or skips conflicts until someone notices duplicate addresses on outbound mail. The fix is not a better merge function. It is one writer for address during the wave, with the other side reading from the authoritative store.
Bi-directional sync feels temporary forever
Teams promise "two-way sync until cutover." Cutover moves because parity never clears. Sync complexity grows: conflict resolution rules, exception queues, manual fixes.
Each new edge case becomes another branch in the sync job. The migration dashboard shows green task counts. The business shows wrong balances.
Conflict resolution rules sound sophisticated until the exception queue needs a full-time owner. Teams add "sync admin" screens, manual override flags, and Slack channels for urgent fixes. That is not migration infrastructure. It is a second operations team paid to keep two truths from fighting in public.
Hidden writers
Dual database pain also comes from batch jobs, admin SQL, and integration feeds that write the non-authoritative store. Single-writer rules must cover every path, not only the main API.
Marketing exports that write promo flags into the modern store while pricing still reads legacy is a typical hidden writer. So is a one-off data fix script run against the "copy" database because someone thought it was safer. Inventory every write path in the wave scope, including scripts that run twice a year.
Bad Shape, Better Shape
Bad: two databases, two writers
Legacy DB <----sync----> Modern DB
^ ^
| |
legacy app new service
(both write Customer)
Failures:
- Conflicting updates on the same primary key.
- Sync lag means UIs disagree by minutes or hours.
- Rollback unclear: which database do you restore?
Good: one authoritative writer per entity
Legacy DB (read-only for Customer) ----CDC/read----> Modern DB
^
| writes Customer
new profile service
During wave:
- Modern service owns
Customerwrites in modern DB. - Legacy reads modern API or replica for screens still on old UI.
- Legacy write paths for
Customerdisabled. - Parity jobs compare sampled fields until sign-off.
What the Pattern Teaches
Databases are where truth lives. Two writable copies of the same entity without rules is not a migration tactic. It is reconciliation as a lifestyle.
CTOs should ask the same question at the storage layer as at the application layer: for each entity, which database row is authoritative, and who is allowed to change it?
If the answer is "sync keeps them aligned," push until someone names a single writer.
Schema differences make the problem worse. Legacy may store customer status as a numeric code; modern may use enums. Sync maps values until a new code ships on one side only. Parity checks on primary keys are not enough. Compare the fields the business actually reads in reports and screens.
Worked Example: fintech client balances
Legacy core holds balances. A new advisory portal writes to a Postgres profile store "for speed."
| Setup | Nightly job | Daytime experience |
|---|---|---|
| Dual write | reconcile differences | advisors see two numbers |
| Single writer (core) | one-way publish to portal read model | one balance, lag bounded |
| After cutover | portal store owns profile; core owns ledger | clear boundaries |
Reconciliation spreadsheets stopped when the core was named writer for balance and the portal stopped silent updates. The portal team initially resisted read-only mode because "Postgres is faster for profile edits." Leadership held the line: ledger-adjacent fields stay on the core until a signed parity window passed and write ownership moved in a later wave.
| Phase | Writer for balance | Portal behavior |
|---|---|---|
| Before fix | both stores | advisors saw mismatches daily |
| Interim | legacy core | portal reads published balance |
| Target | core then portal for profile only | ledger fields never dual-written |
Where This Shows Up: fintech and retail
Fintech: ledger-adjacent data cannot tolerate ambiguous authority. Dual stores without writer rules break advisor trust and regulatory reporting timelines. A balance that differs between CRM and core by even a small rounding rule triggers client calls and compliance review. Single-writer plus one-way publish does not remove lag entirely, but it makes lag visible and bounded instead of random.
Retail: store and e-commerce inventory often sit in different databases during omnichannel programs. Without a single writer per SKU-location, promotions oversell quietly. The web channel sells the last unit while the store POS still shows stock because each database received partial updates. Buy-online-pickup-in-store magnifies the pain: the customer is standing at the counter when the two inventories disagree.
The pattern matches application single-writer rules; only the storage diagram changes. Fix the writer table first, then simplify sync.
Where Write Authority Lives
Bad coexistence:
Legacy DB <----sync----> Modern DB
^ ^
Legacy app Modern service
Controlled coexistence:
Writer DB ----CDC/outbox----> Read model
^ ^
Only writer Consumers read
The Lag Versus Truth Tradeoff
The tradeoff is bounded lag versus ambiguous truth. A one-way projection may be minutes behind. Dual writes can be instantly wrong.
For dual-database coexistence, the useful review is not a generic architecture checklist. It should inspect writer ownership, parity evidence, wave boundaries, rollback, and retirement. If those fields are missing, the team may still be busy, but leadership does not yet have a decision-quality artifact.
What Leaders Should Inspect
For dual-database coexistence, the release review should not start with percentage of traffic shifted or number of services created. It should start with the ownership table. For every entity touched by the wave, name the current writer, the target writer, the read paths that remain on legacy, the jobs that still mutate data, and the condition that disables those jobs. If a batch process, admin SQL script, or vendor feed can still update the entity, the wave is not ready.
For dual-database coexistence, parity evidence should sample production-shaped cases by business segment, not only random identifiers. Compare the values that users, auditors, reports, and downstream integrations care about. In financial services, that means balances, statuses, fee calculations, disclosures, and event timestamps. In manufacturing, it means lead times, lot traceability, reservation state, and label outputs. HTTP 200 on old and new paths is not parity. Matching business outcomes under realistic edge cases is parity.
For dual-database coexistence, rollback design is simple when the old writer never stopped writing. It is difficult when writes have moved. Decide whether rollback means routing traffic back to legacy, replaying events, freezing writes, or operating a manual exception queue. If rollback would reintroduce two writers, the safer response may be fail-closed and human handling rather than automatic reversal.
For dual-database coexistence, require a retirement ticket in the same wave plan as the build ticket. Modernization without retirement is accumulation. The old path may remain read-only for audit or reporting, but writable legacy behavior should have a deletion or disablement date tied to evidence.
Bad Paths to Test
Dual-database coexistence test plans should include concurrent writes, stale reads, replay after partial failure, and rollback after ownership moves. Test a legacy batch job running while the modern service writes the same entity. Test a support-user override, not only a customer-facing request. Test what happens when CDC pauses for an hour and then resumes. Test deletion, merge, and correction flows because those are where hidden writers often appear.
For dual-database coexistence, sample cases should include the ugly tail: old product codes, inactive customers, manually corrected orders, regional exceptions, and records created before the current schema existed. Those cases are not edge noise. They are often the reason the legacy system survived for years.
Coexistence Patterns Worth Comparing
Read-through modernization avoids duplicate storage but preserves legacy limits. Projection-first works for analytics and low-risk reads. Entity or cohort cutover moves real ownership but requires strict guards that block hidden writers such as batch jobs and admin scripts.
In dual-database coexistence, the alternative paths are not steps on a ladder. Each one carries a different mix of risk, cost, and learning. The weak choice is the one that hides the tradeoff until users, operators, or auditors discover it for you.
The Rule: One Fact, One Writer, One Phase
Two databases during coexistence are normal. Two writable copies of the same business fact are an architectural incident waiting for a calendar date. The practical lesson is to demand evidence that fits dual-database coexistence, not a universal checklist. The artifact should expose writer ownership, parity evidence, wave boundaries, rollback, and retirement clearly enough for another team to challenge the decision.
If dual-database coexistence is the decision in front of your team, use the Modernization Snapshot to pressure-test the boundary before it hardens.