9-week build closed · cutover week complete · handoff prep begins
The nine-week build shipped everything within scope plus the P4a Cortex enablement + P4b activation scaffolding + full DC parity coverage. W10 was the cutover / hardening week: absorb Rich's DC-live QA (5 issues expanded to 12), reconcile every reported gap against DC's own snapshot, and close every fixable delta before DC sunset on July 1st. All 5 PRs merged (including the ET J.6 orphan-recovery SALES_V UNION, which turned out to be a source-integrity fix Rich called "an excellent finding"). Two 30-minute Rich sessions this week locked (1) SFMC delay 4–6 weeks due to fixed-IP vs shared-IP-pool IP warming, (2) handoff scope narrowed to SF + Snowflake + dbt setup + tagging only (business will redefine segments in the new MC), (3) 200K residual orphans in WJ + Academy are pre-2018 legacy — Rich contacting Jen. P5 cutover (SFMC bidirectional S3) blocked on SFMC admin account provisioning (Rich promised). P6 handoff docs scaffolded.
Receiving team unchanged: Mike Grabbe + Adrian LeDoux (ET Data Engineering). NTE 188.0% end-of-W10 · user explicit override 2026-06-08 stands. Re-amendment conversation overdue ahead of P5 SFMC connection + P6 hypercare cycles.
What landed in W10 · 5 PRs merged · 7 tickets closed
- 1Monday · Rich RLT-4816 (5 issues → 12) triage + GH #336 + PR #4493 scoping (W10 D1 Mon · 7.0h). Rich emailed 5 CI parity findings vs DC live. Ran fresh Snowflake queries against the DC live share to verify each. Result: 3 real bugs (mart CI 32K duplicate rows · UC/GBR emitted the SAME column with −44% delta · EFSA
student_indie54 vs DC 118K · CCAPactive_opp+9,073%) · 2 DC-inherited semantics (GL teacher/non-teacher overlap 14,420 · WJstudent_indie100%). Opened GH #336 "Rich QA findings triage" with 5-issue table + evidence queries · cross-linked with RLT-4816. PM: Rich expanded ticket to 12 issues. Investigated all 12 · read DC live JSON formulas fromdocs/sf-analysis/calculated-insights/dc-live-formulas/for each · confirmed which are DC-faithful vs real bugs. Started PR #4493 comprehensive-fix design. - 2Tuesday · PR #4493 comprehensive fix (11 files · 6 new snapshots) + 7 RLT tickets closed + DC live baseline captured pre-sunset (W10 D2 Tue · 10.0h). PR #4493 landed on
fix/336-rich-qa-comprehensive: 4 new snapshot models ·int_ir__dc_active_oppcloses CCAP +9,073% → exact ·int_ir__dc_enroll_not_travcloses WJ −100% + EFSA/GY/CCAP gaps ·int_ir__dc_student_indiecloses EFSA −99.95% ·int_ir__dc_adult_non_travelerfortifies GY/Academy. Plus 2 GBR/UC split snapshots:int_ir__dc_global_business_rule+int_ir__dc_unified_contactable— post-fix both = 3,956,949 exact DC match (was buggy 2,189,344 both same column). GL teacher branch addsAccount_Id IS NOT NULLfilter (Rich mut-excl). WJ Option B:wj_gc_only_contactsCTE excludes contacts whose every WJ SO hascustomer_type__c = 'GROUP_COORDINATOR'(−6,813 UIs · 8,640,619 vs DC 8,638,345). Mart adds 3 global passthroughs (currently_traveling / hosting / TRH_host_family).sedrename<bu>_closed_lost_30d→<bu>_closed_lost_last_30_days(Rich naming, 8 BU cols) +latest_tour_*→latest_program_*(RLT-3575 · 48 mart aliases · 8 BUs × 6 metrics). 6 commits · 4 augmentcode auto-fix rebase conflicts resolved · AL03 aliases added to 4 UNION ALL snapshot models. Merged 2026-07-01 06:57 UTC. DC live baseline written todocs/dc-sunset/dc_live_baseline_2026-06-30.md— immutable post-sunset regression reference · 24 metric snapshots. Closed 7 RLT tickets to Done viatransitionJiraIssue: RLT-3169 (currently_traveling/hosting) · RLT-3168 (TRH_host_family) · RLT-3575 (Tour→Program) · RLT-3667 (CAXed naming) · RLT-3971 (Related_Contact) · RLT-3988 (Contact Exists BU/Subunit) · RLT-4816 (Rich closed). RLT-3974 (Referral CI) kept On Hold (Phase I deprioritised). - 3Wednesday · PR #4512 (GL mut-excl projection-level fix) + PR #4525 (Email_Unsub US canonical) + Rich session #1 (SFMC delay · non-boolean unsub) + RLT-4926 close + GBR reverse-engineering + GH #337/#338/#339 (W10 D3 Wed · 8.0h). Post-#4493 audit revealed the GL mut-excl fix was silently no-op — the final SELECT reads
is_group_leader_teacherfromdc_gltsnapshot which discarded theAccount_Id IS NOT NULLfilter I had placed on the CTE. PR #4512 enforces mut-excl at projection level viacase when non_teacher=1 then 0 else dc_glt.teacher end. Overlap 14,420 → 0. Merged 14:31 UTC. · PR #4525 email_unsub: DC'sEmail_Unsubscribe__cioinflates 95-99% per BU (WJ 98.9% · ET 95.4%) because ET's rawEmail_Unsub__cis a counter (0-4) not boolean (values = number of products unsubscribed from · verified in Rich session). US canonicalint_canonical__contact_point_email.Email_Unsub__cpreserves the boolean at 16-33% per BU. Swapcoalesce(dc_eu.is_email_unsubscribed, 0)→coalesce(i.is_email_unsubscribed, 0)reading fromindividual_per_buCTE. Expected post-merge: WJ 8.6M → 2.6M · ET 5.4M → 241K. · Rich 30-min session #1 (seememory/2026-07-01-rich-session.md): new SFMC delayed 4-6 weeks (ET fixed IP vs shared IP pool · IP warming · Andrew Tiemann meeting needed for unsub strategy) · no segment pre-building (business will redefine in new MC) · handoff scope narrowed to SF + Snowflake + dbt setup + tagging only · IR will never exact-match DC (fuzzy match temperature differs). · RLT-4926 closed:currently_travelinggap −26% turned out to be time-dependency — Rich's baseline was 2026-06-29 · DC live TODAY = 14,518 (dropped 61% in 48h). Our dbt 30,787 is BETWEEN both · MORE inclusive than DC live. Rich accepted and closed. · GBR reverse-engineered: literal AND-chain in DC JSON gives 2,189,344 but DC snapshot output = 3,956,949 · confirmed DC uses proprietary formula ≠ published JSON. Best reverse-engineered guess: strong_identity + no_gl + no_active_opp + no_enrolled + (any_trh OR trh_adult) = 3,474,891 (-12% vs DC snapshot). Documented for post-cutover Hybrid COALESCE. · 3 GH tickets opened: #337 [REMEDIATION] Email_Unsubscribe SFMC-source refactor (blocked by #167) · #338 [AUDIT] W10 pre-cutover QA runner + Ygritte results (RLT-4889) · #339 [REMEDIATION] Hybrid COALESCE for unified_contactable post-sunset scalability. - 4Thursday · PR #4534 (dbt tests + handoff notes) + PR #4537 (ET orphan opp recovery via SALES_V UNION · 100% of 35,740 orphans) + Rich session #2 + Academy investigation + GH #340 (W10 D4 Thu · 8.0h). PR #4534: 2 dbt tests + 9-section handoff doc.
test_sales_order_orphan_opp_rate.sqlper-BU (Academy excluded — 97.7% is persistent baseline not degradation; thresholds WJ<50% · ET<15% · others<20%) +test_splink_ir_tier1_hit_rate.sqlmonitors drift (zero-div guarded).docs/dc-sunset/handoff-notes.mdcovers Splink Hybrid B tiering + when to retrain · J.6 source integrity + evidence · data freshness advantage · Splink drift metrics · email_unsub PR #4525 rationale + source field map per BU · GH #339 Hybrid COALESCE plan · contact-point marts sibling refactor · references. · PR #4537 ET orphan opp recovery (RLT-4889 J.6): Rich flagged 675,657 orphan SOs · verified DC has same 670,591 rate (matches within 0.75%) — proves source-integrity issue not our regression. InvestigatedLANGUAGE_KAFKA.JUNO_OPPORTUNITY_LATEST(0 match) ·WOJO.PRT_LOYALTY_OPPORTUNITY(0) ·SALESFORCE_*.OPPORTUNITY(0). DiscoveredDATA_HUB_SHARE.SALES_V(4.4M rows) contains 100% of the 35,740 ET orphan opp IDs. Extendedstg_student_tours__opportunitiesto UNION primary source + recovery from SALES_V (anti-join · sustainable for future data · minimal enrichment fields NULL). Expected: ET orphan_rate 10.4% → 0% · global J.6 1.07M → ~632K. · WJ + Academy triage: WJ orphan IDs are 18-char SF alphanumeric (006...) · not in Phoenix numeric sources · foundWORLD_JOURNEYS.OPPORTUNITIES_Vin US DataHub (1.87M rows · matches format) but NOT replicated to CH. Academy 100% NULLopportunityid__cANDorder_id__cat source — canonical already correctly mapsapplication_opportunity_idfor 1,447 rows · remaining 60,559 are truly NULL pre-2018 legacy. GH #340 [REMEDIATION] opened + moved to In Progress: ReplicateWORLD_JOURNEYS.OPPORTUNITIES_Vto CH for WJ orphan recovery (blocker: data eng · same pattern as ET SALES_V access). · Rich 30-min session #2 (seememory/2026-07-02-rich-session.md): ET fix validated as "excellent finding" · SFMC unsub rates now reasonable (WJ 30% · ET 4.2%) · SFMC admin account promised to enable bidirectional S3 setup · Rich to contact Jen re pre-2018 WJ/Academy legacy · Rich off Fri holiday · target complete pending work by early next week. - 5Friday · #4525 + #4534 merged · #4537 AL03 lint fix + rebase-conflict resolution + Allan reply (W10 D5 Fri · 3.0h). Verified merge queue processed #4525 (email_unsub US canonical) and #4534 (tests + handoff docs) · both landed successfully on
main· PROD CD auto-triggered. · PR #4537: Allan approved but CI failed with AL03 (Column expression without alias · line 127) — theselect 1inside the NOT EXISTS anti-join probe needed an explicit AS. Fixed by addingselect 1 as anti_join_probe. Rebase conflict with augmentcode bot's autofix (bot addedas pon the source alias but kept the bareselect 1· lint still failed). Resolved conflict taking both fixes (myas anti_join_probe+ bot'sas p). Force-pushed6dfbbc974. Full CI passed. Replied Allan confirming ready to merge. - 6Saturday · Sunday · Off (W10 D6-D7 · 0h). Personal time. Rich also off Friday for holiday. Target for W11 D1 (Mon): confirm #4537 merged, run parity dashboard against fresh post-sunset PROD, kick off SFMC admin account handoff with Rich.
What we need from EF / ET
PR #4537 (ET orphan opp recovery via SALES_V UNION) is the last unmerged W10 PR. All checks are green (QA Build · Meta Tests · Compile · Lint · CI Summary). Merge ships the ET orphan_rate 10.4% → 0% fix — the source-integrity solution Rich called out as an excellent finding. Once merged, PROD CD refresh will validate the recovery numerically.
SFMC admin account for me was promised in Rich session #2 (2026-07-02). It unblocks the ~1 day of work needed to set up the bidirectional S3 share (Snowflake CH → S3 → SFMC segments · SFMC engagement events → S3 → Snowflake CH). Also gates the Andrew Tiemann meeting on unsub strategy (central SFMC unsub → source products · CSV-based today).
End-of-W10: cumulative ~$47,005 vs $25k NTE = 188.0% (over cap by ~$22,005). User explicit override 2026-06-08 authorises continued work through cutover. P5 SFMC bidirectional S3 (~1 day post admin access) + P6 hypercare (2 cycles) + Andrew Tiemann unsub strategy consult + Jen consult on WJ/Academy pre-2018 orphans still ahead.
RLT-4816 (5 → 12 issues) · every fixable gap closed
| # | Rich finding | Type | Resolution | Status |
|---|---|---|---|---|
| 1 | GL teacher/non-teacher overlap 14,420 | Real bug | PR #4493 (CTE filter) + #4512 (projection-level mut-excl) | FIXED |
| 2 | Email_Unsub 99% per BU (ET/WJ) | DC snapshot broken | PR #4525 · swap to US canonical (i.is_email_unsubscribed) — Rich confirmed ET raw is 0-4 counter not boolean | FIXED |
| 3 | UC vs GBR both same column −44% | Real bug | PR #4493 · split into 2 separate snapshots · both = 3,956,949 exact DC match | FIXED |
| 4 | CCAP active_opp +9,073% | DC-faithful gap | PR #4493 · int_ir__dc_active_opp snapshot · exact match | FIXED |
| 5 | EFSA student_indie 54 vs 118K | DC-faithful gap | PR #4493 · int_ir__dc_student_indie snapshot · exact match | FIXED |
| 6 | Missing cols currently_traveling / hosting / TRH_host_family | Mart gap | PR #4493 · 3 global passthroughs added | FIXED |
| 7 | closed_lost naming | Naming | PR #4493 · <bu>_closed_lost_30d → <bu>_closed_lost_last_30_days (8 BU cols) | FIXED |
| 8 | tour_cancelled → Tour→Program (RLT-3575) | Naming | PR #4493 · 48 mart aliases across 8 BUs × 6 metrics | FIXED |
| 9 | fall_out per BU gaps | DC-faithful gap | PR #4493 · int_ir__dc_enroll_not_trav closes WJ + EFSA + GY + CCAP | FIXED |
| 10 | enrolled_not_travelled per BU gaps | DC-faithful gap | PR #4493 · same snapshot as #9 | FIXED |
| 11 | WJ Indie GC-only inflation 8.71M | Semantic bug | PR #4493 · Option B · wj_gc_only_contacts exclusion · 8.640M vs DC 8.638M | FIXED |
| 12 | Adult_Non_Traveler per BU gaps | DC-faithful gap | PR #4493 · int_ir__dc_adult_non_traveler · fortifies GY/Academy | FIXED |
| J.6 | Orphan SOs 675K (35,740 ET) | Source integrity | PR #4537 · SALES_V UNION anti-join · 100% recovery of ET orphans | SHIPPED · pending merge |
| J.6 | Orphan SOs (563K WJ · 60K Academy residual) | Source integrity | GH #340 (WJ replication) · Academy legacy pre-2018 accepted (Rich → Jen) | Post-cutover |
12 of 12 Rich QA issues closed in-repo · J.6 ET half shipped · WJ/Academy residuals moved to post-cutover follow-ups (#340). Rich session #2 explicitly validated the SALES_V approach as an "excellent finding" and confirmed WJ/Academy 200K residuals are likely pre-2018 legacy pending Jen consult. Rich closed RLT-4816 + RLT-4926 during the week.
$25 k NTE · spent vs remaining
NTE consumption
NTE CAP CROSSED · 188.0%. Cap was crossed mid-W7. User explicit override 2026-06-08 stands. W10 burn 36.0h (Mon-Fri) roughly matches W9's 36.5h — Tue (10.0h) was the heaviest day (PR #4493 comprehensive fix). Re-amendment conversation overdue ahead of P5 SFMC connection + P6 hypercare cycles.
Scope reality vs $25 k NTE
| Bucket | SOW low (h) | Actual h | Actual $ | Status |
|---|---|---|---|---|
| P0 · M1 Foundation (W1) | 16 | 14.0 | $2,040 | Done |
| P1 · dbt #1 normalize + replication (W2-6) | 25 | 112.0 | $16,180 | Done · #158 closed 54/54 parity |
| P2 · IR Phase 1 + Hybrid B (W6-8) | 37 | ~28.0 | ~$3,920 | Hybrid B AC7 PASS · sunset-safe |
| P3 · Canonical bodies (W6-7) | 37 | ~25.0 | ~$3,500 | Done · 7 bodies all AC7 PASS |
| P3b · CIs consolidation (W7-9) | - | ~52.0 | ~$7,280 | Full DC parity · 12/12 Rich issues closed |
| P4a · Cortex enablement + CI completeness (W9-10) | - | ~22.0 | ~$3,080 | PR #4458 cross-BU JOINs · #4476 30/30 DC parity |
| P4b · SFMC + Marts (W7-11) | 56 | ~5.0 | ~$700 | Activation mart scaffolded · admin account pending |
| P5 · Cutover (W10-11) | 14 | ~13.0 | ~$1,820 | DC sunset survived · handoff notes shipped · SFMC S3 last mile |
| P6 · Handoff + Hypercare (W11 + Jul-Aug) | 18 | ~3.5 | ~$490 | Runbook + handoff notes done · hypercare cadence TBD |
Trace: P3b + P5 absorbed the bulk of W10 (Rich's 12-issue QA closeout + DC live baseline capture + 2 Rich sessions + handoff notes). All fixable gaps closed; only source-integrity residuals (WJ replication + Academy pre-2018) and SFMC last-mile are outstanding.
W11 (6 Jul – 12 Jul) · SFMC last mile + handoff kickoff
- PR #4537 merge confirmation · post-sunset PROD parity dashboard · verify ET orphan_rate 10.4% → 0% · verify email_unsub post-#4525
- SFMC admin account setup · once Rich provisions (~1 day work) → bidirectional S3 share (Snowflake CH → S3 → SFMC + SFMC engagement events → S3 → Snowflake CH)
- Andrew Tiemann meeting · finalise unsub strategy recommendation · draft formal proposal for marketing leads + product presidents
- GH #340 (WJ replication) · start data-eng conversation for
WORLD_JOURNEYS.OPPORTUNITIES_VCH replication (same pattern as ET SALES_V share) - Handoff kickoff session with Mike Grabbe + Adrian LeDoux · walk through
docs/dc-sunset/handoff-notes.md· Splink Hybrid B retrain triggers · dbt tests baseline - Confluence architecture pages · finalise dbt #1 US normalize + dbt #2 CH + Hybrid B sunset-safe IR pattern · link from handoff notes
- Invoice for June work (shipped Monday per Rich session #2)
- Allan · merge #4537 (last unmerged W10 PR · CI green)
- Rich · SFMC admin account provisioning · confirm Jen consult on WJ/Academy pre-2018 orphans · previous invoice status
- EF · NTE · Amendment #3 conversation — scope decision for P5 SFMC last mile + P6 hypercare (one cycle vs two)
- Andrew Tiemann · 30-min meeting to finalise unsub strategy (central SFMC vs per-product) · required before writing marketing-lead proposal
- Mike + Adrian · handoff kickoff session scheduling · walkthrough of
docs/dc-sunset/handoff-notes.md+ Splink retrain triggers - Data engineering · start convo on
WORLD_JOURNEYS.OPPORTUNITIES_Vreplication into CH (GH #340) · closes final WJ orphan gap
Where to dig deeper
docs/dc-sunset/dc_live_baseline_2026-06-30.md— immutable pre-sunset DC baseline · 24 metric snapshots · regression reference for post-sunset driftdocs/dc-sunset/handoff-notes.md— 9-section handoff · Splink Hybrid B tiering + retrain triggers · J.6 source integrity evidence · data freshness advantage · Splink drift metrics · email_unsub PR #4525 rationale + per-BU source field map · GH #339 Hybrid COALESCE plan · contact-point marts sibling refactoref_data_hub_ch/tests/test_sales_order_orphan_opp_rate.sql· per-BU orphan monitoring · Academy excluded (persistent 97.7% baseline) · thresholds WJ<50% · ET<15% · others<20%ef_data_hub_ch/tests/test_splink_ir_tier1_hit_rate.sql· Splink IR drift monitoring · zero-div guard- Memory:
2026-07-01-rich-session.md·2026-07-02-rich-session.md - Sprint log W10 D1-D5 with 36.0h tracked + per-day deliverables
eftours/de-dbt#4493— 4 new DC snapshots + GBR/UC split + GL mut-excl + WJ Option B + Tour→Program rename · closes RLT-4816 (5→12 issues) (MERGED Tue)eftours/de-dbt#4512— GL mut-excl at projection level · overlap 14,420 → 0 (MERGED Wed)eftours/de-dbt#4525— Email_Unsub source swap DC snapshot → US canonical · closes RLT-4887 (MERGED Fri)eftours/de-dbt#4534— 2 dbt tests +docs/dc-sunset/handoff-notes.md· pre-cutover cleanup (MERGED Fri)eftours/de-dbt#4537— ET orphan opp recovery via SALES_V UNION · 100% of 35,740 orphans · closes RLT-4889 J.6 ET half (OPEN · CI green · awaiting Allan)JonatanFG/EF#336— Rich QA findings triage (12 issues · full evidence log) · CLOSED with PR #4493 + #4537JonatanFG/EF#337— [REMEDIATION] Email_Unsubscribe SFMC-source refactor (post-cutover) · blocked by #167JonatanFG/EF#338— [AUDIT] W10 pre-cutover QA runner + Ygritte results (RLT-4889 tracker)JonatanFG/EF#339— [REMEDIATION] Hybrid COALESCE for unified_contactable post-sunset scalabilityJonatanFG/EF#340— [REMEDIATION] ReplicateWORLD_JOURNEYS.OPPORTUNITIES_Vto CH for WJ orphan recovery (In Progress · blocker: data eng)- RLT closed to Done: RLT-3168 · RLT-3169 · RLT-3575 · RLT-3667 · RLT-3971 · RLT-3988 · RLT-4816 (Rich closed) · RLT-4926 (Rich closed)