Technical Methodology

1. Data

Sampling frame

Current: Respondents are recruited via email invitation through panels provided by Alchemer LLC panel services. Alchemer sources panel members from a managed opt-in network; invitations are sent by email to panelists matching the target demographics for each wave. Planned: We intend to shift to a multi-channel approach drawing from an opt-in panel of prior respondents who agreed to re-contact, Facebook Lead Gen ads targeted to under-represented populations, SMS / P2P outreach to registered voters via Politexts (i-360-sourced lists), and cold-web recruitment. Source-of-respondent will be tracked as a structured column on l1_respondents once that transition lands; until then, source tagging feeds partner-facing disclosures as operational practice.

Recruitment

Current: Email-to-panel via Alchemer LLC. Planned: Multi-channel recruitment run in cost-priority order — panel re-engagement (opt-in re-contact), Facebook Lead Gen ads, i-360 SMS via Politexts (P2P), cold web/Facebook. Operational targets in both approaches — not currently algorithmically gated at ingest — are: ≥30% of each wave on a tracked topic drawn from respondents who have not previously answered that topic, and rotation of respondents out of a topic after 3 waves with a one-wave cooldown. These will be bridged into data_integrity as enforced ingest-time gates when the multi-channel transition occurs.

Sample size

Target n ≈ 1,000 completed responses per state per wave, with R+NP oversample where Senate-battery cross-tabs require it (e.g., LA-CCS-2026-002 / -003 oversample base ~683 R+NP).

Demographic block

21 questions appended at the end of every survey (per D27 — Demographic Predictor Block specification). Census two-question approach for race/ethnicity (race + Hispanic separate; bridge synthesizes back to engine's single race_ethnicity field). Religious attendance collected on a 5-point scale (More than once a week / Once a week / A few times a month / A few times a year / Seldom or never), collapsed to the Pew 3-way church_freq (Weekly+ / Monthly-yearly / Seldom-never) on bridge publish per D23. Four MrP cell predictors (race_ethnicity, age_bracket, education, gender — see map_l1_demographics_for_mrp in unified_pipeline.py) plus collected-but-not-used-as-cell-predictor fields: party_id (used in raw partisan reach, not as a fitting predictor — see §3), ideology, area_type (urban/suburban/rural), income, homeownership, evangelical/born-again self-id, ZIP, and a 7-question behavioral-fingerprint panel (TV news, newspaper, Facebook, other social, talk radio, religious service past month, political activism past 12 months).

Data integrity

Every published estimate is gated by an invariant runner (data_integrity.py) that runs at end of every ingest and on every test. Standing checks include null-cell coverage, value-range bounds, FK integrity, partner-renderer provenance, retired-vocabulary scans, and the cross-survey pooling registry (SURVEY_REGISTRY). Silent-data bugs become a new @check + regression test BEFORE any fix ships.

2. Model specification

For each question (binarized into a binary-favorable outcome per response coding), we fit a Bayesian multilevel logistic regression of the form (pooling is within-wave only — each survey wave is a separate fit):

The four random-intercept terms cover race, age bracket, education, and sex. The four pairwise interactions follow Ghitza & Gelman 2013 § 4 ("Deep Interactions with MRP") and have been default-on in the production fit since D16 (April 30, 2026). When the fit spans multiple states, the design matrix Xstate includes two state-level fixed-effect predictors:

• 2024 presidential vote share (MIT Election Lab, county-of-residence-aggregated)
• CES party self-ID share (current cohort, weighted by ACS marginals)

These two predictors are 0.977-correlated across the production cohort (D17 R3 calibration blvali73t) and jointly contribute approximately one independent state-trending signal. Both are kept rather than collapsed; the redundancy is by design (Phase B / Path P1-strict). State-level predictors are inert in single-state production fits.

Inference: NUTS sampler via PyMC (through bambi's formula DSL); production config is 4 chains × 1,500 post-warmup draws × 1,000 tune (warmup), target accept rate 0.95 (see BAYESIAN_DRAWS / BAYESIAN_TUNE / BAYESIAN_CHAINS / BAYESIAN_TARGET_ACCEPT in mrp_engine.py). CI mode (MRP_CI_MODE=1) runs at 2 chains × 300 draws × 300 tune for regression detection only — never for production publication.

3. Poststratification

Demographic marginals are drawn from the U.S. Census American Community Survey (ACS) 5-year tabulations at the state level — race × ethnicity, age bracket, educational attainment, sex. Cells are constructed as the cross-product of the four marginals; the full theoretical grid is 7 races × 7 age brackets × 5 edu × 2 sex = 490 cells per state, of which only cells with non-zero ACS population are retained (the engine emits the active cell count per fit as n_cells in mrp_question_summary). The retained cells sum to 1.0 within each state.

Party raking — calibration pass in flight. The engine's IPF raking infrastructure (mrp_engine.run(..., marginals=...)) accepts external party marginals; CES party-share targets are loadable via ces_ingest.py. The production sweep currently fits without party raking (phase8_sweep_stage.py:244 passes marginals=None) pending the calibration pass that validates the gate impact — Brier, ECE, support-drift, and 95% CI coverage measured with raking on vs. off against the held-out cohort. Calibration is scheduled for the next cycle post the current demo; production cut-over follows on gate-pass and decision filing. The current state is documented here rather than asserted as a methodology preference: we will report MrP-poststratified-to-ACS for the demo, and add "+raked to CES [year]" once the validation evidence is in hand.

Joint multi-state fits — capability validated; production publication remains per-state per-wave. The engine supports joint multi-state fits in which the multilevel structure pools partial information across states via random-intercept structure on state-level predictors (state_vote_share_2024, state_party_self_id_rep_share). The D18 (May 1, 2026) joint A+B+C calibration on the CJ-2026-Q1 multi-state cohort (VA + NC + MA + NJ, 3,706 pooled respondents) cleared every release gate with measurably tighter metrics than Phase A alone — Brier p90 0.246 vs. 0.250, ECE p90 0.065 vs. 0.124, support-drift p90 0.029 vs. 0.071 — confirming the multi-state pool's precision claim empirically. Pooling is within-wave only in current production: per-wave fits are run once per survey via batch MrP calls (run_mrp_and_save(sid, state, domain)); state-level predictors are constants in single-state fits and bambi absorbs them into the intercept (wired but inert until the multi-state cut-over).

What's coming. Four additional state surveys (Oregon, Texas, Colorado, Michigan) are scheduled for the next field cycle. They expand the joint-pool cohort from the current 6 surveyed states to 10 — a more demographically representative base for the multilevel structure. Cut-over to joint multi-state production fits is gated on per-cohort calibration (running the D14 release-gate suite on the specific multi-state cohort before publishing pooled estimates as production).

Pre-registered out-of-sample validation. Before the Texas wave fields, we will produce a synthetic TX estimate from the existing pool using only Texas's ACS marginals and state-level predictors as input — no Texas survey data. The synthetic estimate, with 95% credible interval, is pre-registered. After the TX wave returns, the actual MrP estimate is compared to the pre-registered synthetic: does the synthetic value sit within the actual estimate's CI? This is a real out-of-sample test of the cross-state pool's projection claim. Pass / fail and the magnitude of any miss will be published as a numbered methodology decision. TX is the first-cycle target because its demographic profile (Hispanic + Southern) has the strongest precedent in the current 6-state pool, making it the cleanest test of the methodology in best-case conditions; CO and MI follow as harder tests with the same protocol. This protocol provides the external ground-truth check that our construct library lacks against published vendor surveys.

Single-wave constraint also stands: each survey wave is a separate fit, with temporal pooling on the temporal MrP pilot path (see PR #33, merged 2026-05-12).

DK-inclusive denominator. Per D2 (April 22, 2026), "Don't know" responses are kept in the base. The partner-facing topline is mrp_pct_of_all = mrp_pct_opinionated × scorable_coverage, where scorable_coverage is the fraction of respondents who registered an opinion. This conservatively suppresses support rates relative to approaches that exclude non-responses.

4. Convergence diagnostics

Per D13 (April 30, 2026, Bayesian gate framework), every fit is gated on:

A single failed gate flips mrp_status to failed with a diagnostic reason (no_convergence, insufficient_n, binarization_failed, no_covariates, or engine_error). Diagnostic columns (rhat_max, ess_min, divergence_fraction) are persisted on every mrp_question_summary row for audit. Per D26, partner-facing renderers must filter by mrp_status = 'fitted'; raw-or-failed rows can never be served labeled as MrP-adjusted (data-integrity invariant _partner_renderers_filter_by_mrp_status).

Per D14, the R-hat 1.01 tighten was reverted (D14.1) when an empirical study failed to clear the production cohort; the production gate remains 1.05 until empirical study clears 1.01.

5. Calibration metrics

Per cycle, MrP estimates pass internal held-out validation against a 7-wave cohort of constructs with known historical patterns. Four metrics are gated:

The support-drift gate was repinned from 0.06 to 0.08 per D16 after the Phase A calibration (cohort b1si9wv5n) showed empirical p90 of 0.071 with the demographic-interactions model — within tolerance but above the original 0.06 ceiling. This is a deliberately empirical threshold (Professor rule #3 — no statistical theater).

Per-fit quality grading (D26 — mrp_quality field)

Each mrp_question_summary.mrp_quality row is graded good, marginal, or degenerate via a tier ladder:

1. Convergence gates fire first. Divergence fraction > 1% → degenerate. R-hat > 1.05 → degenerate. Bulk ESS < 400 → demoted to marginal (best case).
2. CI half-width tier follows. ≤ 3 percentage points (and converged) → good. ≤ 7 pp → marginal. > 7 pp → degenerate.

Convergence-first discipline closes the silent-trust failure mode where a deceptively narrow CI on an unidentified model would otherwise score "good." The frequentist (FE) fallback path grades as degenerate until the bootstrap-CI persistence work lands.

6. Release gates

No estimate is published unless every gate clears (D13 framework). The release-gate pipeline:

1. Per-question convergence gates (R-hat, ESS, divergences) must clear, OR mrp_status flips to failed with reason.
2. Cell-level posterior predictive check (PPC) per BDA3 ch. 6.
3. Held-out cohort calibration (Brier / ECE / coverage / support drift p90 thresholds) must all pass.
4. Cell-churn gate: ≤ 30% of cells move > 5 percentage points across consecutive waves on the same survey (catches model-instability false-positives).
5. Calibration staleness invariant: production calibration must be < 30 days old.
6. Forbidden-phrase invariant on partner renderers (no "guaranteed", "proves", "definitely", causal-overclaim language) per D11 + D8.

A BayesianSamplingError is raised on any gate failure; the question is excluded from publication and logged to mrp_question_flags for operator follow-up. Production-path config + slow-test CI workflow added in D14 (post-merge methodology hardening).

7. VIP scoring

The Voter Intelligence Pack (VIP) score is a composite of 4 sub-metrics (Reach, Base Health, Cross-Partisan Strength, Durability Profile), each normalized 0–100 and equally weighted. Scores are computed per reform topic (construct), not per raw survey question. The Base Hardening (BH) and Coalition Broadening (CB) axes form the Durability Quadrant — each with its own posterior 95% credible interval propagated from the underlying MrP cell estimates via bootstrap (D6 v0; v1 with drift trails is on the roadmap).

Durability thresholds (asymmetric, D29)

The internal position field takes values durable / brittle / insufficient; partner-facing surfaces render these via D28 label translation as "Consolidated coalition" / "Developing coalition" / "Reach not estimable". The durable classification requires BH ≥ 50 AND CB ≥ 55 (each on a 0–100 scale), per D29 (May 13 2026). Thresholds are asymmetric: BH and CB measure structurally different properties, and the original symmetric 55/55 cutoff (D6) was a generic durability benchmark that produced uniform brittle classification across the partner-facing CJ reform construct library (observed BH cluster 48–54; CB cluster 58–72). BH calibrated to 50 against the observed distribution in LA wave 2; CB unchanged. D29 is provisional pending 9-state cross-state validation — the revisit trigger is documented in the decision. Sensitivity has been tested across BH ∈ [40, 65] and CB ∈ [50, 65]; will be re-validated when ballot-outcome ground truth becomes available (Q4 2026 cycle).

Reach surfaces

vip_scores_v2.reach_pct is raw bivariate (sample-weighted favorable share), not MrP-adjusted; r_reach, d_reach, demo_reach are similarly raw. The MrP-adjusted topline lives in mrp_question_summary.mrp_pct_of_all with mrp_status = 'fitted' filter (D26). Per D8, every partner-facing renderer making an "MrP-adjusted" or "poststratified" claim is invariant-checked against its source query (data_integrity._partner_memo_provenance_claims_match_source).

8. Data pipeline

7-stage automated pipeline executed via unified_pipeline.run_full_pipeline():

1. INGEST — survey CSV / Politexts API / survey-platform bridge → raw L2 responses
2. NORMALIZE — vendor-specific value normalization → engine canonical (L1_RACE_MAP, L1_AGE_MAP, etc.)
3. RESOLVE — cross-survey QID alias resolution → canonical question IDs
4. LOAD — write L1 + L2 to Supabase; bridge propagates sampling_frame, demo_source per-row
5. SCORE — bivariate + raw distribution writes to vip_scores_v2; universal regression sweep (BH-FDR-corrected, behavioral & religious-attendance dimensions included)
6. MrP — run_mrp_and_save fits per question, writes per-cell mrp_estimates and per-question mrp_question_summary with diagnostics; flips mrp_status per D26
7. QC — release gates fire (§6), failed estimates excluded; data_integrity invariant runner sweeps the full database

Survey-platform-published surveys flow through the same pipeline. The bridge (publish_to_vip.py) writes to the shared Supabase tables (l1_respondents, l2_responses, mrp_question_summary) without touching VIP engine code; the engine's next sweep picks up the new survey and runs MrP.

9. External validation

External ground-truth comparison against independent surveys is not available for our construct library; these questions were designed specifically because they aren't asked elsewhere. Four validation surfaces compensate — three in place or complete, two in flight:

1. Leave-one-state-out cross-validation — complete (2026-06-03). We ran a geographic hold-one-state-out validation across all 28 CJ questions confirmed present in all 6 surveyed states (LA, MA, NC, NJ, OK, VA) — 168 fold/QID combinations. For each state, the MrP pipeline was trained on the remaining five states and used to predict the held-out state's direct MrP estimate. Results against pre-registered gates:

   Metric	Result	Gate	Pass?
   LOO-CI empirical coverage	87.5% (147/168)	≥ 80%	✓
   MAE vs direct MrP	2.38 pp	≤ 5 pp	✓
   QIDs exceeding 8 pp band	0 of 28	0	✓

   Per-state coverage: NC 100% (28/28); VA 96%, MA 93%, NJ 93%; LA 86%; OK 57%. Excluding Oklahoma: 93.6% coverage, 2.24 pp MAE.
   
   Honest limitations. Oklahoma is the model's consistent weak spot (57% coverage across all 28 QIDs). Oklahoma's political and demographic composition is the most distinct of the six states; the model trained on the other five has insufficient basis for Oklahoma's specific covariance structure. Partner-facing Oklahoma estimates carry wider uncertainty. Two questions were flagged with policy-context limitations: CJ-BAIL3 (attitudes toward fixed vs. discretionary bail are shaped by each state's bail reform history in ways demographic composition alone does not predict — Louisiana the most skeptical of judicial discretion, Oklahoma the most supportive in opposite directions from the training pool) and CJ-JUV1 (systematic over-prediction from a lower-support cluster in NJ and VA; the cross-state model anchors on the higher-support training states by ~5 pp). For both questions, direct MrP per state is the reliable estimate; cross-state inference should be presented with wider uncertainty. The validation run also identified and fixed two methodology bugs mid-cycle — a leaky holdout (K-AI-19) and a binarizer substring error (K-AI-20) — confirming that the LOO harness functions as an error-detection layer, not only a pass/fail gate. Validated results incorporate both fixes.
   
   → Full LOO validation results — per-question and per-state breakdown, flagged items with root-cause analysis, and corrections identified during the run.
2. Demographic structure validation (in place). Reform support should run higher among Democrats than Republicans, Black voters than white voters, younger voters than older voters, and urban residents than rural residents — consistent with adjacent Pew, Gallup, and AP-NORC research on criminal justice attitudes. This direction-of-effect check is part of our internal review; we observe the expected pattern within tolerance in the current cohort.
3. Pre-registered out-of-sample validation (in flight, next field cycle). The Oregon, Texas, Colorado, and Michigan surveys scheduled for the next field cycle enable a real out-of-sample test of the cross-state pool. The first cycle target is Texas: before the TX wave fields, the existing 6-state pool will produce a synthetic TX estimate using only Texas's ACS marginals and state-level predictors as input — no Texas survey data. The synthetic estimate and 95% CI are pre-registered. After the TX wave returns, the actual MrP estimate is compared to the pre-registered synthetic; pass / fail and the magnitude of any miss are published as a numbered methodology decision. TX is the first-cycle target because its demographic profile has the strongest precedent in the current pool (Hispanic + Southern, well-represented by LA + OK + NJ data); CO and MI follow as progressively harder tests of methodology generalization. This is a stronger external check than cross-vendor replication because we control the question wording on both sides of the comparison.
4. Ballot-outcome comparison (protocol established; first cycle Q4 2026). When measures relevant to our construct library appear on state ballots, we compare our pre-election MrP estimates to the certified outcome. The standing protocol is written; the first usable ballot-outcome cycle is Q4 2026 — no certified results have been compared to date.
5. Cross-vendor adjacent-question replication (roadmap). A subset of demographic and behavioral-fingerprint questions appear in CES, ANES, and AP-NORC and could in principle be benchmarked against published vendor estimates as a sanity check. No automated cross-vendor replication is currently implemented; this remains roadmap pending sufficient cohort overlap to make the comparison statistically meaningful. Lower priority than the OR/TX/CO/MI out-of-sample experiment, which provides ground-truth on our own construct library.

10. References

• Park, D., Gelman, A., & Bafumi, J. (2004). Bayesian multilevel estimation with poststratification: state-level estimates from national polls. Political Analysis, 12(4), 375–385.
• Ghitza, Y., & Gelman, A. (2013). Deep interactions with MRP: election turnout and voting patterns among small electoral subgroups. American Journal of Political Science, 57(3), 762–776.
• Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., & Bürkner, P.-C. (2021). Rank-normalization, folding, and localization: an improved R̂ for assessing convergence of MCMC. Bayesian Analysis, 16(2), 667–718.
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis, 3rd ed. (BDA3). Chapman and Hall/CRC.
• Capell, A., Bürkner, P.-C., et al. (2023). bambi: a simple interface for fitting Bayesian linear models in Python. Journal of Open Source Software, 8(82), 5125.
• Salvatier, J., Wiecki, T. V., & Fonnesbeck, C. (2016). Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2, e55.
• U.S. Census Bureau. American Community Survey (ACS) 5-year demographic tabulations.
• MIT Election Lab. 2024 U.S. Presidential Election precinct-level returns.
• Cooperative Election Study (CES). Harvard / YouGov annual cross-section, party self-ID marginals.

Methodology questions: contact Preston Robinson, Actionable Intel LLC, via the partner portal contact form. We respond to peer-review and journalism inquiries within five business days.