检查惯常做法是否正确？贝叶斯方法使用MCMC进行AB测试如何在Python中计算贝叶斯因子？

如何解决检查惯常做法是否正确？贝叶斯方法使用MCMC进行AB测试如何在Python中计算贝叶斯因子？

我一直在努力解决玩具数据AB测试问题中的惯常和贝叶斯方法。

结果对我来说真的没有意义。我正在努力了解结果，或者我是否正确地计算了它们（可能）。此外，经过大量研究，我仍然对如何计算贝叶斯因子一无所知。我已经看过R中的软件包，这使它看起来有些简单。 las，我对R不熟悉，希望能够用Python解决此问题。

非常感谢您对此提供的帮助和指导！

以下是数据：

# imports
import pingouin as pg
import pymc3 as pm
import pandas as pd
import numpy as np
import scipy.stats as scs
import statsmodels.stats.api as sms
import math
import matplotlib.pyplot as plt

# A = control -- B = treatment
a_success = 10730
a_failure = 61988
a_total = a_success + a_failure
a_cr = a_success / a_total

b_success = 10966
b_failure = 60738
b_total = b_success + b_failure
b_cr = b_success / b_total

我首先进行了一些功效分析，以确定功效为0.8，α为0.05，实际意义为2％的所需样本数。我不确定是否应该提供预期的转化率，还是基线+一定比例。根据效果的大小，所需的样本数量会大大增加。

# determine required sample size 
baseline_rate = a_cr
practical_significance = 0.02
alpha = 0.05
power = 0.8 
nobs1 = None

# is this how to calculate effect size?
effect_size = sms.proportion_effectsize(baseline_rate,baseline_rate + practical_significance) # 5204

# # or this?
# effect_size = sms.proportion_effectsize(baseline_rate,baseline_rate + baseline_rate * practical_significance) # 228583

sample_size = sms.NormalIndPower().solve_power(effect_size = effect_size,power = power,alpha = alpha,nobs1 = nobs1,ratio = 1)

我继续尝试确定是否可以拒绝原假设：

# calculate pooled probability
pooled_probability = (a_success + b_success) / (a_total + b_total)

# calculate pooled standard error and margin of error
se_pooled = math.sqrt(pooled_probability * (1 - pooled_probability) * (1 / b_total + 1 / a_total))
z_score = scs.norm.ppf(1 - alpha / 2)
margin_of_error = se_pooled * z_score

# the estimated difference between probability of conversions of both groups
d_hat = (test_b_success / test_b_total) - (test_a_success / test_a_total)

# test if null hypothesis can be rejected
lower_bound = d_hat - margin_of_error
upper_bound = d_hat + margin_of_error

if practical_significance < lower_bound:
    print("reject null hypothesis -- groups do not have the same conversion rates")
else: 
    print("do not reject the null hypothesis -- groups have the same conversion rates")

尽管B组（治疗）相对于A组（对照组）的转化率显示了3.65％的相对改善，但评估结果为“不拒绝零...”。

我尝试了一种稍微不同的方法（我猜是一个稍微不同的假设吗？）：

successes = [a_success,b_success]
nobs = [a_total,b_total]

z_stat,p_value = sms.proportions_ztest(successes,nobs=nobs)
(lower_a,lower_b),(upper_a,upper_b) = sms.proportion_confint(successes,nobs=nobs,alpha=alpha)

if p_value < alpha:
    print("reject null hypothesis -- groups do not have the same conversion rates")
else: 
    print("do not reject the null hypothesis -- groups have the same conversion rates")

哪个评估得出“拒绝原假设...”，p值：0.004236。这似乎非常矛盾，特别是因为p值

关于贝叶斯...由于事情需要多长时间，我创建了一些成功和失败的数组（并且仅对100个观察进行了测试），并运行以下命令：

# generate lists of 1,0
obs_a = np.repeat([1,0],[a_success,a_failure]) 
obs_v = np.repeat([1,[b_success,b_failure])

for _ in range(10):
    np.random.shuffle(observations_A)
    np.random.shuffle(observations_B)

with pm.Model() as model:
    p_A = pm.Beta("p_A",1,1)
    p_B = pm.Beta("p_B",1)
    
    delta = pm.Deterministic("delta",p_A - p_B)

    obs_A = pm.Bernoulli("obs_A",p_A,observed = obs_a[:1000])
    obs_B = pm.Bernoulli("obs_B",p_B,observed = obs_b[:1000])
    
    step = pm.NUTS()
    trace = pm.sample(1000,step = step,chains = 2)

首先，我了解您应该烧掉一定比例的痕迹-如何确定要烧录的索引的适当数量？

在尝试评估后验概率时，以下代码是执行此操作的正确方法吗？

b_lift = (trace['p_B'].mean() - trace['p_A'].mean()) / trace['p_A'].mean() * 100
b_prob = np.mean(trace["delta"] > 0)

a_lift = (trace['p_A'].mean() - trace['p_B'].mean()) / trace['p_B'].mean() * 100
a_prob = np.mean(trace["delta"] < 0)

# is the Bayes Factor just the ratio of the posterior probabilities for these two models?
BF = (trace['p_B'] / trace['p_A']).mean()

print(f'There is {b_prob} probability B outperforms A by a magnitude of {round(b_lift,2)}%') 
print(f'There is {a_prob} probability A outperforms B by a magnitude of {round(a_lift,2)}%') 
print('BF:',BF)
-- output:
There is 0.666 probability B outperforms A by a magnitude of 1.29%
There is 0.334 probability A outperforms B by a magnitude of -1.28%
BF: 1.013357654428127

我怀疑这不是计算贝叶斯因子的正确方法。贝叶斯因子如何计算？

我真的希望您能帮助我理解以上所有内容……我意识到这是一篇篇特别长的文章。但是我已经尝试了所有可以找到的资源，但仍然被卡住了！

亲切的问候。