Metadata-Version: 2.4
Name: abpy-toolkit
Version: 0.1.0
Summary: A lightweight A/B testing framework: fixed-horizon tests, CUPED variance reduction, ratio metrics, power analysis, and multiple-comparison corrections.
Author: KKB
License: MIT
Project-URL: Homepage, https://github.com/Kou2004/Abpy
Project-URL: Repository, https://github.com/Kou2004/Abpy
Project-URL: Issues, https://github.com/Kou2004/Abpy/issues
Keywords: ab-testing,experimentation,statistics,cuped,hypothesis-testing,power-analysis
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: scipy>=1.9
Requires-Dist: matplotlib>=3.6
Dynamic: license-file

# abpy — Frequentist A/B Testing Library

A Python library for rigorous frequentist A/B and multivariate testing, with built-in support for CUPED variance reduction, Delta method for ratio metrics, and multiple comparison corrections.

---

## Installation

```bash
pip install abpy
```

---

## Quick Start

```python
from abpy.router import Experiment

# binary metric — direct input
exp = Experiment(type='ab', metric='binary', input='direct', alpha=0.05)
exp.fit({
    'control':   {'n': 1000, 'successes': 100},
    'treatment': {'n': 1000, 'successes': 130}
})
exp.results()
exp.plot()
```

---

## Features

- **Three metric types** — binary, continuous, count
- **Two input modes** — CSV file or direct summary statistics
- **Automatic test selection** — z-test, t-test, Mann-Whitney U, Poisson rate test
- **CUPED** — variance reduction using pre-experiment data
- **Delta method** — correct variance estimation for ratio metrics
- **Multiple comparison corrections** — Bonferroni, Holm, Benjamini-Hochberg
- **Multivariate testing** — all pairwise comparisons with corrections
- **Power analysis** — achieved power for every test
- **Dashboard visualisation** — 5-panel results plot

---

## API Reference

### `Experiment(type, metric, input, alpha, cuped, correction, is_ratio)`

| Parameter | Type | Default | Description |
|---|---|---|---|
| `type` | str | `'ab'` | `'ab'` or `'multivariate'` |
| `metric` | str | `'continous'` | `'binary'`, `'continous'`, `'discrete'` |
| `input` | str | required | `'csv'` or `'direct'` |
| `alpha` | float | `0.05` | significance level |
| `cuped` | bool | `False` | variance reduction (CSV only) |
| `correction` | str | `None` | `'bonf'`, `'holm'`, `'bh'` |
| `is_ratio` | bool | `False` | ratio metric — applies Delta method (CSV only) |

---

## Usage Examples

### Binary Metric — CSV Input

```python
import pandas as pd
from abpy.router import Experiment

exp = Experiment(type='ab', metric='binary', input='csv', alpha=0.05)
exp.fit('experiment.csv')   # requires columns: variant, metric
exp.results()
exp.plot()
```

CSV structure:
```
variant,metric
control,0
control,1
treatment,1
treatment,0
...
```

### Continuous Metric — Direct Input

```python
exp = Experiment(type='ab', metric='continous', input='direct', alpha=0.05)
exp.fit({
    'control':   {'n': 1000, 'mean': 45.2, 'std': 12.3},
    'treatment': {'n': 1000, 'mean': 48.7, 'std': 11.9}
})
exp.results()
```

### Discrete / Count Metric

```python
exp = Experiment(type='ab', metric='discrete', input='direct', alpha=0.05)
exp.fit({
    'control':   {'n': 1000, 'events': 300},
    'treatment': {'n': 1000, 'events': 375}
})
exp.results()
```

### Multivariate with Holm Correction

```python
exp = Experiment(
    type='multivariate',
    metric='binary',
    input='direct',
    alpha=0.05,
    correction='holm'
)
exp.fit({
    'control':    {'n': 2000, 'successes': 200},
    'treatment1': {'n': 2000, 'successes': 240},
    'treatment2': {'n': 2000, 'successes': 260}
})
exp.results()
```

### CUPED — Variance Reduction

Reduces variance using pre-experiment data, requiring fewer users to reach significance.

```python
exp = Experiment(type='ab', metric='continous', input='csv', alpha=0.05, cuped=True)
exp.fit('experiment_with_pre.csv')
exp.results()
```

CSV structure for CUPED:
```
variant,metric,pre_metric
control,45.2,44.1
treatment,48.7,43.9
...
```

### Ratio Metric — Delta Method

For metrics like CTR = clicks / impressions where both vary per user.

```python
exp = Experiment(type='ab', metric='continous', input='csv', alpha=0.05, is_ratio=True)
exp.fit('ratio_experiment.csv')
exp.results()
```

CSV structure for ratio metrics:
```
variant,metric1,metric2
control,3,10
treatment,5,10
...
```

---

## Test Selection Logic

```
Binary metric    → z-test (pooled proportion)
Continuous
  n >= 30        → t-test (Welch's approximation)
  n < 30         → Shapiro-Wilk normality check
    normal       → t-test
    not normal   → Mann-Whitney U
  is_ratio=True  → Delta method → t-test
Discrete/Count   → Poisson rate test
```

---

## Output

Every test returns:

```
Test used            which statistical test was selected
Test statistic       z, t, or U value
P-value              raw p-value
Adjusted p-value     if correction applied
Confidence interval  95% CI on the lift
Effect size          Cohen's h (binary) or d (continuous)
Power                achieved power given sample size
Recommendation       "2nd variant wins" / "no significant difference" / "underpowered"
```

---

## Corrections Reference

| Method | Controls | Best For |
|---|---|---|
| `bonf` | FWER | Few comparisons, false positives very costly |
| `holm` | FWER | More powerful than Bonferroni, same control |
| `bh` | FDR | Many comparisons, some false positives acceptable |

---

## Visualisation

```python
exp.plot()   # renders 5-panel dashboard
```

Dashboard panels:
1. P-value vs significance threshold
2. Confidence interval on lift
3. Effect size (Cohen's benchmark)
4. Power curve vs sample size
5. Metric distribution by variant

---

## Dependencies

```
numpy >= 1.21.0
scipy >= 1.7.0
pandas >= 1.3.0
matplotlib >= 3.4.0
```

---

## License

MIT License — see LICENSE file for details.
