Matiq
2201a57ffa
- README: status 2026-09-02 chain919, bridge 1.594 canon, structural 2.689 (L/R e343b0a, RT_CASC=0), hist 0.341 unreachable
- prd.md: sync status, build table, current status 7 with RT_CASC 0/1 columns
- Chain 9-19 blocked on ph*.npz/live-dump (BLOCKMAP:285)
297 lines
14 KiB
Markdown
297 lines
14 KiB
Markdown
# prd.md — Project Requirements Document
|
||
|
||
## 1. Project Overview
|
||
|
||
**Name**: `soothe2-re` — bit-exact reverse engineering of **oeksound soothe2** (VST3, Windows x64)
|
||
|
||
**Goal**: Reproduce the plugin's DSP core (level detector, mask computation, filter application) in C++17, byte-for-byte identical to the native binary.
|
||
|
||
**Status**: ~95% decompiled. Bridge **1.594 dB** stable; structural **2.689 dB** (L/R `e343b0a`, `RT_VLAW=1 RT_SYN=1 RT_NOWARP=1 RT_NOIIR3=1 RT_IIR12=0`, `RT_CASC=0` gated). Historical best `0.341` (24mm14) not reachable on current HEAD. Target: **<0.05 dB** (bit-exact gate, requires chain calibration).
|
||
|
||
**Golden rule**: Every parameter must have a source (decomp address / live table). Empirical fits must be flagged `EMPIRICAL`.
|
||
|
||
---
|
||
|
||
## 2. Repository Structure
|
||
|
||
```
|
||
re-tools/
|
||
├── prd.md ← THIS FILE (project overview)
|
||
├── README.md ← TOTAL metric source (single source of truth)
|
||
├── AGENTS.md ← runbook: build, env flags, tooling hazard
|
||
├── BITEXACT_PLAN.md ← path to byte-exact (3 steps, criteria)
|
||
├── roadmap.md ← historical B-phase log (archived)
|
||
│
|
||
├── dsp/ ← C++17 DSP reconstruction (THE CANON)
|
||
│ ├── framed_model.{cpp,hpp} ← MAIN: mask-apply chain (empirical bridge)
|
||
│ ├── render48k.cpp ← 48k/4096 structural pipeline (resample→chain→resample)
|
||
│ ├── spectral.{cpp,hpp} ← STFT/ISTFT processor + FIR builder
|
||
│ ├── fn529fe0.{cpp,hpp} ← structural chain 9–19 (FUN_180529fe0)
|
||
│ ├── fnfaith.{cpp,hpp} ← faithful detector cascade (BLOCKMAP transcription)
|
||
│ ├── fft{,_plan,_stage}.cpp/hpp ← FFT engine (canonical + RFFT bit-exact)
|
||
│ ├── twin.{cpp,hpp} ← twin resonator (FUN_180535880, float-parity)
|
||
│ ├── rt_mask_tables.{hpp,.cpp} ← live IIR A/B coefficients (385K)
|
||
│ ├── rt_weights.{hpp,.cpp} ← live kWarp/kBand768 tables (44K)
|
||
│ ├── fftconv.{cpp,hpp} ← FFT convolution FIR application
|
||
│ ├── vlog.{cpp,hpp} ← fast log2 approximation
|
||
│ ├── exp2.{cpp,_tables.cpp,hpp} ← fast exp2 approximation
|
||
│ ├── levelpath.{cpp,hpp} ← level-curve path (xv = log10(am/res))
|
||
│ ├── freqpath.{cpp,hpp} ← frequency-axis warp/freq-domain helpers
|
||
│ ├── leveltrack.{cpp,hpp,hpp} ← envelope follower (attack/release tables)
|
||
│ ├── log2_ln.{cpp,hpp} ← log/ln utilities
|
||
│ ├── filter.{cpp,hpp} ← detection kernel (legacy)
|
||
│ ├── detect.{cpp,hpp} ← detector front-end (legacy)
|
||
│ ├── phase_table.{cpp,hpp} ← phase table for FFT
|
||
│ ├── ms.hpp ← mid-side helpers
|
||
│ ├── params.hpp ← parameter struct
|
||
│ ├── rotor_kernel.hpp ← rotor transform kernel
|
||
│ ├── rt_div_tables.hpp ← division tables
|
||
│ ├── soothe_constants.hpp ← decoded constants
|
||
│ ├── cody_waite.hpp ← Cody-Waite argument reduction
|
||
│ ├── twiddle_{builder,loader}.cpp/hpp ← FFT twiddle factors
|
||
│ ├── dsp_ctx.hpp ← DSP context layout
|
||
│ ├── framed_test.cpp ← CLI bridge renderer (44.1k)
|
||
│ ├── harness.cpp ← legacy harness
|
||
│ ├── *_check.cpp ← bit-exact unit test targets
|
||
│ └── CMakeLists.txt
|
||
│
|
||
├── scripts/
|
||
│ ├── corpus.py ← bridge regression harness (62 cases, --compare)
|
||
│ ├── corpus_structural.py ← structural chain harness (--vs-bridge)
|
||
│ ├── rendersnap2.py ← RENDER_FILE stopper (reads from .rpp)
|
||
│ ├── campaign.py ← parameter sweep cell (~8 min)
|
||
│ ├── cascade_sim.py ← numpy step 9–19 simulator
|
||
│ ├── disasm_func.py ← capstone disasm with RIP constants
|
||
│ ├── iat_name.py ← runtime import resolution
|
||
│ ├── wine_{chain,stage,ptrace}_trace.py ← ptrace-based live trace
|
||
│ ├── dump_dispatch.py ← table/state dumper
|
||
│ ├── probe_{states,mem}.py ← live state/memory probes
|
||
│ ├── hunt2.py ← ctx-instance hunter
|
||
│ ├── scan_{pairs,lutsub}.py ← memory diagnostics
|
||
│ ├── fit_vlaw_{params,by_group}.py ← VLAW law calibration
|
||
│ ├── lawfit22r.py ← law fitting (VLAW α/β/c)
|
||
│ ├── mk_{dist,far,multi6}.py ← RPP generators
|
||
│ └── render_parity.py ← model-vs-render comparison
|
||
│
|
||
├── handoff/
|
||
│ ├── NOTES_LEVEL.md ← live journal (head: 24mm5+)
|
||
│ ├── NOTES_LEVEL_INDEX.md ← journal index by date/topic
|
||
│ ├── BLOCKMAP_529fe0.md ← FUN_180529fe0 method map (53K)
|
||
│ ├── NOTES_TWIN.md ← twin reference
|
||
│ ├── NOTES_CAPTURE.md ← live-capture protocol
|
||
│ ├── NEXT_PROMPT.md ← entry point for new sessions
|
||
│ ├── SESSION_HANDOFF.md ← handoff template
|
||
│ ├── archive/ ← historical NOTES_LEVEL_*.md, SESSION_HANDOFF_*.md
|
||
│ ├── nls_dasm/ ← 183 disassembly files (.dis, .bin)
|
||
│ ├── phase1/ ← phase-1 outputs
|
||
│ ├── rt*.npy ← captured runtime tables (48k/44.1k)
|
||
│ └── *.py ← emit/extract/joint scripts
|
||
│
|
||
├── *.java ← Ghidra scripts (DumpFuns, ImportRtti…)
|
||
├── *.{bin,npz,npy,json,txt} ← datasets, dumps, LUTs (mostly outside git)
|
||
└── soothe-bt/ ← test corpus (~600 renders, outside git)
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Build System
|
||
|
||
**Generator**: CMake 3.10+, C++17, GCC/Clang with `-O3 -march=native`.
|
||
|
||
```cmake
|
||
# Key targets
|
||
soothe2_dsp # shared library (all dsp/*.cpp)
|
||
framed_test # CLI bridge renderer (44.1k)
|
||
render48k # structural renderer (48k/48000)
|
||
twin_check # float-parity unit test
|
||
tables_check # live-table verification
|
||
fftconv_check # FIR convolution check
|
||
vlog_check # fast log2 check
|
||
leveltrack_check # envelope follower check
|
||
levelpath_check # level-curve path check
|
||
exp2_check # fast exp2 check
|
||
fn529fe0_check # structural chain check
|
||
soothe2_harness # legacy harness
|
||
```
|
||
|
||
**External deps**: `libsamplerate` (render48k only), `pthread`.
|
||
|
||
**Tooling hazard**: CMake skips rebuild when source modified within same second. Protocol: `touch` source before build + verify binary mtime.
|
||
|
||
---
|
||
|
||
## 4. DSP Architecture
|
||
|
||
### 4.1 Signal Flow (current canon: structural `render48k`)
|
||
|
||
```
|
||
Host 44.1k → resample → 48k → [per-channel processing] → resample → 44.1k Host
|
||
↓
|
||
┌─────────────────────┐
|
||
│ FramedDetector │
|
||
│ (per 4096 block) │
|
||
│ │
|
||
│ am[] ← envelope │
|
||
│ res[] ← twin resp │
|
||
│ ↓ │
|
||
│ lvl_raw = am/res·sf │
|
||
│ ↓ │
|
||
│ [RT_VLAW=1]: │
|
||
│ cut = α·ln1p(L/β) │
|
||
│ +c [+Δ] │
|
||
│ mask = 10^(-cut/20)│
|
||
│ ↓ │
|
||
│ warp: mask *= │
|
||
│ kBand·kWarp·res^rp│
|
||
│ ↓ │
|
||
│ IIR3 ×2 (bidir) │
|
||
│ ↓ │
|
||
│ mask_out → multiply │
|
||
│ spectrum[k] *= mask │
|
||
└─────────────────────┘
|
||
```
|
||
|
||
### 4.2 Dual Render Path
|
||
|
||
| Path | File | Grid | Use |
|
||
|------|------|------|-----|
|
||
| Bridge | `framed_model.cpp` | 44.1k/2048 | Legacy, TOTAL 1.594 (canon) |
|
||
| Structural | `render48k.cpp` | 48k/4096 L/R | `2.689` (`RT_CASC=0` gated, hist. `0.341`) |
|
||
|
||
**Dual-solution env set**: `RT_VLAW=1 RT_SYN=1 RT_NOWARP=1 RT_NOIIR3=1 RT_IIR12=0` (chain `RT_CASC=1` adds IIR4 double + FIR min-phase, gated)
|
||
|
||
### 4.3 Key Modules
|
||
|
||
| Module | Responsibility |
|
||
|--------|---------------|
|
||
| `FramedDetector` | Per-band, per-frame mask computation. Holds `am_`, `res_`, `track_` state. |
|
||
| `SpectralProcessor` | STFT/ISTFT, OLA overlap-add, FIR application modes. |
|
||
| `fn529fe0` | Structural chain 9–19 (steps 9–19 of FUN_180529fe0): scale→IIR1→IIR2→blend→combine→warp→IIR3→dry/wet. |
|
||
| `fnfaith` | Faithful detector cascade transcription (BLOCKMAP). |
|
||
| `twin` | Twin resonator `|2B/A|` — frequency response per band. |
|
||
| `rt_mask_tables` | Live IIR A/B coefficients (kIIR_A1/A2/A3, kIIR_B1/B2/B3, kRTAtt, kRTRel). |
|
||
| `rt_weights` | Live warp weights (kBand768, kWarp). |
|
||
| `fft` | Bit-exact RFFT (th1a90/th2180) with `buf548`/`mask598` tables. |
|
||
| `fftconv` | FFT-based convolution for FIR application modes. |
|
||
| `vlog`/`exp2` | Fast polynomial approximations matching plugin精度. |
|
||
| `leveltrack` | Envelope follower with per-bin attack/release tables. |
|
||
|
||
### 4.4 Detector Cascade (529c60)
|
||
|
||
```
|
||
complex_spectrum × twin_response → |z|
|
||
→ Haar smooth [0.25,0.5,0.25] × n_iters
|
||
→ peak = max(curve)
|
||
→ sin_peak = sin(param·30 − 90) · 0.115129 · peak
|
||
→ curve = max(curve, sin_peak)
|
||
→ w = -log10(pow(50, ratio·0.001) · ratio·0.001)
|
||
→ acc = acc·w + curve·(1-w)
|
||
→ bands_curve = acc
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Environment Flags (experiment control)
|
||
|
||
| Flag | Effect |
|
||
|------|--------|
|
||
| `RT_VLAW=1` | Two-stage law: `mask = 10^(−(α·ln1p(lvl/β)+c)/20)` |
|
||
| `RT_SYN=1` | STFT without synthesis window (plugin's actual layer) |
|
||
| `RT_WIN=0/1/2` | Analysis window: sym-Hann / periodic / rect |
|
||
| `RT_NOWARP=1` | Skip warp modulation |
|
||
| `RT_NOIIR3=1` | Skip IIR3 ×2 |
|
||
| `RT_IIR12=0` | Skip freq-domain IIR1/2 (critical with VLAW) |
|
||
| `RT_DUMP_BIN=<f>` | Dump tract binary (frame `RT_DUMP_FRAME`) |
|
||
| `RT_VDBG=1` | Print VLAW computations to stderr |
|
||
| `RT_FAITHFUL=1` | Use faithful chain (`fnfaith.cpp`) |
|
||
| `RT_FIRCONV=1/3` | FIR application mode (1=complex-mul, 3=`1.019·mask^1.8345`) |
|
||
| `RT_ENV=live` | Live envelope from kRTAtt/kRTRel tables |
|
||
| `RT_KMAP=1` | k-mapping correction (twin/am scaling) |
|
||
| `RT_EQ=1` | Pre-detector EQ bell |
|
||
| `RT_LUT_OFF=1` | Skip LUT transform |
|
||
| `RT_LUT_A/B/G/M` | LUT parameters (A=-24, B=28, gamma=1, mult=4.2) |
|
||
|
||
---
|
||
|
||
## 6. Test Corpus & Metrics
|
||
|
||
**Location**: `/home/m/soothe-bt/` (~600 renders, outside git).
|
||
|
||
**Key sets**:
|
||
|
||
| Prefix | Content |
|
||
|--------|---------|
|
||
| `tone1kq_*` | Single tone, fc-scan, quiet (-18 dBFS) |
|
||
| `tone1k_*` | Single tone, fc-scan, loud (0 dBFS) |
|
||
| `dual_b1q_*` | Two tones (500+2000), q 0.1…10 |
|
||
| `al_*` | Level sweep (fc=1000) |
|
||
| `comb_*` | 4-band multiband |
|
||
| `burst500_b1` | Primary reference (burst 500 Hz) |
|
||
|
||
**Metric** (Goertzel steady-state):
|
||
```python
|
||
err_dB = 20·log1₀( ta(out,1000) / ta(ref,1000) )
|
||
# trimmed to last 75% of input, matched to plugin render length
|
||
```
|
||
|
||
**Regression guards**:
|
||
```bash
|
||
python3 scripts/corpus.py # bridge + guard
|
||
python3 scripts/corpus_structural.py # structural chain
|
||
python3 scripts/corpus.py --compare scripts/baseline_bridge.json --tol 0.25
|
||
python3 scripts/corpus_structural.py --vs-bridge scripts/baseline_bridge.json
|
||
```
|
||
|
||
---
|
||
|
||
## 7. Current Status (2026-09-02, chain919)
|
||
|
||
| Group | Bridge | Structural `RT_CASC=0` (L/R) | Structural `RT_CASC=1` |
|
||
|-------|--------|-------------------------------|------------------------|
|
||
| t1kq (fc-scan) | 0.226 | 0.356 | 0.717 |
|
||
| t1k (loud) | 1.801 | 5.437 | 5.631 |
|
||
| al (level) | 0.638 | 2.613 | 5.036 |
|
||
| res | 0.628 | 1.256 | 23.031 |
|
||
| dual (q-sweep) | 0.726 | 2.042 | 66.642 |
|
||
| comb (4-band) | 10.149 | 6.004 | 29.172 |
|
||
| **TOTAL** | **1.594** | **2.689** | **31.653** |
|
||
|
||
Historical best structural `0.341` (24mm14, `f40f41e` dual 0.193) not reachable on current HEAD (M/S→L/R + chain).
|
||
|
||
**Decoded**:
|
||
- Layer: STFT without synthesis window, per-bin mask multiply
|
||
- Law: `mask = 10^(−(α·ln1p(lvl/β)+c)/20)` with content-aware Δ
|
||
- FIR: `exp(0.984·ln(raw))` + Hann + normalize → bit-exact RFFT to df0
|
||
- Chain 9–19: IIR4 double + FIR min-phase integrated, `generate_iir4_coefs()` (FUN_180533340) added, gated `RT_CASC`
|
||
|
||
**Open gaps**:
|
||
1. **Chain 9–19** (priority #1): Integrated but uncalibrated (input format level vs cut, blocker `ph*.npz`/live-dump). Target: <0.05.
|
||
2. **k-mapping** (priority #2): twin/am scaling `k(q≥2)=0.403`.
|
||
3. **Δ second-peak rule** (priority #3): Content-dependent gain, requires live-dump.
|
||
|
||
---
|
||
|
||
## 8. Reference Documents
|
||
|
||
| Doc | Content |
|
||
|-----|---------|
|
||
| `README.md` | TOTAL metric, reproduction steps |
|
||
| `AGENTS.md` | Build commands, env flags, hazard, corpus format |
|
||
| `BITEXACT_PLAN.md` | 3-step plan to byte-exact |
|
||
| `handoff/BLOCKMAP_529fe0.md` | Method map for FUN_180529fe0 |
|
||
| `handoff/NOTES_LEVEL.md` | Live working journal |
|
||
| `handoff/NOTES_LEVEL_INDEX.md` | Journal index |
|
||
| `handoff/NOTES_TWIN.md` | Twin resonator reference |
|
||
| `handoff/NOTES_CAPTURE.md` | Live-capture protocol |
|
||
| `handoff/nls_dasm/` | 183 disassembly files |
|
||
|
||
---
|
||
|
||
## 9. Reproduction & Development Protocol
|
||
|
||
1. Install soothe2 VST3 (Windows) under yabridge → `dump_soothe.py` → `soothe_mem.bin`
|
||
2. Ghidra headless: `analyzeHeadless <proj> soothe_x64 -process soothe_mem.bin -noanalysis -postScript <X>.java`
|
||
3. Renders: `sweep.py` → `.rpp`, `reaper -renderproject` → `.wav`
|
||
4. Analysis: `render_parity.py` / `corpus.py`
|
||
5. After C++ edit: `touch` source → `cmake --build dsp/build --target framed_test` → run corpus guards
|