damir iskakov

does your score beat its own baseline?

a single-cell potency score is usually validated by correlating it against a known hierarchy. a score that only restates sequencing depth passes that test. this runs the test that separates them.

damir@macbook ~ % iskakov
╭────────────────────────────────────────────────────────╮
│ ✳ own-baseline  does your score beat its own baseline? │
╰────────────────────────────────────────────────────────╯

  █████ █   █ █   █             ████     █████   
  █   █ █   █ ██  █             █   █      █     
  █   █ █ █ █ █ █ █             █   █      █     
  █   █ ██ ██ █  ██             █   █      █     
  █████ █   █ █   █             ████  ██ █████ ██
  ████   ███  █████ █████ █     █████ █   █ █████
  █   █ █   █ █     █     █       █   ██  █ █    
  ████  █████ █████ ████  █       █   █ █ █ ████ 
  █   █ █   █     █ █     █       █   █  ██ █    
  ████  █   █ █████ █████ █████ █████ █   █ █████
                                    Damir Iskakov

  v0.2.2  ·  MIT

  check        does this score order cells beyond the statistic it is closest to?
  floors       the estimator's measured null floor at a given n, rho and kernel
  primitives   compute the low-order statistics from an expression matrix
  verify       re-run a receipt and report whether it still reproduces

  ownbaseline check cells.h5ad --score obs:cytotrace --ordinal obs:stage \
                    --ordinal-source experimental

  ownbaseline <verb> --help   ·   docs/interpreting.md for what a verdict does not mean

the splash, verbatim. four verbs, and the measured null grid ships inside the package.

get it running

python 3.9 or newer, three dependencies. the same package installs asown-baseline under its descriptive name.

two commands, and a third if you want to see the floor it compared against.

1

install

pip install iskakov
2

run it on your matrix

the ordinal has to be score-independent, and the flag makes you say so.

ownbaseline check cells.h5ad \
    --score   obs:cytotrace \
    --ordinal obs:stage \
    --ordinal-source experimental \
    --json report.json
3

ask what the floor was

measured, not assumed, and answered with no network access.

ownbaseline floors --n 39505 --rho 0.4844

what it actually does

most published potency scores are built on a low-order statistic of the same expression matrix: the number of genes detected, the correlation of a cell’s profile with network node degree, the shannon entropy of the profile, the library size.

the field validates such a score by correlating it against a pseudotime, a marker panel or a known hierarchy. depth correlates with the gold standard too, so the criterion cannot separate a score that carries ordering information from one that restates depth. rank correlation against the primitive does not settle it either: a score at 0.93 with node degree can carry substantial skill beyond it, and another at 0.998 carries none.

own-baseline residualizes the score on the statistic its own authors say it approximates, and asks whether what is left still orders cells. four answers are possible.

adds-beyond the residual still orders the ordinal, above the measured floor for this n, rho and kernel.

tautolog what is left after residualization does not clear the floor. the score was restating its primitive.

sign-flipped the residual orders the ordinal backwards. a margin read without checking direction would have called this a pass.

inconclusive-kernel-disagree the two kernels answer differently. the tool refuses to pick one, and so should you.

what it prints

real runs, captured from the command line. the fixture is synthetic and seeded, so these numbers are reproducible rather than illustrative.

a score built as 0.93 times its primitive plus noise. it tracks the ordinal at auc 0.800 and still fails: the residual is +0.0187 against a floor of +0.0479.

~ % ownbaseline check --score score.npy --ordinal ordinal.npy --primitives prim.npz --ordinal-source experimental
╭─ own-baseline 0.2.2  ·  Damir Iskakov ──────────────────────────────────────────────────────╮
│   score    score.npy                                                                        │
│   ordinal  ordinal.npy  5 levels                                                            │
│   n        4,000   seed 42   kendalltau (variant=b)                                         │
│                                                                                             │
│ ▌ DIRECTION CHECK                                                                           │
│     score      vs ordinal   AUC 0.800   points with the ordinal                             │
│     gene_count               AUC 0.834   points with the ordinal                            │
│     -> score and every primitive point the same way; margins read as written.               │
│                                                                                             │
│ ▌ vs gene_count    rho(score, primitive) = +0.921                                           │
│     conditional skill  +0.0187   CI95 [+0.0030, +0.0350]  (1000 resamples)                  │
│     own floor          +0.0479                                                              │
│     null cell          Null B, align on, 12 levels, n=3,000, rho=0.93,                      │
│                        kendalltau, 1 covariate, 3 coupling strengths, 600 seeds             │
│     NOTE: the nearest measured n is 3,000, below this run's 4,000.                          │
│     The floor falls with n, so this threshold is higher than the true one and               │
│     clearing it is the safe direction to be wrong in.                                       │
│     -> DOES NOT CLEAR (+0.0187 against +0.0479)                                             │
│                                                                                             │
│   VERDICT  does not order cells beyond its primitive on this ordinal.                       │
│   Scope: this says the score orders cells beyond these statistics. It                       │
│   does not say what the residual is. A residual can be developmental                        │
│   position, manifold structure, or a fifth statistic nobody named.                          │
│                                                                                             │
│ ▌ NOT COMPUTED, and therefore not controlled for                                            │
│     PCC(x,degree): no expression matrix and no vector given; pass --pcc-degree <file>       │
│     Shannon_H: no expression matrix and no vector given; pass --entropy <file>              │
│     log10_library_size: no expression matrix and no vector given; pass --log-libsize <file> │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯

wrote report.json

what came out of it

calibrated on 39,505 cells across twelve microscopy-staged timepoints, where the ordering is fixed before the cells are processed.

thirteen published scores audited. of the seven testable, five survive residualization and two do not.

direction has to be checked before any margin is read. in sorted haematopoietic progenitors the gene-count primitive orders cells below chance: hscs carry a median 916 detected genes, the gmps below them carry 1,404, and the primitive scores auroc 0.378 against the known hierarchy. a score that beats that baseline has supplied a sign correction, and nothing in the margin shows it.

what a verdict does not mean

ADDS-BEYOND says the score orders cells beyond the primitive you named. it says nothing about whether what it adds is potency rather than position on a developmental manifold, and nothing about a primitive you did not name.

source, mit ↗pypi ↗the method and the audit ↗