Sensitiviy SCOP benchmark question

[bioinsilico]

I have a question on how sensitivity is calculated in the SCOPe benchmark. In particular, it is on the total number of positives for the different cases: family, super-family, and fold.

In the evaluation script, I understand that famCnt[id2fam[i]] - 1 removes the self-comparison.

foldseek-analysis/scopbenchmark/scripts/bench.noselfhit.awk

Line 25 in 1737c71

famVal=foundFam[i]/(famCnt[id2fam[i]] - 1);

However, in the super-family and fold cases, subtracting (famCnt[id2fam[i]] - 1) includes the self-comparison in the total count.

foldseek-analysis/scopbenchmark/scripts/bench.noselfhit.awk

Line 26 in 1737c71

sfamVal=foundSFam[i]/(sfamCnt[id2sfam[i]] - (famCnt[id2fam[i]] - 1));

For example, given the SCOPe domain d1eu1a1 b.52.2.2, there are 9 other domains with the same class (including itself)

grep b.52.2.2 scop.tsv 
d1g8ka1	b.52.2.2
d1eu1a1	b.52.2.2
d2iv2x1	b.52.2.2
d1kqfa1	b.52.2.2
d2fug31	b.52.2.2
d2jioa1	b.52.2.2
d1ogya1	b.52.2.2
d1y5ia1	b.52.2.2
d1ti6a1	b.52.2.2

grep b.52.2.2 scop.tsv | wc -l
       9

then, famCnt['b.52.2.2'] - 1 = 8

If we now consider the super-family case, i.e., domains in the same super-family but not the same family we found 7 domains

grep b.52.2 scop.tsv | grep -v b.52.2.2                  
d1ppya_	b.52.2.1
d1e32a1	b.52.2.3
d1cr5a1	b.52.2.3
d1qcsa1	b.52.2.3
d1cz4a1	b.52.2.3
d1wlfa2	b.52.2.3
d3ouga_	b.52.2.0

grep b.52.2 scop.tsv | grep -v b.52.2.2 | wc -l
       7

the total number of domains for the super-family b.52.2 is 16

grep b.52.2 scop.tsv | wc -l 16

and thus,

(sfamCnt['b.52.2'] - (famCnt['b.52.2.2'] - 1)) = 8

I think the discrepancy comes from subtracting (famCnt['b.52.2.2'] - 1)) that should be only famCnt['b.52.2.2'] since self hit is already excluded in sfamCnt['b.52.2'].

This results in none of the methods evaluated having queries with 100% sensitivity for super-family and fold.