Inter-rater agreement on 'requires new mathematical machinery' judgments is κ = 0.31 among 22 research mathematicians shown 40 solved problems

open — nothing has been offered for or against it · what this means

no sources · falsifiability unrated

0 ◎ prieto_h (15) · 2 months ago · research-methods, mathematics

Prieto & Halvorsen (2029) presented 22 number theorists and combinatorialists with 40 problems solved between 1995 and 2015 and asked whether each required machinery unavailable at the time of posing. Agreement was poor (Fleiss κ = 0.31), and ratings correlated 0.44 with the rater's own subfield.

rate this claim
  • Accurate 0
  • Falsifiable 0
  • Clear 0
  • Novel 0
  • Important 0

evidence

cited by (1)
[+0] The 2028 survey's six 'requires new construction' problems were rated by referees who knew the eventual human solution in four cases discuss
this claim
supports (0)
no supporting evidence yet
opposes (0)
no opposing evidence yet
whole evidence graph shown depth 1 2 3 both pro con full
what the marks mean
  • supports · opposes · cites this claim · the claim you are reading
  • [+9] net votes on the evidence link — whether this is good evidence here, not whether the claim itself is true
  • Af··· accurate, falsifiable, clear, novel, important — a capital is established, lowercase weak, unrated; no strip at all where nothing has been rated yet
  • [?] no score has been computed for that claim yet
  • field shown only where a claim's fields differ from the one above it, so a field marks where the argument crosses into another

respondeo

Loading responses… open them

discussion

Log in to join the discussion.