"Question rubrics used to evaluate creative quality Jasmine Sun interviewed writing evaluators who were paid to grade AI-generated text for major labs and found the rubrics were often nonsensical — one contractor was told to 'grade them based on the number of exclamation marks' and to score fan fiction on 'factuality.' Her conclusion: 'Some of these like very big companies that are very well resourced simply do not know how to think about what good writing is.' Anyone building or evaluating AI tools for any subjective domain should interrogate whether the scoring system actually captures the quality that matters, or just what is easy to measure. Source: Jasmine Sun, Hard Fork, AI Layoffs and the Human Skill That Eludes AI"