Every paper first submitted to arXiv in the 6 months to 2026-09-21 whose categories include cs.AI, cs.CL or cs.HC was harvested via OAI-PMH (metadata only — no PDFs). Each paper's title + abstract was sent as state to TypeSafe's Jev (jev-latest) System One model with the questions below, verbatim. Code — not the model — did all counting, date filtering, thresholding and ranking.
primary_contribution{
"type": "choice",
"instructions": "What is this paper's primary contribution? Judge by what the paper mainly delivers, not by what it mentions. A paper about how to talk to an existing model at inference time (wording, prompting, conversation strategy) is NOT a paper about building or training a model.",
"criteria": {
"training_or_pretraining": "Builds or improves how models are trained or pretrained: data mixes, objectives, scaling, RL training pipelines.",
"model_architecture": "Proposes or modifies the neural network architecture itself: attention variants, layers, mixture-of-experts, tokenizers.",
"fine_tuning_or_alignment_method": "A method that updates model weights after pretraining: fine-tuning, LoRA, RLHF/DPO, distillation, safety alignment training.",
"infrastructure_or_efficiency": "Serving, inference speed, quantization, caching, hardware, cost or memory efficiency of running models.",
"benchmark_or_dataset": "The main deliverable is a new benchmark, dataset, or evaluation suite for others to test models on.",
"prompting_or_interaction_technique": "A way to get better results from an EXISTING model at inference time by changing what you type: prompt wording or structure, in-context examples, step-by-step elicitation, conversation strategy. No weight updates, no code required in principle.",
"evaluation_of_llm_behavior_or_limits": "Measures or characterizes how existing LLMs behave, fail, or can be misled: hallucination, bias, sycophancy, prompt sensitivity, robustness, overconfidence.",
"human_ai_interaction_study": "Studies people using AI: user studies, how humans prompt, trust, reliance, collaboration outcomes.",
"application_in_a_domain": "Applies existing models to a domain task (medicine, law, education, finance...) and reports how well it works.",
"survey_or_position": "A survey, review, taxonomy, or position/opinion paper rather than new experiments or methods.",
"other": "None of the above fits the main contribution."
}
}needs_code_or_model_access{
"type": "noul",
"instructions": "Applying this paper's main idea requires writing code, fine-tuning, or access to model weights or logits \u2014 it cannot be done by typing into a chat interface."
}about_llm_behavior{
"type": "noul",
"instructions": "The paper reports how LLMs behave, fail, or can be misled (hallucination, sycophancy, bias, prompt sensitivity, position effects, overconfidence, context-length effects, etc.)."
}chat_user_usefulness{
"type": "score",
"instructions": "How useful is this paper to a NON-technical person who uses an LLM (Claude, ChatGPT, Gemini) through a chat window for research, writing, analysis, finding information, or decision-making? Judge ONLY what a person with no coding skills could apply by changing how they write prompts, structure conversations, verify outputs, or choose which tasks to trust the model with. Ignore value that requires code, APIs, fine-tuning, or building systems.",
"criteria": [
"Useless to a chat user: nothing here changes what a chat user types, checks, or expects.",
"Marginal: at most background awareness; no concrete change to practice.",
"Somewhat useful: suggests a practice or caveat a chat user could apply, but indirectly or weakly.",
"Directly actionable in a chat window: a chat user could apply a specific finding in their next conversation with no tools.",
"Changes how a chat user should work: an important technique or caveat that should alter their routine use of LLMs."
]
}topic{
"type": "choice",
"instructions": "Which topic best captures what a chat user would take from this paper?",
"criteria": {
"prompt_structure_and_wording": "How phrasing, format, or structure of a prompt changes results.",
"reasoning_and_step_by_step": "Eliciting or trusting step-by-step reasoning, chain-of-thought, thinking aloud.",
"role_and_persona": "Assigning roles, personas, or audience framing to the model.",
"examples_and_few_shot": "Providing examples in the prompt to steer output.",
"context_and_long_documents": "Working with long inputs, document placement, context limits.",
"verification_and_self_checking": "Getting the model to check, critique, or revise its own output.",
"hallucination_and_factuality": "When and why models fabricate; how factuality varies.",
"bias_and_sycophancy": "Models agreeing with users, flattery, framing effects, social biases.",
"multi_turn_conversation_strategy": "How to manage a conversation over multiple turns.",
"tool_and_search_use_from_chat": "Using web search, retrieval, or built-in tools from a chat interface.",
"writing_and_editing": "Using LLMs for drafting, editing, style, summarizing one's own text.",
"research_and_summarization": "Using LLMs to find, digest, or summarize information and literature.",
"decision_support_and_analysis": "Using LLMs to weigh options, analyze data or arguments, advise decisions.",
"safety_and_misuse_caveats": "Risks, manipulation, privacy, over-reliance, when not to trust the model.",
"other": "None of the above fits."
}
}audience_fit{
"type": "choice",
"instructions": "Which audience would benefit most from what this paper offers a chat user?",
"criteria": {
"general_knowledge_worker": "Anyone using LLMs for everyday office work.",
"researcher_or_analyst": "People doing research, literature review, or data/argument analysis.",
"writer_or_marketer": "People producing written content or marketing material.",
"manager_or_executive": "People making business decisions, delegating, evaluating AI use.",
"educator_or_student": "Teaching, learning, studying with LLMs.",
"developer_only": "Really only useful to software developers despite any framing."
}
}evidence_strength{
"type": "score",
"instructions": "How strong is the evidence behind this paper's main claims, as presented in the abstract?",
"criteria": [
"Opinion / anecdote: position, essay, or claims without systematic experiments.",
"Small or single-model study: systematic but limited \u2014 one model, small sample, narrow task.",
"Multi-model or large controlled study: several models or a large, controlled experimental setup."
]
}generalizes_beyond_one_model{
"type": "noul",
"instructions": "The findings are claimed or shown to hold across several LLM providers, not just one model."
}