Server Quality Checklist

Tool Scores

• list_my_submissionsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions authentication ('authenticated user') and the return format ('JSON string'), but lacks details on rate limits, pagination behavior (beyond offset/limit parameters), error handling, or what 'latest' means (e.g., sorted by time). For a read operation with zero annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with two sentences that are front-loaded and efficient. The first sentence states the purpose, and the second clarifies the return format. There's no wasted text, though it could be slightly more structured (e.g., bullet points).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no annotations, but with an output schema), the description is minimally adequate. The output schema exists, so the description needn't detail return values, but it lacks context on authentication requirements, pagination, or error cases. It meets basic needs but has clear gaps for a read operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the schema provides no parameter descriptions. The tool description doesn't mention any parameters, leaving both 'offset' and 'limit' undocumented. However, with only 2 parameters and default values provided in the schema, the baseline is 3 as the description doesn't add value but the schema has minimal complexity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('fetch', 'returns') and identifies the resource ('list of latest submissions made by the authenticated user'). It distinguishes from siblings like 'get_submission_status' (which focuses on individual submissions) and 'submit_code' (which creates submissions), though it doesn't explicitly name these alternatives. The purpose is specific but could be slightly more differentiated from 'list_problems'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., authentication), exclusions, or compare it to sibling tools like 'list_problems' or 'get_submission_status'. Usage is implied by the purpose but lacks explicit context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_problemsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the return format ('JSON string containing total count and results'), which adds some value, but fails to describe critical behaviors like pagination (implied by offset/limit but not explained), rate limits, authentication needs, or error handling. For a tool with parameters and no annotations, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences that are front-loaded: the first states the purpose, and the second describes the return format. There is no wasted text, and it efficiently conveys core information. However, it could be slightly more structured by explicitly mentioning parameters or usage context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which should document return values), the description doesn't need to explain return details. However, with no annotations, 2 parameters at 0% schema coverage, and sibling tools, the description is incomplete. It covers the basic purpose and return format but misses parameter semantics, usage guidelines, and behavioral traits like pagination or auth requirements, making it adequate but with clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the schema provides no parameter descriptions. The tool description does not mention the parameters (offset and limit) at all, offering no semantic meaning beyond what the schema's titles ('Offset', 'Limit') imply. However, since there are only 2 parameters and the description hints at pagination through the return format, it partially compensates but doesn't fully explain parameter usage. Baseline is 3 due to low parameter count.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Fetch') and resource ('list of problems from the Online Judge'), making the purpose evident. It distinguishes from siblings like 'get_problem_details' (specific problem) and 'list_my_submissions' (user-specific), though it doesn't explicitly mention these distinctions. The purpose is specific but lacks explicit sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, context for use, or exclusions. For example, it doesn't clarify if this is for browsing all problems versus filtered lists, or how it relates to 'get_problem_details' for detailed views. This leaves the agent without usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_problem_detailsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states 'fetch' which implies a read operation, but doesn't mention whether this requires authentication, has rate limits, returns structured data, or handles errors. For a tool with zero annotation coverage, this leaves significant behavioral gaps unaddressed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences that directly address the tool's purpose and parameter meaning. There's no wasted text or redundancy. However, it could be slightly more front-loaded by integrating the parameter explanation more seamlessly rather than as a separate sentence fragment.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which handles return values), no annotations, and only one parameter with good semantic coverage in the description, the description is minimally adequate. However, for a tool that fetches detailed information, additional context about what constitutes 'detailed information' (e.g., problem statement, constraints, sample cases) would be helpful despite the output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides specific semantic context for the single parameter 'problem_id', explaining it's 'the external ID/display ID of the problem' and giving an example format ('PR-114-1-31'). Since schema description coverage is 0% (the schema only has a title 'Problem Id'), this adds meaningful value beyond what the structured schema provides. With only one parameter, the description adequately covers its purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'fetch' and the resource 'detailed information of a specific problem', which is specific and actionable. However, it doesn't explicitly differentiate from sibling tools like 'list_problems' (which likely lists multiple problems) or 'get_submission_status' (which focuses on submissions rather than problem details).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to choose 'get_problem_details' over 'list_problems' for problem information, or how it relates to submission-related tools like 'get_submission_status' or 'list_my_submissions'. There's no context about prerequisites or typical workflows.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_codeB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the action ('Submit code') but lacks critical details: it doesn't mention authentication requirements, rate limits, whether submissions are queued or processed immediately, or what happens on success/failure. For a mutation tool with zero annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise and front-loaded, with the first sentence stating the core purpose and each subsequent sentence clearly explaining a parameter. There is no wasted text, and the structure efficiently conveys essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there is an output schema (which should cover return values), the description doesn't need to explain outputs. However, for a mutation tool with no annotations and three parameters, the description lacks context on behavioral aspects like authentication, side effects, or error handling. It's minimally adequate but has clear gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful semantics for all three parameters beyond the schema, which has 0% description coverage. It explains that 'problem_id' is an 'internal system ID', 'language' includes examples like 'C', 'C++', 'Python3', and 'code' is 'the source code to submit'. This compensates well for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Submit code') and the target ('Online Judge for a specific problem'), which is specific and actionable. However, it doesn't explicitly differentiate from sibling tools like 'get_submission_status' or 'list_my_submissions', which are related but distinct operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing a problem ID from 'get_problem_details' or 'list_problems'), nor does it clarify its role in the submission workflow relative to siblings like checking status or listing submissions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_submission_statusB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves status and result, implying a read-only operation, but doesn't disclose any behavioral traits such as authentication needs, rate limits, error handling, or what the 'result' entails (e.g., output, errors). This is a significant gap for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with two sentences that directly state the tool's purpose and parameter meaning without any waste. Every sentence earns its place by adding value, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (single parameter, no annotations, but with an output schema), the description is minimally adequate. It covers the basic purpose and parameter semantics, but lacks behavioral context. The presence of an output schema means the description doesn't need to explain return values, but it should still address usage and transparency gaps to be more complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning beyond the input schema by explaining that 'submission_id' is 'The ID returned when submitting the code,' which clarifies its origin and purpose. With 0% schema description coverage and only one parameter, this compensates well, providing essential context that the schema alone lacks. However, it doesn't detail format or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with 'Get the status and result of a previously submitted code,' which specifies the verb (get) and resource (submission status/result). It distinguishes from siblings like 'submit_code' (creation) and 'list_my_submissions' (listing multiple), but doesn't explicitly differentiate from 'get_problem_details' or 'list_problems' in terms of scope. This makes it clear but not fully sibling-differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by stating 'previously submitted code,' suggesting it should be used after submission, but it doesn't provide explicit guidance on when to use this tool versus alternatives like 'list_my_submissions' for multiple submissions or 'get_problem_details' for problem info. No exclusions or clear alternatives are named, leaving usage somewhat inferred.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.