Output Format and Evaluation Criteria

2. Learning Outcomes

By the end of this lesson, you should be able to:

1. Select an output format appropriate to the human purpose and intended use.
2. Specify required sections, organization, length, headings, lists, tables, and constraints for an output.
3. Convert vague quality instructions (e.g., "excellent," "professional," "accurate") into observable evaluation criteria.
4. Develop criteria addressing relevance, clarity, completeness, source alignment, audience suitability, accessibility, uncertainty, and reviewability.
5. Assess outputs consistently using common criteria across prompt versions.
6. Distinguish professional appearance from substantive quality and identify when polish conceals weakness.
7. Construct a prompt that includes a clear format specification and a set of observable evaluation criteria.

3. Before You Begin

• Assigned reading: Chapter 4: Prompt Design Fundamentals (sections on Format and Evaluation Criteria).
• Required prior knowledge: Lesson 4.1 (Purpose and Task Definition); Lesson 4.2 (Context, Information Boundaries, and Audience); understanding of the distinction between purpose, task, output, use, and action.
• Optional AI tool: You may use an institutionally approved generative AI system for guided activities. Supplied outputs are provided if you do not have access.
• Privacy reminder: Do not enter personal, confidential, or proprietary information into any AI system.
• Academic integrity: All work you submit must be your own. Document any AI use transparently.
• Estimated preparation time: 15–20 minutes to review Chapter 4 and Lessons 4.1 and 4.2.

4. Retrieval Practice: Concepts from Lessons 4.1 and 4.2

Before we add format and criteria, let's refresh the key ideas from the previous lessons.

5. Opening Scenario: "The Polished but Empty Essay"

6. What Is Output Format?

In the PCAFE-V framework, Format is the required organization, length, structure, style, and presentation of the generated output. Format answers: What should the output look like?

A well-chosen format makes the output usable for the intended audience and purpose, and reviewable for the human evaluator.

Possible formats include:

• Outline — hierarchical structure of topics.
• Checklist — list of items to verify or complete.
• Table — rows and columns for comparison or organization.
• Comparison — side-by-side evaluation of options.
• Explanatory note — concise explanation of a concept.
• Memorandum — formal internal communication.
• Briefing — concise summary for decision-makers.
• Question set — list of questions for reflection or discussion.
• Structured record — data in a consistent format.
• Risk inventory — list of risks and mitigations.
• Step-by-step guide — procedural instructions.
• Plain-language explanation — accessible translation.
• Presentation outline — slides or speaking points.

Why format matters

• Usability: The right format makes it easy for the audience to find and use the information.
• Reviewability: A structured format makes it easier to check completeness and quality.
• Consistency: Specifying format helps produce consistent outputs across different uses.
• Accountability: Format can include placeholders for sources, uncertainty, or verification notes.

Components of a format specification

• Required sections: What headings or parts must be included?
• Order: In what sequence should sections appear?
• Length: Approximate word count, page count, or time duration.
• Headings: Should headings be used? What level?
• Lists: Bullet points or numbered lists?
• Tables: Columns and rows, if applicable.
• Labels: Consistent naming or numbering.
• Placeholders: Fields for data, sources, or uncertainty notes.
• Exclusions: What content is not allowed (e.g., promotional language, speculation).
• Citation fields: Where to include sources.
• Uncertainty fields: Where to note missing information or caveats.

7. What Are Evaluation Criteria?

In the PCAFE-V framework, Evaluation Criteria are observable qualities used to assess whether the output is fit for its intended purpose.

Evaluation criteria are observable: they can be checked, measured, or assessed by a human reviewer. They are not vague aspirations like "excellent" or "professional."

Common evaluation criteria include:

• Relevance: Does the output directly address the purpose and task?
• Accuracy: Are the claims factually correct and verifiable?
• Completeness: Does the output include all necessary information?
• Clarity: Is the language clear and understandable?
• Concision: Is the output appropriately brief?
• Source alignment: Does the output reflect the supplied sources?
• Audience suitability: Is the content appropriate for the intended audience?
• Accessibility: Is the output usable by people with diverse needs?
• Preservation of uncertainty: Are gaps and caveats clearly noted?
• Absence of invented details: Are there no fabricated citations or facts?
• Internal consistency: Does the output agree with itself?
• Reviewability: Is it easy for a human to check and evaluate?
• Actionability: Does it enable the intended user to take appropriate action?

Why vague quality language is insufficient

Instructions like "Make it excellent," "Make it professional," "Make it comprehensive," "Make it accurate," "Make it unbiased," or "Make it accessible" are not criteria. They are value judgments that are open to interpretation. What does "professional" mean? It depends on context, audience, and purpose. Without observable specifics, you cannot evaluate the output consistently, and the AI cannot reliably produce it.

The solution is to convert vague instructions into observable expectations. For example:

• "Make it professional" → "Use a formal tone, avoid contractions, include a subject line, and use standard business letter formatting."
• "Make it comprehensive" → "Include at least three supporting arguments, two counterarguments, and a summary of key evidence."
• "Make it accurate" → "All factual claims must be verifiable from the supplied sources; include page numbers or URLs for each claim."

Using common criteria to compare prompt versions

When you revise a prompt, you should use the same evaluation criteria to compare the initial and revised outputs. This allows you to see whether the revision improved relevance, clarity, completeness, etc., or whether it introduced new problems.

A common set of criteria also helps you identify regressions: areas where the revised output is worse than the initial output. For example, adding more criteria might improve accuracy but reduce clarity.

Polish vs. substantive quality

AI systems are often very good at producing polished, professional-looking text. They can generate well-structured sentences, use appropriate vocabulary, and apply formatting consistently. However, polish does not equal substance.

A polished output can:

• Appear authoritative while containing factual errors;
• Be well-organized but miss key points;
• Use professional language but fail to address the audience's needs;
• Look complete but omit important evidence;
• Be beautifully formatted but be difficult to verify.

Evaluation criteria should focus on substantive qualities (accuracy, completeness, relevance, evidence) rather than merely surface features (grammar, formatting, tone). A checklist that includes both substance and polish can help you balance them.

8. PCAFE-V Walkthrough: Format (F)

Guiding questions for format

• What output form best supports the human purpose and audience?
• What sections or parts are required?
• What is the appropriate length (e.g., words, pages, time)?
• What ordering or structure should be used?
• Should headings, lists, or tables be used?
• What labels or placeholders are needed?
• What content should be excluded or prohibited?
• How should sources and uncertainty be documented?

Weak vs. improved format

Format: One-page briefing note with the following sections:
1. Background (2-3 sentences)
2. Key Changes (bullet list, max 5 items)
3. Impacts on Residents (brief descriptions)
4. Next Steps and Contact Information

Length: Under 500 words.
Use headings and bullet points.
Do not use jargon; define any necessary terms.
Include a placeholder for source references if applicable.

Common mistakes

• Not specifying format at all: Leaving the AI to choose a format that may not suit the purpose.
• Being too vague: "Write a summary" without specifying length or structure.
• Assuming the AI knows what "professional" means: Format should be concrete, not aspirational.
• Ignoring reviewability: Format should make it easy for a human to check completeness and accuracy.

Responsible-use consideration

A well-specified format helps ensure the output is usable and verifiable. Including placeholders for sources and uncertainty makes it easier to identify what needs human review.

Brief practice: format selection

Match the purpose to the most appropriate format:

• Purpose A: "To compare three software options for purchase."
• Purpose B: "To provide step-by-step instructions for setting up a new account."
• Purpose C: "To summarize key findings from a research paper for a team meeting."

Possible formats: table, checklist, step-by-step guide, outline, explanatory note.

9. PCAFE-V Walkthrough: Evaluation Criteria (E)

Guiding questions for evaluation criteria

• What observable qualities define a good output?
• How will I assess relevance, completeness, clarity, and accuracy?
• What specific, measurable indicators can I use?
• How will I know if the output is suitable for the audience?
• What criteria will I use to compare initial and revised outputs?
• How will I distinguish appearance from substance?

Converting vague quality language into observable criteria

Common criteria for comparing prompt versions

When you revise a prompt, use the same set of criteria to evaluate both the initial and revised outputs. This allows you to identify:

• Improvements: Areas where the revised output is better (e.g., more complete, clearer).
• Regressions: Areas where the revised output is worse (e.g., less concise, less readable).
• Unchanged limitations: Issues that persisted despite the revision.
• New issues: Problems introduced by the revision.

This comparison helps you refine the prompt iteratively, moving toward a better output while avoiding unintended trade-offs.

Distinguishing appearance from substance

A polished output can be misleading. To avoid being swayed by appearance, use criteria that specifically address substantive quality:

• Evidence check: Are claims supported by supplied sources?
• Completeness check: Does the output include all required sections and information?
• Accuracy check: Are there any factual errors or invented details?
• Relevance check: Does the output directly address the purpose?
• Uncertainty check: Are missing information and caveats clearly noted?

Professional formatting (e.g., clean headings, consistent fonts, proper grammar) is valuable, but it should not be the only or primary criterion.

Common mistakes

• Using vague quality language: "Make it good" instead of specific criteria.
• Focusing only on appearance: Judging a report by its formatting rather than its content.
• Not using criteria consistently: Evaluating initial and revised outputs with different criteria.
• Ignoring audience needs: Criteria that do not reflect what the audience actually needs.

Responsible-use consideration

Clear evaluation criteria help you avoid over-reliance on AI by making the quality assessment explicit. They also help you identify when the AI output is not fit for purpose, ensuring you do not use flawed output in high-stakes decisions.

Brief practice: convert vague criteria

Convert the following vague instruction into observable criteria:

"Make the presentation interesting."

10. Worked Example: From Vague Output Request to Specific Format and Criteria

Let's extend the worked example from Lesson 4.2 (the sustainability briefing) by adding format and evaluation criteria. Recall that the purpose is to inform an executive committee about a composting program.

Purpose: Prepare a draft one-page briefing note on waste-reduction initiatives in higher education, with a focus on composting programs, to inform an executive committee decision about funding a new program.

Context: Use publicly available information. Exclude internal data.

Audience: Executive committee members who need clear, evidence-based information.

Step 1: Select an appropriate format

The audience is an executive committee. They need a concise, structured briefing that highlights key information quickly. A one-page briefing note with clear sections and bullet points is appropriate. A summary table of benefits and considerations might also be useful.

Step 2: Specify the format

Step 3: Define evaluation criteria

Step 4: Write the revised prompt with format and criteria

Purpose: Prepare a draft one-page briefing note on waste-reduction initiatives in higher education, with a focus on composting programs, to inform an executive committee decision about funding a new program.

Context: Use publicly available information only. Do not reference internal data. Focus on peer-reviewed research, government reports, and case studies from similar institutions.

Audience: Executive committee members who need clear, evidence-based information to make a funding decision. They value concision, clarity, and actionable insights.

Format: One-page briefing note with the following sections:
1. Background (2-3 sentences on the issue of waste in higher education)
2. Key Benefits of Composting Programs (bullet list, max 5 items, with approximate data where available)
3. Implementation Considerations (bullet list of practical factors, e.g., space, resources, partnerships)
4. Conclusion and Recommendation (brief summary, with a note that the committee will decide)

Length: 400-500 words. Use headings and bullet points.
Include placeholders for source references (e.g., [Source: URL]) after each key claim.
Do not include financial projections, internal data, or definitive recommendations.

Evaluation Criteria:
- Relevance: Focus on composting in higher education and institutional benefits.
- Clarity: Use plain language; define any technical terms (e.g., composting, waste diversion rate).
- Completeness: Cover all four sections, with at least three benefits and three considerations.
- Source support: Every factual claim must be traceable to a publicly available source; mark unverified claims clearly.
- Concision: Under 500 words; avoid repetition.
- Uncertainty: Indicate when data is approximate or sources conflict.
- Audience suitability: Professional, respectful, and decision-oriented tone.
- Actionability: Provide enough information for the committee to ask informed questions.

Step 5: Identify remaining questions

• What specific sources are most authoritative for this topic?
• What is the committee's preferred level of detail for implementation considerations?
• How will the briefing be presented (printed, digital, with slides)?

11. Guided Learning Activities

12. Exercises

13. Homework Problems

Formative · Ungraded · Recommended for portfolio preparation and Assignment 1 evidence.

14. Mini‑Project: Format and Evaluation Specification

15. Self‑Check Quiz

Test your understanding of the key concepts from this lesson.

16. Assignment 2 Portfolio Connection

17. Assignment 1 Journal Connection

18. Key Takeaways

• Format makes outputs usable and reviewable. Specify sections, length, structure, and exclusions to ensure the output supports the purpose.
• Evaluation criteria must be observable. Vague quality language (e.g., "excellent," "professional") is insufficient; convert it into specific, checkable criteria.
• Polish does not equal substance. Professional appearance can conceal factual errors, omissions, and flawed reasoning.
• Use common criteria to compare prompt versions. The same set of criteria helps you identify improvements, regressions, and remaining limitations.
• Criteria should address relevance, clarity, completeness, source support, audience suitability, accessibility, uncertainty, and reviewability.
• Format and criteria work together. The format you specify and the criteria you define shape the output's quality and usability.
• Human responsibility remains central. The human defines the format and criteria, evaluates the output, and decides whether it is fit for purpose.

19. Key Terms

Format

The required organization, length, structure, style, and presentation of the generated output.

Format specification

A detailed description of the output's required sections, order, length, headings, lists, tables, labels, placeholders, and exclusions.

Evaluation criteria

Observable qualities used to assess whether the output is fit for its intended purpose.

Observable criterion

A criterion that can be checked, measured, or assessed by a human reviewer, as opposed to a vague value judgment.

Vague quality language

Subjective instructions like "make it excellent," "make it professional," or "make it comprehensive" that are not specific enough to guide the AI or enable consistent evaluation.

Substantive quality

The content-related qualities of an output, such as accuracy, completeness, relevance, and evidence, as opposed to surface features like grammar and formatting.

Polish

The surface-level professional appearance of an output, including grammar, formatting, and tone, which may not reflect substantive quality.

Common criteria

A consistent set of evaluation criteria used to compare multiple outputs (e.g., initial vs. revised prompts).

Reviewability

The ease with which a human can check, evaluate, and verify the output's quality and accuracy.

Actionability

The degree to which the output enables the intended user to take appropriate action.

20. Looking Ahead: Lesson 4.4

In Lessons 4.1, 4.2, and 4.3, you have learned to define purpose, context, audience, format, and evaluation criteria. In Lesson 4.4, you will complete the PCAFE-V framework by adding:

• Verification Requirements: How to specify source identification, claim verification, handling of missing information, conflicting evidence, uncertainty, and the no-fabrication rule.
• PCAFE-V Integration: How all six elements work together to produce a responsible, reviewable prompt.
• Prompt Refinement: How to compare initial and revised prompts using common criteria, identify remaining limitations, and document human decisions.

By the end of Unit 4, you will have a complete PCAFE-V prompt and will be ready to move to Unit 5: Advanced Prompting and Context Engineering.

21. Sources & Further Reading

• Primary: COMP 3270 Chapter 4: Prompt Design Fundamentals (sections on Format and Evaluation Criteria). [Placeholder — definitions and examples from Chapter 4 to be inserted.]
• Supporting: COMP 3270 Appendix B: Prompt Design Toolkit.
• Supporting: COMP 3270 Appendix D: Responsible‑Use Toolkit.

For additional reading on evaluation criteria, consider frameworks for quality assessment in your field (e.g., evidence-based practice, decision analysis). For accessibility, refer to the Web Content Accessibility Guidelines (WCAG) and your institution's accessibility policies.