How to Check Agreement Between Two Reddit Comment Coders

Setting up your comment sample

Start by collecting public threads with a known collection tool. Reddit Comments public build 1.0.7 accepts postUrls with 1 to 50 known public Reddit post URLs and maxCommentsPerPost from 1 to 500. Replace the sample thread and change the saved cap from 250 to 3 for a pilot.

Review OUTPUT fields including `commentsSaved`, `duplicatesSkipped`, `hiddenBranchesRemaining`, `completionReason`, `status`, `isComplete`, `inputResults`, and per-input `commentsFound`, `commentsSaved`, `status`, and `reason`. Treat processing completion separately from full thread coverage.

Extract dataset fields including `postId`, `commentId`, `body`, `authorName`, `status`, `statusReason`, and `observedAt`. Dataset status values are only `ok`, `deleted`, and `removed`. Keep unavailable, deleted, or removed text, missing IDs, and collection failures outside usable coding units. Retain them in a separate missing-decision ledger. Count distinct known IDs in your working table rather than looking for a unique-comment output field.

The current 1.0.7 public build AskMarketing control returned 3 distinct known comment IDs and nonblank bodies, not the saved LocalLLaMA sample. Processing isComplete true coexists with partial/capped input, proving a pilot sample rather than full thread coverage. Parent, depth, and hiddenBranchesRemaining were null and are not used for this coder method.

Eight steps for independent coding and agreement checks

1. State your research question and one narrow binary rule version, such as explicit_current_failure, along with positive, negative, and ambiguous examples.
2. Choose the same finite public sample for both coders and register known comment IDs and source URLs in a private working register.
3. Remove unusable text from the eligible sample with explicit reasons and log them in a missing-decision ledger.
4. Give each coder the same text and context, the same rule version, and a separate decision sheet to mark independently without seeing each other's work, allowing unknown.
5. Join the two decision sheets by the checked comment unit ID, keeping unknown, absent, and invalid decisions in the missing-decision ledger.
6. Build a two-by-two table from paired known units only, and calculate observed agreement, expected agreement, and Cohen's kappa.
7. Review disagreements in their source context after the independent pass to clarify the rule, keeping adjudication separate from the initial statistic.
8. Save original independent sheets unchanged, version the revised rule, and test it on a new or explicitly declared recoded sample.

Writing the rule and running independent coding

State your research question and one narrow binary rule. In this guide, the code is explicit_current_failure. Mark yes for a stated present product failure, and no for text that does not state one. Provide positive, negative, and ambiguous examples for your coders. Permit unknown when the text or target cannot be resolved.

Choose the same finite public sample for both coders. Retain known comment IDs and source URLs in a private working register. Define one whole comment as one coding unit. Remove unusable text from the eligible sample with explicit reasons. Give each coder the same text and context, the same rule version, and a separate decision sheet. Coders must not see each other's decisions during this pass. Coders can mark unknown; do not force a yes or no choice.

Calculating agreement and kappa

Join the two decision sheets by the checked comment unit ID. Record coder A decision, coder B decision, rule version, and whether both decisions are known. A source missing from one sheet is not a disagreement. Build a two by two table from paired known units only.

Observed agreement, written as p_o, is the two diagonal counts divided by n. Expected agreement, written as p_e, is the sum of products of the corresponding coder marginal proportions. Cohen's kappa is calculated as (p_o minus p_e) divided by (1 minus p_e), when the denominator is nonzero. The expectation comes from these coder marginals, not a known population truth label.

Fictional audit example and decision ledgers

All comments, decisions, and counts below are original fiction. Suppose there are 22 eligible text units. Twenty units have two known independent decisions, one has coder A unknown, and one has coder B absent. Only 20 units enter the agreement table.

Fictional missing-decision ledger

This ledger details the 2 excluded decision gaps out of 22 eligible source units, leaving 20 paired-known units to enter the agreement table. Missing decisions are not disagreements and remain outside the n equals 20 denominator.

Main agreement table (20 paired units)

Coder A margins show 10 yes and 10 no. Coder B margins show 9 yes and 11 no. Seventeen of the 20 paired decisions agree, giving observed agreement p_o equal to 0.85.

Expected agreement p_e is calculated as (10 divided by 20 times 9 divided by 20) plus (10 divided by 20 times 11 divided by 20), which equals 0.50.

Cohen's kappa is (0.85 minus 0.50) divided by (1 minus 0.50), which equals 0.70.

Three paired disagreements and two excluded decision gaps remain separate. This fixture does not label any actual customer or prove the code is correct.

Example comment texts and rule handling

• Agreed-yes comment: "Calendar sync fails every morning." (States a present defect directly).
• Agreed-no comment: "The calendar color is blue." (Does not state a defect).
• Ambiguous wish comment: "I wish calendar sync worked." (Could express a current defect or desired capability; it does not prove the writer asks for a feature. Preserve unknown or ambiguous choice and later context or rule clarification without imposing intent).

Diagnostic Table One (High agreement, low kappa)

Coder margins: Coder A has 19 yes and 1 no. Coder B has 19 yes and 1 no. Total units n equals 20. Observed agreement p_o equals 0.90. Expected agreement p_e equals 0.905. Kappa equals negative 1 divided by 19, which is about -0.05263.

Diagnostic Table Two (Undefined kappa)

Coder margins: Coder A has 20 yes and 0 no. Coder B has 20 yes and 0 no. Total units n equals 20. Observed agreement p_o and expected agreement p_e both equal 1. Making kappa undefined. Do not call an undefined value a perfect kappa or invent a universal pass threshold.

Reviewing disagreements and revising rules

Read disagreements in their source context after the independent pass. For the wish comment, the rule can be read as a present defect or a desired feature. Record the disagreement and clarify the rule text. Adjudication is a later joint decision, not part of the original agreement statistic.

Save the original independent sheets and table unchanged. Version the revised rule and test it on a new or explicitly declared recoded sample. Deliver the agreement table, missing-decision ledger, and concrete rule changes before applying the code more widely.