AntFleet

Anatomy · c0feb3ca-1

last_checked is updated for UNREACHABLE entries, contradicting “last verified” intent and masking stale checks

mediumbugclosed in 4b9b492
repo 6f7fc663·PR #30·reviewed 1 week ago·closed 1 week ago

The vulnerable code

skills/skill-update-check/SKILL.md:97-103

979. **Update `last_checked` only — do NOT advance the locked SHA.** For every entry (including UP-TO-DATE ones), set a `last_checked` timestamp so the next run knows when each was last verified:
98   ```bash
99   jq --arg at "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" \
100     '[.[] | .last_checked = $at]' \
101     skills.lock > skills.lock.tmp && mv skills.lock.tmp skills.lock
102   ```
103   **Never auto-advance `commit_sha` in `skills.lock`**, even when the security verdict is PASS. Advancing the lock is a supply-chain trust decision that requires explicit human approval. The notification in step 10 will tell the operator how to advance manually.

The reasoning

Opus

`last_checked` jq update produces a JSON array, but lock entries are shown as standalone objects

mediumbugmedium
  • skills/skill-update-check/SKILL.md:10-19
  • skills/skill-update-check/SKILL.md:92-98
The Step 1 example shows a single object as 'each entry', implying skills.lock is either an array of such objects or NDJSON. The jq filter `[.[] | .last_checked = $at]` assumes skills.lock is already a JSON array. If skills.lock were NDJSON or a single object, this jq invocation would corrupt the file (collapsing it into an unintended array, or erroring). The doc does not specify the top-level structure of skills.lock, which is a data-loss risk: a mismatch will overwrite skills.lock with `skills.lock.tmp` that may not be parseable next run.

Recommendation

Explicitly state that skills.lock is a JSON array `[ {entry}, {entry} ]` and show that form in Step 1. Also gate the `mv` on jq's exit status (`set -e` or `&&` already does this, but add a non-empty check on the tmp file).

GPT-5

last_checked is updated for UNREACHABLE entries, contradicting “last verified” intent and masking stale checks

mediumbughigh
  • skills/skill-update-check/SKILL.md:97-103
The instruction says last_checked helps the next run know when each was last verified, but it updates every entry unconditionally. UNREACHABLE entries were not verified; updating last_checked for them falsely indicates verification happened and can hide persistent connectivity or access issues.

Recommendation

Only set last_checked for entries that were successfully reached and evaluated (UP-TO-DATE or CHANGED). For UNREACHABLE entries, either leave last_checked unchanged or add a separate last_attempted field, plus an error_reason, to track attempts without implying verification.

The agreement

Both frontier models flagged this within the same line range. AntFleet's unanimous gate fired — the finding posted on the PR. Closed in 4b9b492.

The fix

97
98### 6. Build the report at `articles/skill-update-check-${today}.md`
99
100Lead with a verdict line; then a triage table sorted by priority; then per-skill detail blocks for CRITICAL/HIGH/MEDIUM (LOW gets a compact list, no detail blocks). Up-to-date / unreachable / missing-upstream go in a compact footer table.
101
102```markdown
103# Skill Update Check — ${today}

Closure

Closed 1 week ago

SHA: 4b9b49251c8c9808bf147d55aa2930352af2e8c0

View closure receipt on GitHub →

Tweet thread template

tweet 1 of 8193 / 280

Two frontier models reviewed PR #30 on 6f7fc663. Both found this bug: medium bug: last_checked is updated for UNREACHABLE entries, contradicting “last verified” intent and masking stale checks

tweet 2 of 8127 / 280

The vulnerable code (skills/skill-update-check/SKILL.md:97-103): (full snippet at https://www.antfleet.dev/anatomy/c0feb3ca-1)

tweet 3 of 8280 / 280

What Opus saw: "The Step 1 example shows a single object as 'each entry', implying skills.lock is either an array of such objects or NDJSON. The jq filter `[.[] | .last_checked = $at]` assumes skills.lock is already a JSON array. If skills.lock were NDJSON or a single object, t…

tweet 4 of 8280 / 280

What GPT-5 saw: "The instruction says last_checked helps the next run know when each was last verified, but it updates every entry unconditionally. UNREACHABLE entries were not verified; updating last_checked for them falsely indicates verification happened and can hide persist…

tweet 5 of 897 / 280

Both flagged the same line range. AntFleet's unanimous gate fired — the finding posted on the PR.

tweet 6 of 893 / 280

The fix landed in commit 4b9b492: (view diff at https://www.antfleet.dev/anatomy/c0feb3ca-1)

tweet 7 of 881 / 280

AntFleet reviews every PR with two frontier models. Only unanimous findings post.

tweet 8 of 877 / 280

Full anatomy + reasoning + diffs: https://www.antfleet.dev/anatomy/c0feb3ca-1

Paste into X composer one tweet at a time. X has no multi-tweet intent API.

view receipt →all receipts →view disagreements →case studies →