The modelwatch logo: a clay owl perched on a Git branch

I ship lots of little projects with AI features (recently on the personal side: Side Eye, interface0, Congruence, daegent, Copapers, plus a variety of work-related products). And because of this, I have a niche problem: each of these projects is littered with references to models — claude-sonnet-5.5, gemini-3.8-flash, and so on. And those strings inevitably go out of date. Either a new version of the same model comes out, or there ends up being a better model for the use case (title generation, or decision model use, or whatever).

Sure, you can use “latest” aliases like ~google/gemini-flash-latest. I do sometimes for low-stakes situations. But the issue is that you get no chance to approve the change or run your evals. The new version ships and your app just flips over to it automatically. This can lead to degraded performance or even breaking changes in some cases (e.g. tool use or reasoning structure changes).

For package upgrades, we all use tools like Dependabot, which monitor for relevant upgrades and open PRs for you to review and approve. And I thought this would be a nice flow for this LLM use case as well.

So: meet modelwatch. It’s open source, runs in GitHub Actions on your own private or public repos, and is free to set up (you pay for your own inference via OpenRouter with a model of your choice, on the order of cents to tens of cents per repo per week for normal-sized projects using default settings when there are changes to make, or $0 with no changes).

The install script guides you through automatically creating your own GitHub app and installing it on the repos you want to track, which pushes a PR with a .github/modelwatch.yml file specific to that repo. That file keeps track of what model strings to replace and allows you to define any custom behavior (custom upgrade “lanes,” files to ignore, additional agent instructions, and so forth.)

The modelwatch.yml file for a given repo ends up looking something like this:

models:
  # Follow new Sonnet releases automatically.
  - anthropic/claude-sonnet-5

  # Follows your own choices for "title generation." In this case, Luna is the current choice in the codebase.
  - id: openai/gpt-6-luna
    lane: title-generator

  # Keep this "held" / pinned because changing would mean rebuilding embeddings.
  - id: openai/text-embedding-3-small
    hold: Changing models requires re-indexing our stored documents.

lanes:
  title-generator:
    chain:
      - openai/gpt-6-luna
      - google/gemini-3.8-flash # Your new current preference — this would cause modelwatch to open a PR upgrading title-generator above and in the code to gemini-3.8-flash.

notes: >
  There's a pricing table src/data/llm-prices.json, make sure to update this.

ignore:
  - test/fixtures/

In this case, when the action next runs (weekly by default), you’d get a PR on the repo upgrading Sonnet 5 to Sonnet 5.5, moving GPT-6 Luna to Gemini 3.8 Flash, and updating the pricing table in the data directory. Nothing would change in your test/fixtures directory. You’d then review the PR, run your evals if you had them, and merge in.

I’m excited to permanently remove “upgrade models on projects” from my weekly todo list.

Looking for more to read?

Want to hear about new essays? Subscribe to my roughly-monthly newsletter recapping my recent writing and things I'm enjoying:

And I'd love to hear from you directly: andy@andybromberg.com