Build a docs site for your CLI with Thally: from zero to published, plus what the docs found in my code

Favour Bamgboye — Sat, 24 Oct 2026 15:45:33 GMT

I maintain PyDoctor CLI, a Python CLI that scans a project for vulnerable, outdated, conflicting, and unused dependencies. I rebuilt its documentation on Thally and published it at pydoctor-docs.thally.app. This article walks you through doing the same for your own tool: setup, configuration, writing content, and publishing. It ends with the obstacles I hit and how I resolved them, because most of them weren't Thally problems. Writing accurate docs exposed bugs in the tool itself.

You should be able to follow along in under an hour for a small CLI.

What you'll build

A live documentation site at https://YOUR-SITE.thally.app
A GitHub repository you own, containing the site as MDX pages
An introduction, a working quickstart, a practical guide, and a troubleshooting page
Automatic redeploys whenever you push to the production branch

Prerequisites

A Thally Cloud account at app.thally.io. A new workspace gets a one-time 7-day trial. It starts when you first publish, not when you browse the setup wizard, so check the dates before your site needs to stay up.
A GitHub account where you can install the Thally GitHub App.
Node.js and npm for editing and checking pages locally. Publishing the starter site in Step 1 doesn't need them.
The tool you're documenting, installed and working. You will need to run every command you document. More on that below.

Step 1: Create the site in Thally Cloud

Go to app.thally.io, sign in, and select Create a site.
Enter a site name, a short description, and the thally.app address you want. Thally checks the address is free before you continue.
At Docs source, choose Create a new docs repository. Thally creates a GitHub repository from its starter template and manages the first deployment.

When the GitHub App asks for access, this path needs All repositories, because the app has to initialise a repository that doesn't exist yet. If your organisation only allows selected-repository access, you'll need an existing Thally repository instead.
At Product sources, you can connect the repository of the tool you're documenting so Thally Track can watch it for merged changes. I connected pydoctor-cli. If you'd rather write everything by hand, choose I'll do this later.
Review the settings and select Publish site. When the workspace Home page shows the site as Live, open it from the site summary.

You now have a live site full of starter content, and a repository containing it.

Step 2: Clone the repository and learn the three files that matter

Clone the repository Thally created and install its dependencies. Replace YOUR-GITHUB-USERNAME and YOUR-DOCS-REPO with the values shown in your site summary.

git clone https://github.com/YOUR-GITHUB-USERNAME/YOUR-DOCS-REPO.git
cd YOUR-DOCS-REPO
npm ci

Most of your work happens in three places:

File	What it controls
`src/content/*.mdx`	The pages: frontmatter plus MDX content
`docs.json`	Tabs, sidebar groups, and page order
`src/data/site.ts`	Product name, description, and links

A page's path in docs.json is its file path under src/content/ without the extension. This is the navigation for my site:

{
  "tabs": [
    {
      "tab": "Documentation",
      "groups": [
        { "group": "Getting Started", "pages": ["introduction", "quickstart"] },
        { "group": "Reference", "pages": ["cli/commands", "configuration", "scanners"] },
        {
          "group": "Guides",
          "pages": ["guides/ci-integration", "guides/fixing-issues", "guides/troubleshooting"]
        },
        { "group": "Release Notes", "pages": ["changelog"] }
      ]
    }
  ]
}

Set your product name and description in src/data/site.ts so the header and metadata don't say "Documentation".

Step 3: Write the content

Generate a first draft (optional)

Because I'd connected pydoctor-cli as a product source, I used Thally's Generate product documentation to produce a first draft from the code. It arrived as a pull request from Thally's bot that replaced the starter pages with an introduction, quickstart, CLI reference, configuration page, scanner reference, and two guides.

The draft was a good starting point. Exit codes, environment variables, and the health-score bands all matched the source. But a generated draft only describes what the code says. It can't tell you what the tool actually does on a real machine, and that's where the important gaps turned out to be (see Obstacles).

Verify every command by running it

This is the step that matters most. For every command, flag, and code sample:

Run it, in a fresh virtual environment.
Compare the real output with what the page says.
Try the failure cases: no network, no virtual environment, a project with real problems.

python -m venv .venv && source .venv/bin/activate
pip install pydoctor-cli
pydoctor --version
pydoctor diagnose --json > report.json
echo $?

Use the components for the reader's job

Thally's MDX components cover what technical docs need. I used three:

/ for the quickstart, so each action is numbered and self-contained.

for things that will hurt the reader if they miss them, such as installing the wrong package:


  The PyPI package `pydoctor` is a different project. Both install a
  `pydoctor` command, so installing both breaks whichever came first.

/ for the troubleshooting page. Give each accordion an id and it gets a stable link that opens that entry directly, which is ideal for linking from error messages and other pages:
```
  
    **Symptom:** ...
    **Cause:** ...
    **Fix:** ...
  
```
https://pydoctor-docs.thally.app/guides/troubleshooting#scan-incomplete opens that entry already expanded.

I wrote every troubleshooting entry as symptom, cause, fix. Readers arrive with a symptom, so that's what the titles describe.

Validate before you push

Run Thally's content checker and a production build:

npm run check
npm run build

npm run check is defined in the starter and runs Thally's CLI checker for you, so you don't need to install anything else. The checker lints your content against Thally's rules, and the build catches MDX that won't compile.

Step 4: Publish

Commit and push to the production branch you chose in the wizard:

git add docs.json src/content
git commit -m "docs: add troubleshooting guide"
git push

Thally Cloud builds and deploys the new version automatically, and the deployment history records the commit. Open the live URL and check the sidebar, a code block, and any links with # anchors.

Obstacles, and how I resolved them

Thally itself was the easy part. The obstacles came from verifying the docs against the real tool.

1. The tool's name collides with another package. Searching for "pydoctor" returns Twisted's API documentation generator, which is also on PyPI as pydoctor. I installed both in one environment to see what happens. They write into the same pydoctor/ package directory and both provide a pydoctor command, so one silently breaks the other. Resolution: a in the quickstart's install step and a troubleshooting entry with the clean-reinstall commands.

2. The security scan reported "healthy" when it hadn't run. I ran the quickstart on a machine with no access to the OSV vulnerability database. The scan reported "No issues detected" with a score of 100/100. Network errors were being swallowed. Documenting that honestly would have meant writing "a clean result might mean nothing", so I fixed the tool instead. Resolution: the scan now fails closed with a SEC_SCAN_INCOMPLETE error and exit code 1 (pydoctor-cli 2.0.1), and the docs explain how to diagnose network and proxy problems.

3. CI failed on one runner out of twelve. After the fix, the project's own CI failed only on macOS with Python 3.13. The tool was right: that runner had a pre-installed certifi with a fresh advisory, which pip left in place because it already satisfied the requirements. The same log showed a second bug. One advisory listed under two IDs (GHSA and PYSEC) was scored twice, costing an extra 25 points. Resolution: advisories are de-duplicated by alias, CI installs dependencies with --upgrade-strategy eager, and the troubleshooting page now has an entry for runner-to-runner differences.

4. fix said "fixed" when nothing changed. Testing the fixing guide on a real Django project, pydoctor fix printed "✔ pillow fixed" dozens of times, but nothing was upgraded. uv add --upgrade pillow keeps an existing pin like pillow==12.2.0 and still exits successfully. The same run flagged packages such as whitenoise and django-cors-headers as unused, because Django loads them from strings in settings.py. Resolution: fix now targets a version, verifies the installed version afterwards, and asks once per package. The unused scanner understands import-name mappings, module paths in strings, and Procfiles (pydoctor-cli 2.1.0). The fixing guide documents exactly what fix runs and what it prints when an upgrade is blocked.

The pattern across all four: the docs didn't just describe the tool, they tested it. A page that tells someone to run a command is a promise that the command works. Keeping that promise meant changing the code, then documenting the new behaviour.

Takeaways

Getting a site live is fast. The Cloud wizard gives you a live site and a repository in minutes, without a local setup.
A generated draft is a starting point, not the finish. It's accurate about what the code says, but only running the tool tells you what it does.
Write troubleshooting as symptom → cause → fix, and give each entry an anchor you can link to.
Run check and build before every push. They're fast and they catch the mistakes that would otherwise ship.

The finished site is at pydoctor-docs.thally.app, and its source is at github.com/iamAgbaCoder/pydoctor-docs.

Next steps

Write and format pages covers every MDX component the starter supports.
Organize the navigation explains tabs and sidebar groups in more depth.
Customize the site shows how to set colors, logos, and fonts.

Favour Bamgboye · Developer Experience