Jonas Kamsker

Broken by Default: Claude Cowork on Windows

Thu, 19 Feb 2026 12:00:00 +0000

The Promise

I’ll be honest: the idea of Cowork excited me. An AI agent that lives on my desktop, manages files, automates tasks - the kind of thing that makes you feel like you’re living in the future. Point it at a workspace, give it a job, go make coffee.

So I installed it on Windows. Clicked the button. Waited.

And then Cowork looked me dead in the eyes and said: “The Claude API cannot be reached from Claude’s workspace.”

Which is a weird thing to say when I’m literally on the internet right now.

Two Error Messages, Zero Clarity

Cowork doesn’t fail with one error. It fails with two, and they look like completely different problems.

Error 1 (the misleading one):

“The Claude API cannot be reached from Claude’s workspace…”

Your first instinct: is Anthropic down? You check. There isn’t an outage. You can resolve api.anthropic.com from your terminal. Port 443 is open. Your host machine has internet. Everything is fine.

Except nothing is fine.

Error 2 (the real one, hidden behind “longer loading”):

CLI output was not valid JSON … Output: sandbox-helper: host share not mounted at /mnt/.virtiofs-root/shared: not a mount point

There it is. Buried in the second error message like a confession at the bottom of a Terms of Service. The VM’s filesystem is broken, the CLI process outputs an error string instead of JSON, and Cowork’s parser falls over because it expected structured data and got a cry for help.

Two symptoms. Three root causes. One very confused developer.

And here’s the part that really bothers me: Anthropic is positioning Cowork squarely at non-technical users. Knowledge workers. The marketing says “Claude Code power for the rest of your work.” If I - someone who debugs Hyper-V networking as a semi-regular hobby - spent hours in PowerShell diagnosing this, a non-dev user has exactly zero chance. They see “API unreachable,” they Google it, they find nothing useful, and they uninstall. That’s not a speed bump - that’s a product cliff with no guardrail.

What’s Actually Happening Under the Hood

Here’s the thing the “API unreachable” error doesn’t tell you: Cowork doesn’t run on your machine. Not really.

Cowork runs inside a dedicated Linux VM - a full virtual machine running on Hyper-V (Windows’ built-in hypervisor, the same technology that powers WSL2 and Docker Desktop). Under the hood, Cowork talks to Microsoft’s Host Compute Service (HCS) - a low-level API for creating and managing VMs that sits beneath the friendlier tools like Hyper-V Manager. This means Cowork’s VM may not show up as a normal “VM” in Hyper-V Manager - it’s registered at the platform level rather than as a classic Hyper-V VM. But here’s what tripped me up: Cowork does not share WSL2’s VM. It’s not a distro running inside the WSL2 lightweight utility VM. It’s not piggybacking on Docker’s backend either. It boots its own completely independent virtual machine, with its own kernel, its own root filesystem, and its own networking stack.

Think of Hyper-V as an apartment building. WSL2 is one tenant. Docker Desktop is another. Cowork moves in as a third - separate apartment, separate lease, separate plumbing. They share the building’s foundation (the hypervisor), but nothing else. When Cowork’s plumbing breaks, WSL2 keeps running fine. The reverse is also true - except for one fun bug where Cowork’s virtual network permanently breaks WSL2’s internet until you manually find and delete the offending network configuration using Windows’ HNS diagnostic tools. Good neighbors.

The whole thing is managed by a dedicated Windows service called CoworkVMService (cowork-svc.exe). The VM bundle lives inside Claude Desktop’s app data — on my machine (Microsoft Store install) it was at %LOCALAPPDATA%\Packages\Claude_*\LocalCache\Roaming\Claude\vm_bundles\claudevm.bundle\ (some non-Store installs use %APPDATA%\Claude\vm_bundles\claudevm.bundle\). It contains a ~9.4 GB Linux root filesystem (rootfs.vhdx - VHDX is Hyper-V’s virtual hard disk format), a Linux kernel (vmlinuz), an initial RAM disk for bootstrapping (initrd), and a persistent state disk (sessiondata.vhdx) that stores the VM’s session data between restarts. That last file becomes very relevant in about three paragraphs. On macOS, Cowork uses Apple’s Virtualization Framework instead of Hyper-V - same concept, different hypervisor, and roughly a month more maturity since it launched first.

The VM connects to the outside world through a virtual network adapter (vEthernet (cowork-vm-nat)) on its own private IP range (172.16.0.0/24). Two Windows services make this work: HNS (Host Networking Service) orchestrates the virtual network - think of it as the VM’s network card and cabling. WinNAT (Windows Network Address Translation) then provides the actual internet routing - it translates the VM’s private IP addresses into your host’s real ones so traffic can flow in and out. Without HNS, the VM has no network. Without WinNAT, the VM has a network that goes nowhere.

Host folders get shared into the VM via VirtioFS, a high-performance file sharing protocol designed for virtual machines (similar to how Docker mounts host directories into containers). Your workspace folder appears inside the VM at /mnt/.virtiofs-root/shared/.

When either the networking or the filesystem sharing breaks, Cowork doesn’t degrade gracefully - it faceplants into error messages that point everywhere except the actual problem.

The Diagnosis: Three Layers of Broken

I pulled up an admin PowerShell and started poking. What I found was a layer cake of failure - each layer independently capable of killing Cowork, and all three broken simultaneously.

Layer 1: The VM’s Network Adapter Had No DNS

Get-DnsClientServerAddress -InterfaceAlias 'vEthernet (cowork-vm-nat)'

ServerAddresses : {}

Empty. The virtual network adapter that Cowork’s VM uses to resolve domain names had no DNS servers configured. The VM was living in a world where domain names were a theoretical concept.

Meanwhile, the host was merrily resolving DNS on its own adapters, completely unaware that its VM tenant was sitting in the dark.

Layer 2: WinNAT Was Just… Gone

This is the one that really got me. The virtual network existed - HNS (the service that manages Cowork’s virtual network, remember) showed it, subnet 172.16.0.0/24, gateway 172.16.0.1, all looking perfectly normal:

Get-NetNat

Output: nothing. Empty. The WinNAT object that’s supposed to provide outbound internet access for the VM’s subnet simply wasn’t there.

The VM had an IP address. It had a gateway. It had a virtual switch. What it didn’t have was a NAT rule to actually translate its traffic to the outside world. A house with a front door and a brick wall where the street should be.

The Cowork VM logs confirmed it:

API reachability: PROBABLY_UNREACHABLE

And later, having given up on optimism:

API reachability: UNREACHABLE

This is a known issue, by the way. Multiple users have reported it. The installer creates the virtual network via HNS but doesn’t reliably create the corresponding WinNAT rule that gives that network internet access. And if you’re running VPN software, it gets worse - VPNs are fundamentally incompatible with Cowork’s NAT setup because VPN split-tunnel rules don’t apply to NAT’d VM traffic. The VPN doesn’t know Cowork’s VM exists. It can’t route for a tenant it’s never met.

Layer 3: The VM’s Virtual Disk Was Corrupted

Even if networking were perfect, Cowork still wouldn’t have started. Remember sessiondata.vhdx? The VM’s persistent state disk? It was in an inconsistent state, which meant the VirtioFS host share - the file sharing bridge between your Windows folders and the VM - failed to mount.

The sandbox helper process tried to set up the environment, discovered the mount point was broken, and printed an error to stdout. The Cowork CLI, expecting a stream of JSON, got plaintext instead. Parser meets unexpected input. Parser loses.

sandbox-helper: host share not mounted at /mnt/.virtiofs-root/shared: not a mount point

That’s the line that produces the “CLI output was not valid JSON” error. Not a JSON problem. Not a CLI problem. A filesystem problem wearing a JSON mask.

The Fix: Three Commands and a File Rename

Once you know what’s actually broken, the fix is almost anticlimactic.

Fix 1: Give the VM DNS

$alias='vEthernet (cowork-vm-nat)'

# In my case: my LAN resolver + Cloudflare as fallback
Set-DnsClientServerAddress -InterfaceAlias $alias -ServerAddresses @('10.0.0.45','1.1.1.1')
Clear-DnsClientCache

Restart-Service CoworkVMService -Force

You can verify it took:

Get-DnsClientServerAddress -InterfaceAlias 'vEthernet (cowork-vm-nat)'
# Before: {}
# After:  {10.0.0.45, 1.1.1.1}

If you have specific DNS requirements (corporate resolvers, etc.), swap in whatever makes sense for your environment.

Fix 2: Recreate WinNAT

Windows 11 Home note: Some users report Get-NetNat / New-NetNat aren’t available or fail on Home editions (missing NetNat/WMI components). If you’re on Home, you may be dealing with a different class of problem than “missing NAT rule” and may need a different workaround or a supported Windows edition.

This is the big one. Without this, the VM has no internet - period.

New-NetNat -Name cowork-vm-nat -InternalIPInterfaceAddressPrefix 172.16.0.0/24

Name          : cowork-vm-nat
Active        : True

Two lines. That’s it. The entire difference between “API unreachable” and a working Cowork instance is a single New-NetNat call that Windows silently decided not to persist.

Fix 3: Reset the VM State

The corrupted sessiondata.vhdx needs to go. But we’re cautious, so we rename instead of delete:

$svc='CoworkVMService'

# Auto-detect bundle path (Store install uses a versioned package folder)
$bundle = (Get-Item "$env:LOCALAPPDATA\Packages\Claude_*\LocalCache\Roaming\Claude\vm_bundles\claudevm.bundle" -ErrorAction SilentlyContinue).FullName
if (-not $bundle) { $bundle = "$env:APPDATA\Claude\vm_bundles\claudevm.bundle" }

$session=Join-Path $bundle 'sessiondata.vhdx'
$bak="$session.bak.$(Get-Date -Format 'yyyyMMdd-HHmmss')"

Stop-Service $svc -Force
Rename-Item -Path $session -NewName (Split-Path $bak -Leaf)
Start-Service $svc

In my case, Cowork created a fresh sessiondata.vhdx on next start. The old one sits there timestamped, waiting for the forensic investigation you’ll never do. If your install never creates sessiondata.vhdx at all - or it’s still broken after this - you’re likely hitting a different setup bug and may need a full reinstall.

After the Fixes

Restart everything properly - and I do mean properly:

Quit Claude Desktop from the system tray (not just closing the window - actually Exit)
Restart CoworkVMService if it isn’t already running
Relaunch and point Cowork at a simple, local workspace path (not OneDrive, not a library - a plain C:\SomeFolder)
If it still flakes, temporarily disable VPN/tunnel adapters and try again

That last point deserves emphasis. VPN software, network tunnels, and virtual adapters are the most common reason Windows “loses” NAT rules or DNS configurations for Hyper-V virtual switches. Remember: Cowork’s VM has its own networking stack, completely separate from WSL2 and Docker. When your VPN reconfigures routing tables, it doesn’t know or care that there’s a third VM that also needs internet.

Debugging Depth Meter: [████████░░] 8/10

  Layer 1: DNS empty on VM adapter - annoying but diagnosable
  Layer 2: WinNAT missing entirely - invisible unless you know to check
  Layer 3: VM disk corrupted - produces an error that looks like a JSON bug
  Bonus:   VPN adapters silently antagonizing all of the above

Why This Was Hard to Find

The error messages are the real villain. “API unreachable” sends you down the wrong path entirely - you start checking Anthropic’s status page, your firewall, your proxy settings. The second error about invalid JSON sounds like a Cowork bug. Neither says “your Windows NAT layer is missing and your virtual disk is corrupted.”

The diagnostic path requires you to already know that Cowork runs its own dedicated Hyper-V VM (not inside WSL2, not Docker’s VM - its own independent instance via HCS), that this VM relies on WinNAT for internet access, and that VirtioFS mounts can go stale when sessiondata.vhdx gets corrupted. That’s not something a normal user would ever figure out. It’s barely something a developer would figure out without falling down the right rabbit hole.

And that’s the core tension. Cowork is marketed at the people least equipped to debug it when it breaks. The fix is three PowerShell commands, but the path to discovering those three commands requires knowledge that Anthropic’s target audience definitionally does not have.

The relevant logs live in %LOCALAPPDATA%\Packages\Claude_*\LocalCache\Roaming\Claude\logs\ (Microsoft Store install) or %APPDATA%\Claude\logs\ (non-Store install) - specifically cowork_vm_node.log for VM networking status and main.log for the CLI/sandbox errors. If you’re hitting anything like what I described, start there.

FAQ (Partially Helpful, Fully Honest)

Why did WinNAT disappear? Best guess: a Windows Update, a VPN install, or a Hyper-V reconfiguration silently cleared it. Windows doesn’t warn you. It just lets NAT rules evaporate like a goldfish releasing a memory.

Will this happen again? Probably. WinNAT has the persistence of a New Year’s resolution. Check Get-NetNat after major updates.

Wait, so Cowork, WSL2, and Docker are all separate VMs? Not exactly. WSL2 runs a lightweight utility VM, and multiple WSL2 distros share that same underlying VM/kernel. Docker Desktop can run either inside WSL2 (as the docker-desktop distro) or as a separate Hyper-V VM depending on configuration. Cowork appears to run work in its own isolated VM environment and creates its own networking artifacts (like cowork-vm-nat). Three tenants, one building, zero coordination on plumbing.

Does Cowork break WSL2? It can. Issue #26216 documents Cowork’s virtual network (managed by HNS) permanently breaking WSL2’s internet. The fix is deleting the cowork-vm-nat network entry via Get-HnsNetwork | Where-Object { $_.Name -eq "cowork-vm-nat" } | Remove-HnsNetwork, which you’ll need to redo every time Claude Desktop recreates it.

Is this a Cowork bug or a Windows bug? Both. The error messages are Cowork’s fault - surfacing “WinNAT missing” instead of “API unreachable” would save hours. But Windows silently dropping NAT configurations isn’t Cowork’s doing. It’s Cowork trusting Windows to hold its drink. Windows dropped it.

Can a non-developer fix this? No. And that’s the problem worth talking about.

The Takeaway

Three layers of broken. Three commands to fix. Two hours to figure out which three.

The root cause, stripped of narrative: Cowork’s Linux VM - a dedicated Hyper-V instance, separate from WSL2 and Docker - lost outbound internet because Windows dropped the NAT rule, lost DNS because the virtual adapter was misconfigured, and couldn’t mount host folders because the virtual disk was corrupted. Each failure produced a different symptom, none of which pointed at the actual problem.

If you’re on Windows and Cowork won’t start, run Get-NetNat and check whether Get-HnsNetwork | Where-Object { $_.Name -eq "cowork-vm-nat" } returns anything. An empty Get-NetNat combined with an existing cowork-vm-nat HNS network is a strong signal you’re in the “missing NAT rule” failure mode. Everything else is cleanup.

And yes, I debugged a Linux VM networking issue by writing PowerShell. The year is 2026 and nothing makes sense.

Forgejo's CLI Can't Show Build Details? Fine. I'll Do It Myself.

Wed, 18 Feb 2026 12:00:00 +0000

TL;DR: Forgejo Actions has no usable API or CLI surface for runs, logs, or artifacts. I built fj-ex, a fj-style companion that scrapes the web UI’s embedded JSON so humans and AI agents can manage CI from the terminal. Yes, it’s scraping. No, I don’t feel bad about it.

A Confession and a Problem

I’ll be honest: I’m not great at CI/CD. I don’t enjoy writing pipeline configs, I don’t enjoy debugging them in production, emotionally, and I especially don’t enjoy playing “spot the difference” between yesterday’s green run and today’s red run - where the only change is that the build system woke up and chose violence.

If a build breaks at 09:03, I want it fixed by 09:04. Not a 40-minute archaeological dig through logs that read like a toaster having a panic attack.

So I do what any (un)reasonable developer in 2026 does - I let AI agents handle it. They write the workflows, they iterate on failures, they fix the weird YAML indentation issues. I review the results. It’s a great arrangement.

This works beautifully on GitHub, because gh - GitHub’s CLI - covers everything. Runs, logs, artifacts, cancellations, reruns. An AI agent with access to gh can see what’s happening, read the logs, download artifacts, retry failures, all without ever touching a browser. It has hands and feet. It can walk around and get things done.

Then my company started migrating to self-hosted platforms. Forgejo, specifically - open source, lightweight, GitHub-compatible Actions. Great choice for a lot of reasons.

One problem: the moment we moved, my AI agents lost their legs.

The Gap

Forgejo CLI (fj) is solid. Repos, issues, PRs, releases - all there, all from the terminal. It’s the gh equivalent, and for the things it covers, it covers them well.

But Forgejo Actions? Nothing. No run listing. No log downloads. No artifacts. No cancel. No rerun. These features exist exclusively behind the web UI.

To put it another way:

What gh lets an agent do: list runs, read logs, download artifacts, cancel jobs, rerun failures - never open a browser.

What fj lets an agent do: …have you tried clicking?

For a human, that’s annoying. You alt-tab, you click around, you lie to yourself that this is fine.

For an AI agent? It’s a brick wall. Agents don’t have browsers. They have terminals and CLI tools. If there’s no command for it, it doesn’t exist. My agents went from autonomously managing the full CI/CD lifecycle on GitHub to being completely helpless the moment a build failed on Forgejo. They were a brilliant brain in a jar - with no network adapters.

I was back to manually debugging pipelines. The one thing I was specifically trying to not do.

What I Needed

The dream was simple - give my agents (and myself, when I’m feeling brave) the same experience gh provides:

# What's happening?
fj-ex actions runs --limit 20

# Let me read the tea leaves (whole run)
fj-ex actions logs run --run-index 42

# ...or a specific job inside that run
fj-ex actions logs job --run-index 42 --job-index 0

# Grab the goods
fj-ex actions artifacts get --run-index 42 --artifact build-output --out-file build-output.zip

# Nope, kill it
fj-ex actions cancel --run-index 42

# Hope springs eternal
fj-ex actions rerun --run-index 42

Terminal. One line. Done. Something an agent can call, parse the output of, and reason about.

And it had to feel like fj. Same --host/-H, --repo/-r, --remote/-R flags. Same git-remote inference. Same subcommand style. Not a fork, not a replacement - a companion. Hence the name: fj-ex, Forgejo CLI Extension.

I didn’t need to build a new brain. I needed to build knees.

How I Got There

It started as a few PowerShell scripts. Just enough to stop the bleeding while we migrated repos. And while hacking those together, I found the detail that made this whole project viable.

Forgejo’s frontend developers - bless their hearts - left the keys in the ignition.

Their web UI embeds structured JSON directly in data-* HTML attributes. On a run page, the initial UI state (and sometimes artifact metadata) is right there in the markup (HTML-escaped, but still JSON once you decode entities):

<div
  data-initial-post-response="{&quot;state&quot;:{&quot;run&quot;:{&quot;jobs&quot;:[{&quot;id&quot;:123,&quot;name&quot;:&quot;build&quot;,&quot;status&quot;:&quot;failure&quot;}]}}}"
  data-initial-artifacts-response="[{&quot;id&quot;:15,&quot;name&quot;:&quot;build-output&quot;,&quot;size&quot;:204800}]">
</div>

No headless browser needed. No DOM spelunking. Just: fetch the page, yank the attribute(s), decode HTML entities, parse the JSON, pretend this was an API all along.

That’s not an API… but it is data.

The PowerShell scripts worked until I wanted one more feature and realized I was fighting PowerShell harder than the actual problem. My developer ego demanded type safety for what is essentially a glorified curl script. So I rewrote it in Rust - not because it was the right choice, but because the only two modes I have are “quick hack” and “mass rewrite in a systems language.”

fj-ex essentially pretends to be a browser. Same HTTP requests, same cookie-based auth, same CSRF token dance for mutations like cancel and rerun. Since there’s no API token for any of this, it logs in the human way - username, password, stash the session cookies. Is storing credentials ideal? No. Is there an alternative when you’re authenticating against a login form that doesn’t support tokens? Also no. The README doesn’t hide this.

What It Actually Feels Like

Here’s why this matters beyond the technical trick: an AI agent with fj-ex installed can now do the full loop.

Build fails → agent runs fj-ex actions runs to see what happened → reads the logs with fj-ex actions logs run --run-index <n> (or ... logs job --run-index <n> --job-index <n>) → figures out the issue → pushes a fix → monitors the rerun. All autonomously. All in the terminal. No human required to go click around in a web UI on the agent’s behalf.

Forgejo handed my agents a beautiful map of the world. I just gave them back their shoes.

And for the times I do look at CI/CD myself, it’s just… nicer. Logs are text in my terminal. I can pipe them into grep, rg, less, whatever. Artifacts download to my current directory. Switching repos is a -r flag away. The whole thing gets out of the way and lets me get back to the part of my job I actually enjoy.

Is This Cursed? A Little.

Cursedness Meter: [███████░░░] 7/10

  ✅ Works today
  ✅ No headless browser required
  ✅ Structured JSON, not regex-over-HTML
  ❌ Scraping behind auth
  ❌ Stored session cookies
  ❌ Will break if Forgejo redesigns

On a cursedness scale from “regex to parse HTML” to “running production on SQLite,” this lands at “screen-scraping behind auth with stored credentials.” So, you know, Tuesday.

But the alternative was either contributing the missing API endpoints upstream - a much larger undertaking, and one I’d still welcome - or telling my agents “sorry, you’re on your own” every time a Forgejo build failed. I chose the pragmatic option.

And if Forgejo does add proper API support for Actions someday? Great. The commands stay the same, only the plumbing changes. Migrating away from scraping would be the happiest refactor I’ve ever done.

FAQ (Half Useful, Half Honest)

Does this break if Forgejo changes the UI? Yes. That’s the pact. I scrape, they ship, I pray.

Is storing session cookies ideal? No. But neither is opening a browser in 2026 to check if a build passed.

Why Rust? Because the alternative was maintaining PowerShell, and I’ve suffered enough.

Will you replace scraping with a real API later? The day Forgejo exposes Actions in the API, I will refactor with mass joy and mass cargo rm.

Go Get It

fj-ex is on crates.io, on GitHub, and (of course) on Codeberg with pre-built binaries for Linux, Windows, and macOS.

My agents have their legs back. My Forgejo tabs are closed. And I’m back to doing what I do best - reviewing the work someone else did and mass-approving it pretending I understood every change.

And yes, I wrote Rust to avoid clicking a website.

The Site Is on Fire. Here's Your FTP Password. Good Luck.

Wed, 18 Feb 2026 12:00:00 +0000

A Confession

I’ll be honest: I have a thing for dirty code.

Not my dirty code - I write pristine, well-architected systems that definitely don’t have // TODO: fix this later comments from 2019. No, I mean other people’s dirty code. Legacy systems. The kind of codebase where the README says “just works” and the reality says “just barely.” Hand me FTP credentials to a decade-old PHP site that’s held together by session variables and sheer institutional denial, and something in my brain lights up like a Christmas tree.

Most developers see a legacy dumpster fire and feel dread. I feel the same thing a cave diver feels staring into a dark hole in the ground: this is going to be awful, I might not come back the same person, and I absolutely cannot wait to go in.

So when a client’s legacy SilverStripe site landed on my desk a while back - “not working,” they said, which undersells it the way “the Titanic had a minor hull issue” undersells maritime history - I didn’t groan. I cracked my knuckles.

The admin filters were decorative - you could select them, the UI would update, and nothing would change. The statistics page took minutes to render. And the CSV export button? That one didn’t produce a slow page. It produced a dead server. The people who actually needed this data couldn’t get it; the site was a locked filing cabinet that electrocuted you when you touched the handle.

The existing workaround was… creative. Since the export was broken, someone had set up an AI agent to read every outgoing invoice as PDF (which was mirrored to a dedicated email address) - and scrape the numbers out of them to reconstruct the data the admin panel couldn’t provide. An AI reading every invoice to reverse-engineer the database. In production.

I’ll spare you my exact reaction. What I said was “I’ll take a look at the site.”

The ask was straightforward: build a bot that extracts the data the admin panel can’t export, and send the reports to the client. That’s it. No fixing the site. No debugging. Just get the data out.

All I had to work with was admin login credentials and the live web UI. No server access. No FTP. No SSH. No deployment pipeline. Just a username, a password, and a website that killed itself the moment you asked it to do anything useful.

The plan was simple: build the bot, get the data flowing, hand it over, walk away.

You can probably guess how that went.

Build the Bot (The Part They Paid Me For)

First order of business: the client needed order exports today, and the export button was one of the things producing 500s. So I needed a workaround that bypassed the broken UI entirely.

I wrote a Python exporter that logs into the admin panel and triggers the same SilverStripe GridField export actions a human would click, except it does it over HTTP, with structured logging, and without crashing. Added redaction to the HTTP logs so cookies and CSRF tokens wouldn’t end up in version control, because I’ve read enough post-mortems to know that’s how you get a second incident.

Even while the building was on fire, I refactored the exporter into a clean src/ layout with shared modules and ran python -m compileall as a sanity check. Professionalism is a disease - you can’t turn it off even when you should.

By mid-afternoon, non-technical team members could trigger exports through a chat slash command and get the CSV back in their messaging client. No admin panel required. No 500s. Just data. The PDF-scraping workaround? Quietly retired.

I also wrapped the exporter in a serverless function and added CI that automatically exports the past month’s data on every push and uploads it as an artifact.

Job done. Bot built. Data flowing. Walk away.

I did not walk away.

The Brain Goblin

The bot worked. The reports were landing. The client was happy. The rational thing was to close the laptop and move on.

But the site was still broken. The admin panel was still 500ing. The filters were still lying. And somewhere in the back of my skull, a small, irresponsible voice was saying: you could fix this. You know you could fix this. It would be so satisfying to fix this.

I call this voice the brain goblin. It’s the part of my brain that sees a dumpster fire and doesn’t think “someone should deal with that” - it thinks “I should deal with that, right now, tonight.”

There’s a catch, though: I’m not a PHP developer. I was, once, in the dark ages - back when mysql_real_escape_string was considered security and deploying meant dragging files into FileZilla. But that was a long time ago. I couldn’t fix this myself.

But I could build the tools to let an AI agent fix it for me.

The agent is a better PHP developer than I am. It can read SilverStripe framework code, trace ORM call chains, spot N+1 patterns, and suggest fixes in a language I haven’t seriously written in over a decade. What it can’t do is see what’s happening on a remote server. It can’t read logs that live behind FTP. It can’t query a database it doesn’t have credentials for. It can’t upload a patched file.

The agent was a brain in a jar. My job was to build the jar some legs.

I got FTP access. And from there, the plan took shape: build CLI tools that let the agent fetch evidence over FTP, tail logs, query the database, and upload surgical fixes one file at a time. Most of the tooling was done by midnight on day one. I have the commit history to prove it, and the commit messages to prove I was losing it.

Build the Operating Room

The constraints were, as they say, chef’s kiss. FTP/FTPS access. No SSH. No deployment pipeline. Changes had to be uploaded file-by-file, like it was 2003 and we were all pretending this was fine. Some requests died before SilverStripe could even log them - Apache would just emit “End of script output before headers,” the server equivalent of a shrug, and move on.

But the FTP folders had secrets. MySQL credentials in a config file - that got me database access. FTP alone was painfully slow for exploratory debugging, though. I needed something closer to shell access. So I did what any principled engineer would do: I uploaded a temporary, purpose-built web shell, used it to gather intel on the server environment, and removed it when I was done. Resume material? No. But the only door they gave me was FTP.

So I built a portable CLI toolkit, piece by piece:

FTP log tailing. A script that connects over FTPS, downloads the latest log entries, and formats them locally. Because tail -f is great when you have SSH. When you have FTP, you improvise.

Read-only database CLI. A command-line tool that talks to the remote database, with guardrails - only SELECT, SHOW, DESCRIBE, and EXPLAIN are allowed, and it rejects multi-statement queries. I needed the agent to inspect data, not accidentally DROP TABLE a client’s order history because of a fat-fingered semicolon. Yes, I wrote unit tests for the query guardrails. During the production firefight. Like I said: disease.

FTP put for file-by-file deploys. Upload exactly one changed file and nothing else. Verify by re-downloading the remote file and comparing SHA256 hashes - or git diff --no-index for the paranoid (me). This became the deployment “pipeline.” It’s the saddest CI/CD you’ve ever seen, and it worked perfectly.

The repo stopped being “just an exporter” and became a portable operations toolbox - the agent’s nervous system for a site we could only reach by FTP.

Bring the Site Into Version Control (So You Can Think About It)

You can’t reason about code that exists only on a remote server you access via FTP. So I mirrored the entire webroot locally and committed it. A download script using eight parallel FTP connections pulled over a thousand source files - PHP, JS, CSS, templates, configs - with zero failures. For the first time in this site’s life, it was in version control.

I told the agent explicitly: source files only. No media. Do not download images.

It downloaded 300 GB of media before my SSD ran out of space.

The agent had decided, with the quiet confidence of a golden retriever carrying a tree branch through a doorway, that “mirror the webroot” meant mirror the webroot. Every customer photo. Every generated thumbnail. Every asset uploaded since the mid-2010s. My SSD just tapped out first. We had a conversation about boundaries.

The webroot itself was a geological record of deployment strategies. website_v1, website_v2, website_old, website_really_old - all sitting right next to the actual production folder, like roommates who’d stopped acknowledging each other. Thousands of orphaned images in the root directory. A zip export from literally a decade ago that “should have been deleted a few days after generation.”

I also started writing structured bug reports as separate markdown files, because by this point I had enough threads going that my brain alone was not a reliable storage medium.

Speaking of error logs: I pulled roughly a year’s worth. About two thousand logged error events. Promising. Then I noticed ninety-five percent of them were the same bug on a single endpoint, screaming on repeat like a car alarm nobody disconnects. The actual admin 500s I was hunting? Barely a whisper underneath.

From here, fixes became normal engineering: the agent traces the issue through the codebase, suggests a patch, I review it, commit, upload the changed file, verify by re-downloading and diffing.

Same site. Suddenly legible.

What Was Actually Broken

Here’s where it gets fun. “Fun.”

The Type Filter That Referenced a Ghost Column

The admin orders page had a “Type” dropdown filter. When you selected an option, the UI sent a query parameter like q[Type]=CouponItemID. The server-side code obediently tried to filter on a column called CouponItemID.

That column didn’t exist.

The dropdown values were raw internal identifiers that some past developer had wired directly into the UI - and at some point the schema had changed underneath them. The filter was sending SQL WHERE clauses into the void. The database responded by dying. A reasonable reaction, honestly.

The fix: change the dropdown values to semantic keys and map them to the columns that actually exist. Fifteen minutes of work, once you know where to look. Finding where to look took considerably longer.

The Filters That Didn’t Filter (And the Export That Paid the Price)

This was the big one. Not one bug, but a constellation of failures that all pointed in the same direction: the database was always loading everything.

Most of the admin filters were decorative. They looked functional - dropdowns selected, date ranges set, UI updated - but under the hood, the filter state wasn’t being applied to the actual query. Every admin page load was hitting the unfiltered dataset. Tens of thousands of orders, every time.

The only thing keeping the paginated view alive was pagination itself: slice results to 50 rows, hand them to PHP, done. Slow, lying, and demoralizing - but survivable. The server stayed up. The page rendered. Admins could at least see their orders, even if the filters were theater.

But even the paginated view was lying. result.count() was still counting the entire unfiltered dataset on every load. The UI would show “1–50 of 19,550” regardless of what you’d filtered to, because the count didn’t know about the filter either.

Then someone clicked “Export as CSV.”

That button was the kill switch. SilverStripe’s GridField export does one thing the paginated view never did: it removes pagination. All of it. When the filters work, that’s fine - you’re exporting a bounded dataset. When the filters are broken and “the entire result set” means every order ever taken, combined with summary fields that traverse ORM relationships per row - classic N+1 - you get a request that churns through tens of thousands of rows, firing relationship queries for each one, until PHP runs out of either time or memory. The paginated dashboard could limp along; the export button just killed the server outright. Apache would return a raw error before SilverStripe’s error handler even woke up.

The fix came in three layers. Mitigation: default to a bounded date range when no explicit filters are set, because exporting the entire history of a company in one HTTP request is not a feature, it’s a dare. Correctness: make the export action carry the current filter state, which required changes in both the GridField JavaScript and the server-side code. Performance: override the export columns to skip expensive per-row relationship traversals.

Date Filters That Filtered on the Wrong Date

The admin date presets - “Last 7 days,” “Last 3 months,” “Today” - were filtering on the wrong timestamp column. The paginator count didn’t reflect the filtered list. Filter state wasn’t persisted across page loads.

That last one had a framework-specific root cause: in SilverStripe’s GridField, data manipulators apply in component order. The date filter was running after the paginator, so the paginator computed totals against the unfiltered list. The UI would show all results regardless of the date range. Later pages were empty. The admin panel was gaslighting its own users.

Multiple fixes: correct the timestamp column, fix the preset logic, insert the date filter component before the paginator, and add session-backed persistence. A sane default (“Past 3 months” instead of “everything ever”) and a visual highlight of the active preset followed the next day.

“We Can’t See the Error”

The meta-bug. The reason everything else took so long.

Some failures happened entirely in PHP’s shutdown phase - fatal errors, uncaught exceptions, memory exhaustion. SilverStripe’s error handler never ran. Apache logged a one-liner. The admin saw a blank page or a generic 500. Nobody knew what broke.

I added temporary, gated observability: log fatal shutdown errors and uncaught exceptions to a writable temp file (readable over FTP), and optionally stream detailed error output to the response - but only for authenticated admins, and gated behind a flag that defaults to off. Just enough visibility to diagnose, without turning production into a confessional booth for every visitor.

Once the agent could see the errors, every other fix fell into place within hours.

The Statistics Page That Rendered the Entire Universe

This one wasn’t a 500 - it was a 200 that took minutes.

The statistics admin page rendered all tabs server-side in a single initial request. Orders, coupons, registrations - everything, all at once, like a waiter who brings every item on the menu to your table and asks you to pick. Under the hood, a filter stats code path was iterating tens of thousands of records, firing a COUNT(*) per record against a table with millions of rows and no useful indexes. A single text-matching filter query took about 13 seconds. Multiply by 20 filters.

The fixes: rewrite aggregate queries to avoid N+1 loops, cache repeated lookups per request, apply a default date range on entry so the page doesn’t try to summarize all of recorded history, and optimize relationship queries to stop building enormous IN (...) lists. Performance went from “go make coffee” to “tolerable.” Still not fast. But the building was no longer on fire.

The Server Punches Back

One more thing. After deploying a fix, I got an immediate Parse error: unexpected '?' in production.

The fix used PHP’s ?? null coalescing operator. The server’s PHP version was too old to support it.

I rewrote it as isset($x) ? $x : $default, re-uploaded over FTP, and added “the server is running ancient PHP” to my mental model.

Oh, and sometimes I’d upload the correct fix and the site would still show the old behavior. OPcache had decided the previous code was fine, actually, and combined JS caches weren’t helping either. The mitigation was a combination of ?flush=all, clearing the combined-files cache directory, and occasionally toggling settings in the hosting panel to nuke the opcode cache. You haven’t lived until you’ve debugged a fix that’s correct but invisible because the runtime is nostalgic.

Aftermath

Once the fires were out, I turned the one-off tooling into something reusable: user export CLI with registration filters, integration with an email marketing platform as a serverless function, and - critically - documentation. Split and structured so that the next person who gets handed FTP creds and a vague description of “it’s not working” has a slightly less terrible day than I did.

A couple weeks later, another production issue surfaced: path handling bugs where $_SERVER["DOCUMENT_ROOT"] resolved to /framework under SilverStripe routing instead of the actual webroot. Replaced brittle document root references with Director::baseFolder() and hardened the CLI image generation scripts to normalize paths properly. Same workflow - patch, commit, FTP upload, verify. The toolbox held up.

Oh, and they gave me SSH access. After I’d already fixed everything. Classic.

The Vibe Check

Deployment Sophistication Meter: [██░░░░░░░░] 2/10

  ✅ Changes are in version control
  ✅ Diffs are verified post-deploy
  ✅ Database access is read-only by default
  ✅ Exports work without the admin panel
  ✅ Unit tests exist for the FTP and DB tooling
  ✅ AI agent can autonomously trace bugs through the codebase
  ❌ "Deployment" means FTP put
  ❌ "Rollback" means FTP put again, but the old file
  ❌ "Monitoring" means running a script that tails a log over FTPS
  ❌ The server's PHP version doesn't support ??
  ❌ SSH arrived after the war was over

FAQ (Mostly Honest)

Why didn’t you just set up SSH? They gave me SSH. After I’d already fixed everything. You work with the door they give you, even if they install a better door the day after you’re done.

You uploaded a web shell to production? A temporary one. Removed afterwards. When your only access is FTP and you need to understand the server environment now, you do what the situation demands.

Isn’t this just a glorified set of curl scripts? Yes. But glorified curl scripts with guardrails, unit tests, structured logging, and documentation. That’s the difference between a hack and a tool.

Why Python for the exporter? Because it needed to exist in an hour, not be beautiful.

You wrote unit tests during a production firefight? For the FTP helpers and the DB read-only guardrails. In the same week I was uploading PHP files one at a time over FTPS. Professionalism doesn’t have an off switch.

How much did the AI agent actually do? More than me. I built the toolbox - the FTP scripts, the DB CLI, the deploy workflow. The agent did the actual PHP archaeology: tracing SilverStripe framework code, identifying the broken filters, spotting the N+1 patterns, writing the patches. I’m not a PHP developer anymore; the agent is. I just gave it legs and pointed it at the fire. Though I did have to stop it from downloading 300 GB of customer photos onto my SSD, so the supervision wasn’t optional.

Why didn’t you just stop at the bot? Because the brain goblin doesn’t negotiate. The bot was the job. Everything after was compulsion dressed up as initiative.

Do you actually enjoy this kind of work? I know what I said. Don’t judge me.

The End

The site works now. The exports run. The filters filter. The admins can do their jobs without a 500 greeting them at every turn. I documented it, handed it back, and walked away. A clean break. Just like I promised.

The bottleneck was never intelligence - it was access. The AI agent knew more PHP than I’ve forgotten. It just couldn’t see the server. Every hour I spent building tooling - the log tailer, the DB CLI, the FTP deploy script - paid for itself ten times over. The fixes found themselves once the agent had eyes.

And yes, I deployed production fixes over FTP. The server’s PHP was so old my syntax was too modern for it. SSH arrived after the war was over.

I regret nothing.

Drafting Effective Tasks for AI Pair Programming

Thu, 09 Oct 2025 12:00:00 +0000

When I use ChatGPT or GitHub Copilot X (the Codex web UI) as my remote programming partner, the quality of the help I receive depends on the clarity of the brief I give it. The raw notes below evolved into a structured ritual that keeps the conversation focused on outcomes instead of diving into premature implementation details. This article distills that routine into a reusable, human-readable checklist.

Start with the feature, not the fix

Begin every engagement by writing down the feature you are trying to ship. Resist the temptation to anchor the discussion around a particular class, method, or algorithm; those are solutions. Instead, explain what the user should be able to do when you are finished. I keep the following guard rails in mind:

Frame the feature as a capability or improvement that someone can experience.
Describe current pain points or gaps in capability using plain language.
Clarify success with an explicit before/after statement: “Today it works like this…after the change it should work like that.”

This keeps the assistant centred on value, not code snippets.

Gather context before requesting changes

Once the feature outcome is clear, copy the entire brief into the Codex plan mode and ask it to examine the repository. Useful prompts include:

Here is a feature that I want to implement. Please analyze the codebase and collect information about the current implementation and what changes would be necessary.

Codex will typically respond with an outline of the affected areas, relevant files, and unanswered questions. Treat that response as a reconnaissance report. If anything feels off, refine your brief and rerun the request until the plan feels credible.

Close the loop inside ChatGPT

After Codex has explored the repo, paste every version of its response back into the ChatGPT conversation. Ask ChatGPT to consolidate those findings into a detailed plan. Automating the copy/paste with a browser userscript saves time here. The compiled plan should include:

The architectural approach and any new structures that need to be introduced.
How responsibilities will shift between existing components.
Open questions or risks that deserve further investigation.

Only when the plan satisfies you should you move on to execution.

Turn plans into actionable task lists

With the high-level plan in hand, return to Codex and request a task list with checkboxes that you can track as you implement. A simple follow-up prompt works:

Please create a .md file with the epic and checkboxes per task. Then go on and implement the first few tasks:
[PASTE TASKS HERE]

Saving this checklist in the repository keeps the scope visible and lets you revisit the remaining tasks after the initial iteration is complete.

Embrace parallel exploration

Finally, keep multiple tasks moving in parallel. Four active threads strike a balance between exploration and depth: enough variety to collect strong ideas, but not so many that you lose track of progress. As you iterate, capture learnings in the original ChatGPT conversation so your future self-and any collaborators-can reason about the trade-offs that shaped the work.

By ritualising the way you brief AI collaborators, you ensure that each session starts with clarity, produces tangible artefacts, and ends with a confident plan of attack.

Distilling Multiple AI Iterations into a Single Winning Pull Request

Thu, 09 Oct 2025 12:00:00 +0000

Generating several candidate implementations with Codex is easy; choosing the right one (and capturing the learnings) is the real art. These notes outline how I shepherd multiple AI-produced branches through a structured review so that the best ideas survive and the rest inform future work.

Expect variation and iterate deliberately

Codex output can swing wildly in quality. Instead of betting on a single response, request multiple iterations of the same feature. I typically generate four parallel branches and treat them as competing design ideas. Each run should:

Start from the same source branch to keep diffs comparable.
Follow the same high-level plan that ChatGPT produced earlier.
Produce a short changelog or markdown summary describing the approach.

Capture evidence for every branch

Before deciding which branch survives, gather objective evidence:

Clone or check out each pull request into its own worktree (for example, ../LiteDB.worktrees). Keeping them side-by-side makes comparison painless.
Record the repository status using gh pr list or similar tooling so you always know which branch maps to which Codex run.
Sync with the master plan stored in documentation (such as docs/Spatial-Revamp) to ensure every iteration addresses the same checklist of requirements.

This creates a paper trail that you can paste back into ChatGPT when it is time to synthesize.

Let ChatGPT be the reviewer

Once the branches exist, ask ChatGPT to evaluate them. Provide the command output, diffs, and any markdown summaries produced by Codex. Helpful prompts sound like:

Please check out all PRs in different worktrees and evaluate which PR is the best or what combination would be ideal. Mixing approaches is allowed-take the best of all worlds.

Encourage the model to grade each branch against the plan, call out missing pieces, and suggest how to blend strengths if necessary.

Aggregate the findings

When multiple evaluation rounds occur, collate the feedback before asking for a final verdict. A reliable pattern is:

Here is another set of findings. Please check whether they add anything new:
[PASTE SUMMARIES HERE]

ChatGPT can then produce a decision matrix such as:

Category	Source	Action
Correct normalization	#58	Keep as baseline
Descriptor metadata	#60 / #57	Add to persistence
Automatic backfill/index creation	#61	Integrate

This makes trade-offs explicit and tells you exactly what to keep, merge, or discard.

Finish with a curated merge

Armed with the matrix, return to Codex (or your local environment) and perform the final merge manually. The goal is not to auto-merge everything but to craft a single high-quality pull request that:

Preserves the strongest implementation choices.
Documents any deliberate omissions or deferred tasks.
Links back to the evaluation artefacts for future reference.

By treating each AI-generated branch as a hypothesis, you transform a noisy stream of drafts into an orderly, evidence-backed workflow that consistently ships better code.

Polyfill C#: Two Ways to Ship One Library Across Two Frameworks

Sun, 05 Oct 2025 12:00:00 +0000

TL;DR: You’re multi-targeting a library across netstandard2.0 and net8.0. Different APIs are available on each. You need the same public surface but different guts. Strategy 1: #if directives. Works, but scales like a dumpster fire. Strategy 2: partial classes with file exclusion in the .csproj. Cleaner, saner, and your future self won’t file a grievance.

The Situation

You have a library. It targets two frameworks - say, netstandard2.0 (because the world is full of legacy projects that aren’t going anywhere) and net8.0 (because you’d like to use APIs invented after 2017). Both targets need to expose the same public interface, but the implementation has to differ because the available APIs are completely different.

This is a solved problem. It’s solved in two ways. One of them is good.

Strategy 1: `#if` Directives (The “It’s Fine” Approach)

The classic. The familiar. The thing you reach for first and regret third.

#if NET8_0
    // net8 specific code - spans, modern goodness, joy
#else
    // netstandard specific code - string allocations, suffering
#endif

For a single method with a two-line difference? Perfectly fine. For an entire class where half the methods have different implementations, different helper types, and different using statements? You end up with a file that looks like a ransom note assembled from two different codebases. The syntax highlighting turns into abstract art. Code review becomes a puzzle game where the goal is figuring out which lines actually compile on which target.

It works. It always works. It just stops being pleasant somewhere around the fourth nested #if.

Strategy 2: Partial Classes + File Exclusion (The Grown-Up Approach)

This is the one. Split each class into three files: the shared surface, the .NET Core implementation, and the .NET Standard implementation. Then tell MSBuild which files belong to which target.

The .csproj:

<PropertyGroup>
  <TargetFrameworks>netstandard2.0;net8.0</TargetFrameworks>
</PropertyGroup>

<ItemGroup Condition="'$(TargetFramework)' == 'netstandard2.0'">
  <Compile Remove="**\*.NetCore.cs" />
  <None Remove="**\*.NetCore.cs" />
</ItemGroup>

<ItemGroup Condition="'$(TargetFramework)' == 'net8.0'">
  <Compile Remove="**\*.NetStd.cs" />
  <None Remove="**\*.NetStd.cs" />
</ItemGroup>

The shared interface - MyClass.cs:

public partial class MyClass
{
    public partial void CommonMethod();
}

The .NET 8 implementation - MyClass.NetCore.cs:

public partial class MyClass
{
    public partial void CommonMethod()
    {
        // The good implementation. Spans. Performance. Happiness.
    }
}

The .NET Standard fallback - MyClass.NetStd.cs:

public partial class MyClass
{
    public partial void CommonMethod()
    {
        // The "it works and that's enough" implementation.
    }
}

When building for net8.0, MSBuild excludes the *.NetStd.cs files entirely. When building for netstandard2.0, the *.NetCore.cs files disappear. Each target only sees the files it’s supposed to. No #if spaghetti. No guessing which code path is active. Just clean, separate files that each do one thing for one target.

Which One Should You Use?

Honestly? Start with #if. If the conditional block is small and isolated, it’s the right call - introducing three files for a two-line difference is overkill.

The moment you catch yourself scrolling past a wall of preprocessor directives trying to find where the netstandard path ends and the net8 path begins, switch to Strategy 2. Your code reviewers will thank you. Your IDE will thank you. The next person to touch this file - who is statistically likely to be you, three months from now, with no memory of why any of this exists - will thank you.

Approach Comparison:

  #if directives:    Quick to write. Painful to maintain. O(n²) regret scaling.
  Partial + exclude: More files. More setup. Zero ambiguity about what runs where.

Pick your pain. I pick the one with fewer #endifs.

A Thunderbird's Tale: Taming Google Calendar

Mon, 08 Sep 2025 12:00:00 +0000

TL;DR: Thunderbird won’t sync shared Google Calendars through the normal flow. The official guide is useless. iCal links work for your calendars but not shared ones. The fix: manually build a CalDAV URL from the calendar’s ID and paste it in. Google’s OAuth prompt appears, you sign in, and suddenly the calendar exists. The whole thing took 90 minutes of debugging and 30 seconds of actual solution.

I Just Want My Calendar

I like Thunderbird. This is a controversial opinion in some circles, but I stand by it. It handles email, it handles calendars, and it doesn’t try to upsell me on a premium tier every time I open it.

What it does not handle gracefully is syncing shared Google Calendars. And by “not gracefully” I mean “not at all, through any documented method, without manual intervention that Google and Mozilla apparently agreed to never tell anyone about.”

Attempt 1: The Official Guide

I started where any reasonable person would - Mozilla’s official support docs. The process seemed straightforward: go to ≡ > New Account > Calendar > On the Network > Next, enter your Google email, and let Thunderbird’s auto-discovery find your calendars.

I entered my email. Thunderbird thought about it for a moment. Then it found… nothing. No Google sign-in prompt. No calendar list. Just a blank screen and the quiet sound of my afternoon evaporating.

Dead end. Next.

Attempt 2: iCal Links (Partial Credit)

Google Calendar lets you grab a “Secret address in iCal format” for each calendar. It looks something like:

https://calendar.google.com/calendar/ical/your.email%40gmail.com/private-a1b2c3d4e5f6/basic.ics

I pasted this into Thunderbird and - it worked! For my own calendars. Perfect sync, no issues.

For the shared calendar I actually needed? Thunderbird rejected the iCal link like a bouncer checking IDs. Same format, same source, different result. Helpful.

So: personal calendars via iCal? Fine. Shared calendars via iCal? Absolutely not. This is the kind of inconsistency that makes you question whether software is a mature engineering discipline or an elaborate prank.

Attempt 3: The CalDAV Discovery

While poking around Thunderbird’s calendar settings for the calendars that did sync successfully, I noticed something interesting. Thunderbird wasn’t actually using the iCal URL I’d given it. Behind the scenes, it had quietly swapped in a CalDAV URL:

https://apidata.googleusercontent.com/caldav/v2/[calendar-id]%40group.calendar.google.com/events/

That’s… not documented anywhere obvious. Google doesn’t hand you this URL for shared calendars. Thunderbird doesn’t tell you it’s using it. It’s like finding out your car has a turbo button that nobody mentioned because it’s behind the glove compartment.

The Fix (It’s Embarrassingly Simple)

Once I knew the URL pattern, the rest was assembly:

Open Google Calendar settings for the shared calendar.
Copy the public address in iCal format - it contains a long calendar ID like a1b2c3d4e5@group.calendar.google.com.
Extract that calendar ID.
Slot it into the CalDAV URL template: https://apidata.googleusercontent.com/caldav/v2/[CALENDAR-ID]/events/
Paste the constructed URL into Thunderbird’s calendar location field.

The Google OAuth screen appeared. I signed in. The shared calendar materialized in Thunderbird like it had been there all along, casually pretending the last 90 minutes hadn’t happened.

Frustration-to-Fix Ratio: [██████████] 10/1

minutes of debugging
seconds of actual solution
lines of documentation that would have prevented this

Why This Is Annoying

The information to make this work exists in the system. Thunderbird knows the CalDAV pattern - it uses it internally. Google exposes the calendar ID - it’s right there in the iCal URL. Neither party connects the dots for the user. It’s like two people each holding half a map and refusing to stand next to each other.

If you’re hitting this same wall - shared Google Calendar, Thunderbird, auto-discovery failing, iCal links rejected - this is the fix. Build the CalDAV URL yourself, paste it in, and move on with your life.

You’re welcome. I’m going to go close 14 browser tabs.

The Problem: ZeroTier Stops Working

Tue, 11 Apr 2023 12:00:00 +0000

ZeroTier has been a reliable solution for establishing VPN connections between devices, even when they are located behind NAT. However, after installing ZeroTier on my brother’s Raspberry Pi home server, I encountered a baffling issue: it would work fine for the first hour or so, and then lose connection from outside. The only way to re-establish the connection was to reboot the Raspberry Pi.

This problem was further compounded by the fact that if I had an ongoing file copy operation, I could still connect to the Raspberry Pi from the specific machine involved in the transfer. But if I switched to another device, like my laptop, I couldn’t connect at all. The sudo zerotier-cli info command showed the status as “OFFLINE.”

The Investigation: Searching for the Root Cause

After scouring the internet for answers and receiving advice from fellow Redditors, I opted for a workaround instead of pinpointing the root cause. This involved creating a script that checks ZeroTier’s status periodically and restarts the service if it’s offline.

Here’s the script I used (check_zerotier.sh):

#!/bin/bash

# Check if the response from "zerotier-cli status" contains "OFFLINE"
status=$(/var/lib/zerotier-one/zerotier-cli status)

if [[ $status == *"OFFLINE"* ]]; then
    echo "OFFLINE! Restarting zerotier-one"
    # If it does contain "OFFLINE", restart zerotier with the command "service zerotier-one restart"
    /usr/sbin/service zerotier-one restart
fi

if [[ $status == *"ONLINE"* ]]; then
    echo "ONLINE! Doing nothing"
fi

To automate the process, I added an entry in the crontab file to run the script every 10 minutes:

*/10 * * * * /usr/scripts/check_zerotier.sh 2>&1 | /usr/scripts/formatLog.sh 2>&1 >> /var/log/zt_check.log 2>&1

I also created a formatLog.sh script to format the log output:

#!/bin/bash

while read line; do
  current_time=$(date +"[%Y-%m-%d %H:%M:%S]")
  echo "$current_time $line"
done

The Workaround: A Reliable Solution

By implementing this workaround, I managed to bypass the issue and ensure that ZeroTier remains functional on my brother’s Raspberry Pi home server. While I didn’t identify the root cause of the problem, this solution has been effective in keeping the connection stable and preventing any disruptions.

Conclusion

Sometimes, finding the root cause of an issue can be like searching for a needle in a haystack. In cases like these, it’s essential to focus on finding a reliable workaround that can keep things running smoothly. In my case, a simple script and a cron job were enough to prevent ZeroTier from going offline and maintain a stable connection. I never pinned down the underlying cause (and at the time I cared more about keeping the box reachable than doing deep archaeology), but this watchdog approach kept the connection stable.