Rendered at 22:00:05 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
simonw 1 days ago [-]
> We funded the open-source SQLite VFS shim that helped isolate the race condition almost immediately, and will help track down similar bugs in the future.
Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.
buggymcbugfix 1 days ago [-]
Not only are they funding open source, they are actively allowing users to host their own control plane via headscale which is a libre implementation of the tailscale control protocol and developed by an engineer who works for Tailscale by day. This instantly made me trust and like them, even if at first I was cautious because I naturally mistrust anything that gets a lot of hype. I've been running headscale on NixOS, hosting my own tailnet since last year and everything just works out of the box. Tailscale lives up to the hype! <3
chillaranand 17 hours ago [-]
Important features like App Connectors won't work with HeadScale.
Unfortunately, HeadScale is not a drop in replacement for TailScale. Most people don't realize this until they start self hosting.
Due to this, I had to migrate from Tailscale to NetBird, which is completely open source.
Is this because HS hasn't implemented it yet (which is on the community), or because Tailscale does something to prevent them from working specifically?
serf 8 hours ago [-]
it's a proposed and planned feature by the looks of it :
The only nit on my end is that you have to configure a few settings on client and server to avoid sending diagnostics to TS.
For macOS, you need to build the open source tailscale client. Or create file in esoteric file location to opt out using App Store.
For iOS I don’t even think it’s possible to opt out.
bradfitz 17 hours ago [-]
Oh? I thought we added that option on iOS recently. I'll go check.
binhex 1 days ago [-]
Yeah, tailscale seems to have leadership with their head on right, I agree with the way they handle a lot of things.
saghm 1 days ago [-]
Yeah, this part also stuck out to me:
> Because this wouldn’t be a quick or easy fix, we reached out to the SQLite developers for a professional support contract. This was a great decision. It gave us direct access to their deep expertise and experience, and we had many detailed technical conversations about our architecture and our incidents.
They were willing to pay to get help solving the problem, and then pay again to make sure that the problem is easier to avoid in the future! That kind of long-term thinking seems pretty rare nowadays...
jjordan 1 days ago [-]
Probably the most refreshing thing I've read in a while. Glad to support them moving forward if this is indeed their modus operandi.
LoganDark 1 days ago [-]
Tailscale is the best. It's infinitely better than Hamachi, ZeroTier, etc. My only gripe is that they have some really weird SSO requirements like GitHub, etc. and then that provider becomes a permanent part of your identity.
saghm 1 days ago [-]
I've been a fan of Tailscale since encountering it for the first time at a previous job at a small startup. Someone asked if anyone had a Linux machine when we were all testing out something, and I mentioned I had a personal Linux desktop but wasn't sure how to connect it to the VPN for access, and it turned out that the solution was just literally running two commands in the terminal after installing tailscale from the repos. Compared to my first job where connecting to the VPN from Linux required hours of mucking around with openswan (or was it strongswan? so many swans...) and trial and error with various config files, it was unfathomably straightforward.
inigyou 20 hours ago [-]
That strongswan thing is the kind of design HN praises about open protocols by IRC (just the other day...) but in practice is so flexible it can't keep itself upright and it's unusable in practice.
Meanwhile tailscale or wireguard, by being actually opinionated, avoids needing much configuration at both ends.
gmac 13 hours ago [-]
Yes — configuring strongSwan as a bog-standard VPN server was so hard to fathom I made GitHub repo for it [1]. To be fair, some of the complexity comes from OS support that seems specifically designed to make secure setups difficult, presumably at the behest of various Three Letter Agencies.
I have now mostly switched to Wireguard for this, which is much more sane [2].
With a desktop its usually possible from the network setting GUI? Worked like that last time I needed to use a VPN for access to a corporate network.
throwway120385 1 days ago [-]
Some of the NetworkManager VPN plugins have weird side-effects. Like the one for OpenVPN force-disables split tunnel configurations and there's no way to turn that off, meaning that all traffic routes through the VPN even if you'd rather only send stuff for certain destination addresses through.
saghm 23 hours ago [-]
Based on the experience that I (and other coworkers, including on other distros) had, whatever configurations our VPN needed did not seem to work out of the box on network manager.
miki123211 9 hours ago [-]
A lot of this is just weird Linux on the desktop weirdness.
On Windows and Mac, VPN config is usually just installing a client, signing in, and then it all basically works. Sometimes you don't even need the client and can set it up in settings.
artimaeis 1 days ago [-]
Just FYI - with Tailscale you can switch SSO providers by putting in a support ticket. I did it last year and it was a breeze.
LoganDark 1 days ago [-]
Oh, thank you! I might do that.
felixg3 1 days ago [-]
I think their reasoning on not being an identify provider but acting solely downstream is very clever.
jdjdjebeb 1 days ago [-]
Y? What's wrong with providing username/password authentication
semiquaver 1 days ago [-]
Being an identity provider for anything important is the freaking worst. Exposes you to a million problems. You need human support for login problems and lost MFA tokens, and you are an attack magnet.
miki123211 9 hours ago [-]
Which is why you want magic links.
Don't be the identity provider, have the email host be the identity provider (which it is anyway if you have a forgot password prompt).
Agreed 100% that nobody should still be using passwords in 2026 though.
l72 8 hours ago [-]
As a user, I really don't care for magic links. The whole, start the log-in process, switch context, wait for email (sometimes up to a minute), click on it, have it open a new tab in a different window than where I started is just a pain. I feel like I spend half my day logging in to services these days.
The only time I like magic links is for services where I am "not really a user". For example, an appointment reminder for my doctor where I need to validate my insurance. Great, send me an email 24-hours before with the reminder and a magic link, as I don't want to think about an account there.
efreak 3 hours ago [-]
I detest magic links. I have probably 9-10 accounts with a certain service that uses magic links (I mostly use them to get an API key or download a file every few months and never visit directly otherwise) because
1. the only login method they support is magic link and login with Google/Facebook. There's no password field so my password manager can't fill out my login details for me.
2. I don't have a Facebook account, and I don't keep my Google account signed in. Even if I did, I don't want them using these as my primary email address. (I use a catch-all and don't give out the primary address; I give each service it's own address)
3. A bunch of newsletters and forums (hackaday, etc) mention them by name every few weeks, often in the subject field, making it hard to search for. They don't send me any newsletters themselves, and I don't participate in their forums so there's nothing from them directly. They can't do anything about this, but it makes things harder.
4. If you enter a new email address that they've never seen before, the process exactly the same until after you click the magic link you received. They don't tell you they've never seen it before until you verify the address.
5. As such, if I can't recall the exact address I used to sign in, I automatically create a new account and don't find out until after I click the magic link and I'm asked to enter a new username or use the existing username they've already created. Proceed to settings and create a new API key for a new account instead of just adding one to my existing account.
This is an extreme case, yes, but I have similar issues with several services that only support magic links and GitHub login, and it's hard to configure services to use a new catch-all address when they harvest my address from GitHub login and won't let me change it. Magic links are a great tool, but they should _never_ be the only option. They should supplement existing login methods. Password managers exist for a reason. Several services that I have since stopped using for this reason have in the past decided that they should start sending emails to the addresses they harvested from login with other service. Two services have gotten my magic link address stuck in their system because it's not associated with an account (the account has been deleted) and thus I can't manage subscriptions.
rainsford 23 hours ago [-]
One maybe non-obvious benefit I see in that from the individual Tailscale user perspective is that only having SSO rather than username/password keep the security of my login in the same bin as the larger customers where Tailscale makes their actual money.
A username/password would be perfectly fine for my use-case as an individual user using Tailscale for my homelab and personal devices. But I suspect the majority of real paying customers don't want a separate authentication flow just for Tailscale and would rather use the SSO they already have. I have confidence Tailscale wouldn't half-ass usernames and passwords is they offered it even if it was mostly just for non-paying hobbyist users, but I'm also sort of glad I get to bucket my account security risk with the people paying their bills.
LoganDark 1 days ago [-]
You become responsible for keeping those credentials secure, among a whole host of other problems. (Abuse like credential stuffing (more), data breaches (more), account farms (more), and so on.)
altairprime 24 hours ago [-]
How are the requirements weird? They support any OIDC endpoint whether your own or a vendor’s, and, while uninteresting to HN folks, they also now support passkeys for having no SSO provider at all.
freeAgent 1 days ago [-]
Yeah, I have my Tailscale tied to my Apple account, which just feels weird. I can add a Passkey account to my Tailnet and make it manager, etc., so that’s what I’ve done. The owner is my Apple account, but I actually do everything admin-wise with a Passkey account.
LoganDark 1 days ago [-]
Oh I had no idea they supported Apple. Maybe they didn't back when I signed up? I'm seemingly stuck with GitHub forever now though.
shye 19 hours ago [-]
Should be possible to change, just not automated. You have to reach out to support.
throwaway27448 1 days ago [-]
This isn't exactly new; the other model is to just hire them directly. Unfortunately this has not generally led to open source outside of the enterprise world getting similar funding, but it is what it is.
x0x0 1 days ago [-]
It's common for databases.
This is Percona's business model. They employ core pg/mysql developers and you can buy a support package from them. Same for enterprisedb. Pretty reasonably priced packages (like maybe $10k-ish/core IIRC) get you 24x7 support. I've only had to escalate issues once but inside 10 hours we got a pg core committer to debug some very strange vacuum behavior.
SchemaLoad 22 hours ago [-]
They could have just spent $300,000 in tokens to AI slop rewrite SQLite in Rust.
edoceo 20 hours ago [-]
That would be lame. Instead they spent less and improved the world for everyone who used SQLite (which is really a lot of people)
AtlasBarfed 1 days ago [-]
I wish they'd buy the carcass of Keybase from Zoom.
It seems very similar to Tailscale: immense utility from a free-tier product for the general public, which leads to trust and a large enterprise market.
Corrado 1 days ago [-]
Wow, that’s a great idea! I loved keybase back in the day and really thought it solved a real world problem in a unique and interesting way. Then Zoom bought it (for some reason) and left it to rot.
snorremd 1 days ago [-]
You might already know and used "for some reason" as sarcasm. For those that do not: Zoom did the usual aqui-hire. They needed to show their enterprise customers they were taking e2e encryption seriously, so they bought out the guys who did exactly that kind of encryption with Keybase.
A shame really, because Keybase was kind of unique in the way the facilitated social proof of identity tied to cryptographic keys. A shame though that they never found a really sustainable business model. When they turned to crypto currency towards the end of their independent run I knew it probably wouldn't last.
fodkodrasz 12 hours ago [-]
the UX was so terrible that it was inevitable it won't stick. It was clunky and alien on every productivity platform, I attempted to work in a team leveraging it, and it was painful on windows and macos alike.
I just deleted it as I noticed it was responsible of about 30% of active cpu (and battery drain) usage on my mac, while being logged out due to inactivity. Honestly: I never particularly liked it, and I could not feel the concept especially tempting, just went with the team.
Avery is one of the few people I have enough respect for to look up to.
rcleveng 17 hours ago [-]
Avery is amazing, reading his writings was one of the highlights of my time at Google. There were many talented and articulate people there, the level of discourse used to be amazing.
devmor 1 days ago [-]
Their CEO is a very nice and personable guy too. Has given me and others advice on random topics of his interest with no nonsense plenty of times.
xyst 21 hours ago [-]
At least for now. All it takes is one greedy executive decision to stop supporting headscale. Given it’s an American company, it’s entirely possible.
jkaplowitz 12 hours ago [-]
Tailscale is Canadian, headquartered in Toronto.
garganzol 1 days ago [-]
In real life, an imaginary "leadership" quickly evaporates upon the first hurdle of any kind until proven otherwise by real actions. If I was in your shoes, I would be very careful with blanket statements like these.
fragmede 17 hours ago [-]
This would be the "proven otherwise by real actions" part. They've been around since 2017 and have had plenty of time to hit various hurdles. Past performance is no guarantee of future performance and it is within the realm of possibility for them to go of change course like Elon did. But who are you? Are your Internet comments some proenoucement of truth that can never ever be taken back? So you said something that, years later, turns out to be wrong. So what?
Reading the article, it sounds more like they funded this by buying a SQLite support contract, and the SQLite developers created this in the process of helping them track down the issue.
EastSmith 1 days ago [-]
Started using them like 2 weeks back, happy to see how they work.
2afTq 8 hours ago [-]
New? The sqlite VSF shim is much older. Nice spin.
> A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used.
This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQLite page[0] outline that it can only ever occur if there are multiple connections open, so the writer and the checkpointer must have been on different threads.
> The bug only affects databases in WAL mode when there are two or more database connections open on the same file, in separate threads or processes
To be honest, I'm surprised that someone using SQLite would try to access it directly from multiple threads or processes without fear of data racing.
bhaak 12 hours ago [-]
> Multiple processes can have the same database open at the same time. Multiple processes can be doing a SELECT at the same time. But only one process can be making changes to the database at any moment in time, however.
One writer, multiple readers is a specifically supported way of using SQLite.
Why should you be worried if it is used as designed?
petcat 9 hours ago [-]
> Why should you be worried if it is used as designed?
Well this whole article is about a company discovering a catastrophic corruption bug even though they were using it as designed.
I think the lesson is that if you're ever actually worried about concurrency then just don't use sqlite. We can see here that concurrency is hard and the bugs are old and deep.
chuckadams 7 hours ago [-]
I guarantee that any hand-rolled replacement will have more and worse bugs.
21 hours ago [-]
maitrungduc 1 days ago [-]
[flagged]
1 days ago [-]
pseudohadamard 19 hours ago [-]
I don't know enough about the scale of Tailscale's operations to comment strongly on this, but if they're fairly significant shouldn't that have read "is exactly how MariaDB is meant to be used" or "exactly how Postgres is meant to be used"? SQLite has a "lite" in the name for a reason, but it's often pushed into places where it's being asked to do things it was never really designed for.
geocar 15 hours ago [-]
> SQLite has a "lite" in the name for a reason
I would not think of SQLite as "lite" anything. It's SQL In The Executable.
It has a better security and data-durability track record than both Postgres and MySQL, and often beats them in the sorts of things applications do with databases:
Seems like it absolutely is "designed" for this use case.
jlokier 12 hours ago [-]
> SQLite has a "lite" in the name for a reason
It"s actually SQL "ite" as in rocks, minerals and fossils. Their version control system is called "Fossil".
zbentley 18 hours ago [-]
This particular bug doesn’t seem to arise from SQLite’s “lite” nature. It’s a TOCTOU inside the DB when applying WAL segments in a checkpoint, which is a pattern used in extremely similar ways by Postgres and MySQL. They don’t seem to have similar bugs, but I don’t think there’s any reason to believe that this is due to their being client/server rather than coordinated-file databases.
anitil 18 hours ago [-]
It says a lot about sqlite that a bug becomes front-page news on HN. I'm impressed that Tailscale took this seriously enough to engage with a commercial support contract. I'd love to work for a company that cared so much about correctness.
Lio 11 hours ago [-]
Indeed, SQLite has got to be one of the best tested pieces of software with famously 100% test coverage.
If you already know the bug is caused by a race between writing and checkpointing then it's easy. I'm unsure how the linked article makes your point at all though.
andai 1 days ago [-]
SQLite: 92 million lines of tests
Dijkstra: Tests can only prove the presence of bugs, never their absence!
otterley 1 days ago [-]
Everyone knows that tests don't prevent all bugs. But they are very good at preventing known bugs from recurring in the future.
layer8 11 hours ago [-]
Everyone doesn’t seem to know that, because tests are often cited as a way to ensure that AI-generated code is correct.
andai 5 hours ago [-]
I was very excited about formal proofs, which are now very cheap to produce, in service of validating AI generated code.
But I had a funny experience recently where an agent implemented an entire feature completely wrong (exactly backwards, actually, in a way that defeated the purpose, introduced security issues etc.).
It happily supplied tests for the new functionality, and all the tests passed.
What I realized was, even formal verification wouldn't have helped here -- it would have just written a mathematical proof that the incorrect functionality was correctly implemented!
So there's a gap here, where first, the human's intention needs to be formally specified (by the human, or at least the human needs to be able and willing to verify it), and then the slopswarm can hack away at it...
layer8 1 hours ago [-]
> the human's intention needs to be formally specified
Yes, that has always been the case with formal proofs. Even a mathematical proof with Lean is useless if the proven formal statement doesn’t match what the mathematician meant. And that’s why humans working with formal languages won’t be going away. Natural language is imprecise by nature, even when we have precise ideas in mind. In that case, it’s often better to specify what we mean in a formal language than in natural language.
otterley 2 hours ago [-]
> first, the human's intention needs to be formally specified
This never changed with AI; in fact, I think it made this need more visible than it ever had been before. You can't get away with not being able to describe in detail what you want. As with working with humans, any ambiguities will be interpreted, and not always in the way you hoped.
ameliaquining 1 days ago [-]
I admit to curiosity as to whether static analysis could have caught this. E.g., Rust's type system (yeah yeah I know) catches all data races, unless they originate in unsafe code, which this one might or might not have; a hypothetical Rust SQLite would probably need a lot of unsafe (https://github.com/tursodatabase/turso has 556 unsafe blocks in the core), and I don't have a sense of whether the particular part that contained this bug would be included in that.
Interesting. I don't think this kind of technique could realistically have caught the bug in advance, though; you're not going to model the entire codebase in TLA+, so the investigation in that post modeled just the parts that turned out in hindsight to be relevant.
fragmede 22 hours ago [-]
> you're not going to model the entire codebase in TLA+
Not by hand in the olden days but with AI to help with it, why not?
ameliaquining 19 hours ago [-]
LLMs are not at the point where they can reliably complete this kind of task without making mistakes. The model checker can tell you whether the formal design specification contradicts itself, but it can't tell you whether it matches the implementation, nor whether it matches the programmers' intentions. And it's too much for humans to review by hand.
This is importantly different from formal code verification, in which the specification that needs to match the programmers' intentions is small enough to review, while the bulk of the LLM's output is proof tactics and such, which don't need human review because the verifier guarantees* that they're right.
I suppose you could just tell Fable "hey try using TLA+ to look for bugs in this code" and see whether it finds any.
That's dynamic, not static; it doesn't help much with a race that's hard to trigger.
IsTom 1 days ago [-]
It sounds like it was a bug caused by pushing on performance/safety ratio, it could easily have been avoided by adding enough mutexes in the code, but for performance reasons (I assume) they don't do that. To write it like that in Rust you'd have to use unsafe blocks and then you don't gain much from the typesystem in this case.
andai 1 days ago [-]
>Rust's type system (yeah yeah I know)
What do you know?
1over137 23 hours ago [-]
Presumably he is referring to rust fanboys' tiresome criticisms of everything C.
0x457 1 days ago [-]
It can prove absence of specific bugs though.
inigyou 22 hours ago [-]
no it can't. what if this bug is still there but the timing window is now one CPU instruction?
aw1621107 21 hours ago [-]
> what if this bug is still there but the timing window is now one CPU instruction?
I think a model checker or something similar that can exhaustively cover a search space would suffice, though at that point I think the boundary between "test" and "formal verification" becomes somewhat fuzzy.
ashton314 18 hours ago [-]
> exhaustively cover a search space
This is the gotcha: in stateful environments you'll have state-space explosion so big you'll have to come up with some compromising measure of "equivalent". You're right that that would constitute a proof by exhaustion of correctness. Unfortunately in practice it's entirely impractical.
aw1621107 16 hours ago [-]
I was thinking that CDSChecker [0]-like tools like Rust's Loom [1] might be able to prune the write+checkpoint search space enough to make exhaustive checking feasible, though I'm far from knowledgeable enough about the implementation to say with any certainty how plausible my hunch is.
I mean, exhaustive tests are one way you can prove something.
inigyou 13 hours ago [-]
You can't even exhaustively test every 64 bit integer value.
aw1621107 10 hours ago [-]
Just because you can't exhaustively test one thing doesn't mean you can't exhaustively test anything.
IX-103 8 hours ago [-]
Not being able to exhaustively test simple problem absolutely means you can't exhaustively test massively more complex problem. For a monotonically increasing function f, which I think we can agree a function mapping from the size of the input set to the runtime of an exhaustive execution of all elements in the set qualifies as, if |small| < |large| then f(small) < f(large). Therefore if f(small) is too big then f(large) is definitely too big.
aw1621107 4 hours ago [-]
> Not being able to exhaustively test simple problem absolutely means you can't exhaustively test massively more complex problem.
"Simple"/"complex" doesn't necessarily imply "small"/"large", respectively. Testing whether the Collatz conjecture holds is "simple", but its search space is (potentially) infinitely large. Testing correctness of concurrent data structures is (probably) a more "complex" problem, and yet it's still possible to perform exhaustive tests of their functionality [0].
In the end, the practicality of exhausting the 64-bit integer search space is irrelevant since it says nothing about the size of the write + checkpoint search space. I could just as easily say that it's easy to exhaust the 16-bit integer search space, but that doesn't mean exhaustively testing for this particular bug is easy.
Jesus Christ. I have this comment saved, but it gets me every time.
bch 1 days ago [-]
This was really, really interesting - what a triumphant adventure.
A few (very, very, very pedantic) things that stood out:
> We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky).
(Emphasis mine) - it would be "risky", not "potentially risky" - then the "calculated risk period" starts and it's "potentially problematic".
In the SQLite report[0] (11.2) I wish they downplayed this less - a mention of the rarity, then technical details - I'm friendly with a few of the devs/previous-devs, have the utmost respect for their skill and accomplishments (and by extension, faith that the developers I do not personally interact with are also excellent), appreciation and fondness for the huge accomplishment that is SQLite, and on and on... this is world-class work. Maybe section 11.2 wasn't really aimed at me, or I'm too critical. To be fair to all involved, what a minor quibble for such an interesting problem/fix. I hope my comment isn't a fly in the ointment.
Last bugfix point[1] - ugh. What a sinking feeling that must've been to deploy a fix then be flooded with not-green - and a lesson[2] against smuggling other changes in a changeset "just because we're already here"? Happy it turned out non-catastrophic, but did result in a rare (not remembering other instances of top of head) recall[3] from SQLite. That it was throwing errors at the same time SQLite and Tailscale were testing the other WAL-issue bug must've upset some stomachs for a moment.
Very nice article, and I appreciate SQLite's explanation of the bug too. And how extremely cool Tailscale appears to have been about it (paying for the VFS shim, etc.).
I'd have liked to have heard more about the decision to checkpoint so frequently that put them on this path though. Presumably that's to keep the WAL tiny for very fast recovery. Trying to mitigate some of the deleterious effects of inserting a DBMS into your network layer, I suppose? Tricky stuff. Wonder how that compares to typical etcd snapshot frequencies too.
buggymcbugfix 1 days ago [-]
Agreed, I was also wondering about this. Maybe the aggressive checkpointing was for preventing WAL-overflow?
nujabe 19 hours ago [-]
Was wondering the same, doesn’t mention if they tried checkpointing less frequently
bobtheborg 1 days ago [-]
Great read. So glad they took the time to tell this story. (And glad they, as a for profit corporation, took out a support contract with SQLite. I hope they continue to do so even though this problem is resolved.)
sandeepkd 1 days ago [-]
>In our control plane, we take manual control of the checkpoint process so we can run fast and consistent backups.
> running boring technology in a non-standard way is a risk.
It was a good read and reminder that the industry is loosing experts gradually. I am not a DBA and yet I have heard about this behavior at least couple times in the past as something to avoid. Its just one of those things which didnt get a chance to be documented cause experts avoided it and regulars didn't get into
inigyou 22 hours ago [-]
I've never heard of any reliability reason you shouldn't run checkpoints whenever you want - only performance reasons. Can you elaborate?
Backing up sqlite by copying the file (e.g. rsync) while it's open is a surefire way to eventually get corruption caused by a race condition, but it seems like tailscale wasn't doing that. They were probably using the proper sqlite backup API.
But you don't need checkpoints for consistency and I think the backup API will not copy both the old and new versions of pages just because they're in the WAL, in other words I think checkpointing then backing up should give you the same pages as backing up without checkpointing. So the whole thing seems unnecessary.
sandeepkd 8 hours ago [-]
My generalist reading of this (which is applicable in lot of cases) is that a dedicated flow is handling the house keeping part of the job, aka resetting the checkpoint after writing the WAL to disk. These one off house keepers are common in a lot of softwares and they expect to work alone, they are tested to work alone. They always have a set of ritual, rules and order in which they do all the things. Now if some other thread takes off some of those jobs then they break this routine for the house keeping job and this new thread/person may not always know what else has to be done before and after this one particular job for the sake of completeness.
On second question of why the tailscale developers did it, its possibly for the same reason why they invested this much into debugging this issue. Some one believed the current behavior did not fit into their architecture, they want to be more performant and take control over things. A big part of me considers this is a required exercise to try, grow and learn. The only thing they could have for improvement would be to have these old hands on architect kind of folks on their team who might have hinted/pointed them to the problem a long before. Challenge/chances are that these older folks would have even stopped them from going in this direction in the design phase itself.
sandeepkd 1 days ago [-]
In other words there exists a concept of HOT and COLD backups for this reason only.
arendtio 54 minutes ago [-]
Everybody here knows that SQLite is not production-ready and does not scale.
This is just proof that it can't be used in real-world applications.
SCNR
throw0101a 1 days ago [-]
See perhaps recent video "Reliability Lessons From SQLite - Richard Hipp | SSW 2026":
> Abstract: SQLite is a C-language library that implements a self-contained, in-process relational database engine supporting full-featured SQL, an advanced query planner, and ACID transactions. By many estimates, SQLite is the most widely used software library in the world today.
> Over its 26-year history, SQLite has gained a reputation as software that "just works". This talk goes over the design choices and development practices that have, at least in the opinion of the lead developer, resulted in that reputation.
Great writeup, and it was great to see them step in an pay the developers of SQLite to help them fix the bug. I get tired of corporations asking open source authors to fix problems that affect the corporation for free. And while I'm sure it was frustrating for folks to have these outages, I find such puzzles pretty fun to get to the bottom of.
stillpointlab 17 hours ago [-]
It gives me a warm feeling when companies invest in open source support in this way. Helping great projects get even better is somehow better than releasing yet another project.
myshapeprotocol 1 days ago [-]
Tracking down a 16-year-old edge case in database internals is peak engineering perseverance. Incredible deep dive.
jbs789 1 days ago [-]
As a simple user of SQLite, I think this level of debugging is incredible and appreciate being a beneficiary of the ecosystem and hard work of others. Thank you!
w10-1 1 days ago [-]
The irony is that the SQLite developers get a support contract iff someone runs off the path in anger and finds an ancient bug. But perhaps that's part of what make it a quality team: devotion thriving without adverse incentives.
rmunn 20 hours ago [-]
While you might be correct, I wouldn't necessarily assume the "and only if" part of your statement. There might be companies that choose to proactively purchase support contracts. And there are companies that have paid $150K/year for https://sqlite.org/consortium.html access. Which means, among other things, that they get first priority for any needs they have:
> Consortium members have the guaranteed, undivided attention of the SQLite developers for 23 staff-days per year and for as much additional time above and beyond that amount that the core developers have available. There are no arbitrary limits on contact time. The consortium will never be over-subscribed. New SQLite developers will be recruited and trained as necessary to cover the 23 day/year support commitment.
The SQLite home page lists five companies that have paid for consortium access. I can easily imagine that there are more who don't want to pay $150K/year but would pay $1.5k/year, proactively, to get "private, expert email advice from the developers of SQLite" when they need it.
catapart 23 hours ago [-]
As others have said: great article! I did find myself wanting them to get to the point, but once they started describing the bug and the fix, it was very satisfying. I'm very happy there are companies out there on the frontiers of functionality not only funding fixes and debugging measures, but taking the time to write up the details so we can all benefit.
Tailscale just moved up in my priorities list. Was going to host my next website with hostinger, but now I'm going to at least try to run a personal server with tailscale to make it public. I might not be able to figure it all out, and may end up going with the VPS route, but this gave me some appreciation for the company that makes me willing to try the less familiar method.
inigyou 20 hours ago [-]
I don't think Tailscale can do that though? It can't open a port from the internet to your private network.
What a brutal bug. I'd never entertain a that bug in SQLite could be causing problems in code I wrote.
hedgehog 1 days ago [-]
The fun part is unless your app is pretty high traffic you might see the bug only once, or twice, and never have a reasonable way to even get close to a fix or know what your real exposure is.
sgarland 22 hours ago [-]
That is the correct mentality, but sometimes, you’re wrong. I found (not entirely true; someone else [0] found it first, but my report from my company got traction [1] on it) a weird bug in ProxySQL several months ago involving mirroring and fast routing, where if you had a configuration that was illogical from the standpoint of documented behavior, it would duplicate queries, which led to super fun times for writes.
I spent hours empirically trying various scenarios before concluding that no, it was a ProxySQL bug.
I mean, ProxySQL and SQLite are at very different tiers of reliability. I encountered multiple unreported data-corrupting (and some resource exhausting/connection mis-pinning) bugs in ProxySQL within a few months of using it for the first time, and I wasn’t using it for anything particularly complex or advanced—just a basic connection pool, no failover or caching/rewriting/replica awareness, but a lot of frontends and QPS.
deepsun 22 hours ago [-]
Normal code has 50% to 90% ratio of code coverage by unit-tests. Dynamic-typed languages (Python, Ruby) usually require more, like 100% - 120%.
SQLite has 59,000% ratio [1]
Yet it didn't help for a bug to left unnoticed for 16 years :( I don't know what we can do for the industry. I doubt one can formally verify a project like SQLite, and keep it maintainable.
One might be able to formally verify the sqlite pager layer, where this problem occurred. It's just writing pages to disk and reading pages from disk, and providing ACID guarantees.
inigyou 22 hours ago [-]
This reaffirms my belief that SQLite is not well suited for systems with significant concurrency. It replaces fopen, not postgres. Although this corruption is a rare bug and sqlite is usually extremely stable, it's usually not worth it from a performance and features standpoint either.
Here they were trying to do a backup by forcing a checkpoint and then copying the file. Systems like postgres let you do online continuous backups.
zbentley 18 hours ago [-]
Eeeh, this particular bug was a race when applying WAL to the data files in a checkpoint. Postgres’s checkpointing is theoretically just as vulnerable to this class of bugs as SQLite, though it doesn’t seem to have equivalent issues today.
Online backups/replicas are nice until you’re charged for network traffic or have to recreate the replica from scratch, at which point the initial-restore-then-hook-up-the-WAL-stream dance is prone to all sorts of racy issues. If you’re lucky enough to have only a single process talking to the DB, SQLite seems like a nice way to sidestep that complexity while keeping a simple backup story.
Heck, this is basically the Redis model: a single process/thread coordinates all access to the data, and occasionally forks off a background job to snapshot the state somewhere. From that perspective, the Tailscale controller binary is a database; SQLite is just the data file format.
q3k 21 hours ago [-]
Yeah, I also feel 'just use SQLite [no matter what]' is just the pendulum swinging hard after the 'just use mongodb [no matter what]' of yesteryear.
It's so sometimes just performative. I remember when Tailscale had a similar performative approach with 'just use a JSON file on disk'. Then etcd. Then SQLite. Like sure, you can keep picking the absolite mininum technology for your needs and then change it every couple of years... Or you could just immediately go with a solid Postgres (or Postgres-like setup, eg. yugabyte) and save yourself a bunch of faffing about with weird solutions and migrating between them. But I guess that doesn't drive engagement on your blog.
sealeck 20 hours ago [-]
Postgres has also had pretty serious bugs, eg fsyncgate
21 hours ago [-]
21 hours ago [-]
punnerud 15 hours ago [-]
Checked and this 100% compatible SQLite3 database (also C-API) did not contain the bug:
https://github.com/punnerud/mpedb
(Disclaimer: My own project)
And this statement is wrong in the article:
“ Because SQLite is a single-writer database with serialisable transactions, our transaction history was completely linear and deterministic. (This wouldn’t be true in a multi-writer database like Postgres or MySQL.)”
Actually possible in mpedb to replay multi-writer, and actually better than SQLite3. Try to reply now() in a statement, that is not deterministic in SQLite but is in MPEdb.
Wait, it's AI? AI probably has information about this bug by now, since it was discovered and fixed in March.
wwilson 20 hours ago [-]
Nope, it's a generic PBT harness. The human who built it knew about the bug, but the workload was specifically designed not to specifically target this particular bug, and may have found several others as well (we are in the process of reporting these upstream).
minimaltom 1 days ago [-]
I'm equal parts intrigued and skeptical- I guess if the prompt doesn't lead on there is a bug there then I'm impressed.
wwilson 1 days ago [-]
Stay tuned!
truetraveller 1 days ago [-]
Oh wow. I checked your site, but I (still) don't see the ability for average joes (and their pet AI agents) to signup. Could be huge, even if just offered in 5 minute slots. Is your email contact still valid in your profile? Want to reach out (Moose is the name). FYI, got a job at Antithesis a while ago after some rigorous interviewing.
wwilson 1 days ago [-]
We are working on getting self-serve signups to you ASAP. For now, just email us. We don’t bite.
asveikau 1 days ago [-]
> SQLite corruption is possible, but it’s highly unusual and not something you should encounter in normal operation
If there's a hardware failure, for example a flaky SD card, it's not out of the question. A mobile app with a lot of usage will see it.
(Yes, I know this appears to be a server use case.)
e28eta 1 days ago [-]
The sentence you’ve quoted actually links to a list of ways to corrupt SQLite, which I think is interesting in its own right: https://www.sqlite.org/howtocorrupt.html
I believe your flaky SD card is category 4, Disk Drive and Flash Memory Failures.
devmor 1 days ago [-]
I have encountered this exactly once, and it was in fact a flaky SD card - running a small web service off of a raspberry pi, with the SQLite DB stored on the SD.
LgWoodenBadger 1 days ago [-]
Maybe it's just me, but the explanations of the cause don't align.
One clue was that during corruption incidents, our metrics showed that SQLite would report copying more pages from the WAL file than were actually available. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong.
vs
it thinks some of the pages have been copied from the WAL into the main database file, but they haven’t. Those pages never get written to the database file, and that data is permanently lost.
The first says "more were copied than existed" but the second says "fewer were copied than should have been."
Like I said, it's probably just me interpreting something incorrectly.
nearlyepic 1 days ago [-]
I haven't looked into the actual code fix, but given the "reset" name I have to think it has to do with SQLite "thinking" it has copied more pages than it actually did.
i.e. The checkpoint starts, and a write hits after the modifications to data structures have been done but before the data has actually been put in the database. The process starts over again, but doesn't undo the changes it made to indexes etc. Hence the db thinks it holds pages that don't exist.
That's my interpretation, anyways.
thayne 23 hours ago [-]
Those seem consistent to me. Some pages weren't written to the WAL (yet?), but something else referenced them or otherwise indicated they existed, so then the other process tried to read them, resulting in the issue in the first quote.
inigyou 22 hours ago [-]
The explanation says they were written to the WAL and then not written to the main database file.
surgical_fire 1 days ago [-]
My interpretation is that they haven't been copied because they didn't exist?
If you have 10 pages and it tries to copy 20, either those 10 pages wouldn't really be copied, or bogus data would be written.
That's how I read at least. Those things are not mutually exclusive.
ball_of_lint 1 days ago [-]
Or you could have 10 pages, it actually copies 9, and reports 20 anyways.
surgical_fire 1 days ago [-]
That's also possible
jpiasolutions 1 days ago [-]
Same bug, two angles. Nothing really copies 20 pages, the "20" is just a broken counter. SQLite's internal tally of how many WAL pages it already saved to the main db gets corrupted and reads too high. Since it trusts that tally, it assumes those pages are already saved and skips writing them for real, so when the WAL resets they're gone. The "copied more than existed" number is the bug showing in your metrics; "pages never written" is the actual damage.
danpalmer 1 days ago [-]
Glad this got found and fixed, but I continue to be astounded at the amount of work people put into making SQLite do things that would be much simpler with other systems.
zbentley 18 hours ago [-]
Maybe, but there’s nothing about this particular bug that’s due to SQLite not being a networked database. Postgres is just as likely to have TOCTOU races in its checkpointer, which works roughly the same way.
waterTanuki 22 hours ago [-]
Tailscale likely deals with a lot of security-sensitive traffic, given the nature of the service. I'm guessing one of the requirements they were given was to have a tiny blast radius in case encryption keys got leaked, and that meant isolating each customer's tailnet (meta)data to it's own sqlite db rather than letting everyone share a postgres cluster.
inigyou 19 hours ago [-]
How about having devices generate and not share their private keys, instead?
waterTanuki 17 hours ago [-]
That's already how it works at the data-plane layer. I'm talking about the control-plane layer where tailscale needs to maintain a server that gets its state from somewhere (they need to know where to route your data, what your permissions ACLs are, device names, etc.). If everyone shared the same db cluster accessed over the network there is a real risk of an accident leaking the encryption keys (of the database itself, not your tailnet) to the db they would have to maintain, and leaking semi-sensitive info like these device names and IP addresses. Using sqlite means each tailnet's metadata is isolated to the container running it.
inigyou 13 hours ago [-]
Is it? They're backing them all up to S3 so what if the S3 keys leak?
antonvs 1 days ago [-]
Yeah it’s a pity that wasn’t addressed in the article.
This is a little like “we shot ourselves in the foot and then performed surgery on our foot, and everything is resolved now.”
bluehatbrit 24 hours ago [-]
That assumes that they do actually believe it's a mistake. They didn't explain the reasons they've gone for this architecture in much/any detail. I'd be interested in hearing them talk more about that in the future.
Perhaps you or I would make a different decision based on the aims that lead them there. But there isn't enough information to say whether or not their decision was a mistake, even if it has lead to a peculiar bug. It may be perfectly legitimate and we just don't know some of the constraints they had.
vluft 20 hours ago [-]
huh. ran into almost-certainly this, but blamed it on litestream and rearchitected a bit as a result. will have to see if I can reproduce the issue as we were using with the patched sqlite
declan_roberts 1 days ago [-]
> Now we’re in summer, we’re confident that we’ve found the bug, that we understand it—and more importantly, that we’ve fixed it.
This is the feeling I chase as a software engineer. It's the greatest motivator.
opengrass 9 hours ago [-]
I wish Microsoft wrote an apology like that.
jeffbee 1 days ago [-]
Block device upfuckery layers are powerful against databases. Years ago some colleagues wrote one that provides most of the hazards described by "Parity Lost and Parity Regained"[1] to test FoundationDB, which immediately uncovered several flaws in a project that described itself as well-tested. It's easy to do this with all the probing features that Linux (and others) provide today.
With opus 4.7-ish to fable 5, immediately after release - before they locked it down, it was shockingly easy to find crash bugs in a lot of very heavily used DBs and other software.
inigyou 22 hours ago [-]
Did you personally find them?
jeffbee 1 days ago [-]
I'll accept crashing in preference to a database acknowledging transaction that weren't durably committed.
dilyevsky 1 days ago [-]
For sure! I haven't had the opportunity to burn many tokens on this but given how quickly it was able to fuzz it into crashing i'm sure it can manage.
jnwatson 1 days ago [-]
I'm definitely going to use the word "upfuckery" instead of fault injection the next time I need it.
manoji 1 days ago [-]
Such a good write up . Having explored a little bit of sqlite internals for a codecrafters challenge i was mildly happy i could follow along what was happening .
chewbacha 19 hours ago [-]
Wild to have worked in the industry long enough that 16 years doesn’t feel that long ago.
gwking 1 days ago [-]
I wonder if this also affected litestream disproportionately, because litestream also inserts itself into the checkpoint process.
why-el 1 days ago [-]
I would assume they actually moved off of litestream here no? otherwise how can their frequent manual checkpointing even succeed when litestream locks for the same behavior.
gwking 20 hours ago [-]
I meant does the rare SQLite bug tend to manifest in litestream for the same reason it showed up in Tailscale.
klaas- 1 days ago [-]
> This investigation is a useful reminder: running boring technology in a non-standard way is a risk. The common paths and standard configurations are incredibly well-tested and reliable. Most people use SQLite in a standard configuration and never face this sort of issue. Everything we were doing was a public, documented, supported configuration—but by taking manual control of the checkpointing process and running at our own aggressive pace, we stepped off the well-trodden operational path.
I feel like they missed a key takeaway from their own argument here, they should not be running a non-standard configuration :)
raggi 23 hours ago [-]
We have very good reasons for our checkpointing model, related to our backup + disaster recover strategy, along with resource cost. It might be worth writing about one day, so I'll not give away all the details, but in very short form, we organize a backup strategy that has minimal pause time, avoids doubling the page cache cost of the database, and enables extremely fast byte-copy restores in disaster recovery.
inigyou 22 hours ago [-]
okay but explain why you are using sqlite and copying the file to S3 instead of using any client/server DB and its online backup feature?
raggi 7 hours ago [-]
SQLite has an online backup API as well, but it is slower and requires a significant additional page cache cost.
The team chose SQLite early on (there are some blog posts about this) and then we vertically scaled against the SQLite architecture. There are subtle ways you come to depend on the proximity/latency when you scale with local storage that mean switching requires a lot of non-obvious work - it’s probably the largest hazard for embracing SQLite in a growing saas - but at the same time you can push the vertical scale pretty far, which has great margins.
Had we scaled a different architecture of database there’s little reason to believe it would have been plain sailing as seems to be implied here.
This reminds me why RFC 2119 includes the description it has regarding SHOULD.
Like this is a great practical takeaway, sure, but eh.
grahar64 1 days ago [-]
Awesome write up. Finding these bugs in such a well used piece of software is like donating to humanity
quarkcarbon279 19 hours ago [-]
love the simplicity of this article. Reminds me so much of foundational software engineering. i love databases def not cosmosdb
riknos314 1 days ago [-]
> Whenever corruption occurred, we had to stop the control plane process on the shard while we repaired or restored the database. This was painful for tailnets on that shard, because their entire control plane disappeared during that recovery window.
Gotta love single points of failure...
tptacek 1 days ago [-]
This is maybe one of the purest examples of the Bell Curve Meme in software engineering. The things you would do to the system Tailscale operates to eliminate all single points of failure (generally, and in the specific case where, where the "single point of failure" applies only to a small cohort of customers) would make the system less resilient, and increase failures.
Generally, you do complex distributed systems without on-paper single-points-of-failure anywhere when you absolutely have to, because those systems don't have transient failures. That's not mesh networks like Tailscale at all.
You don’t need the control plane most of the time. I had a zero downtime headscale upgrade because once the nodes negotiate through the control plane they can talk to each other all the time. The data plane is peer to peer.
It’s problematic because you can’t run connections but it doesn’t stop the world.
dilyevsky 1 days ago [-]
Well it's SPoF for a shard not the whole thing so in practice that makes this considerably better. Secondly, it's not a datapath SPoF so presumably all tailscaled processes would continue running and exchanging traffic no problem - you just won't be able to launch any new ones.
kccqzy 1 days ago [-]
What are some solutions to avoid database corruption being single points of failure? I can’t think of any off the top of my head. I don’t think people typically consider database corruption to be a kind of failure common enough to design for, unless you have unusual requirements.
AlotOfReading 1 days ago [-]
The general answer to this is Byzantine consensus, which cryptocurrency blockchains are designed to solve. If your nodes are willing to fail a little more politely (e.g. no lying, immediately crashing, etc) you can use something cheaper like raft/paxos.
But yeah, it's a lot cheaper to build a reliable system than it is to be resilient.
spockz 1 days ago [-]
The shard was already a way to make it not a single point of failure.
Spivak 1 days ago [-]
This is a great example of outages looking different from the perspective of the operator vs the user. Because there's many shards the blast radius of failure is contained to a small subset of users but for those users it's an outage. The way it's designed you can't lose any shards without impacting users. Compare to say Elasticsearch where it's possible to lose nodes and lose shards without the user noticing. One approach isn't universally better than the other.
spockz 1 days ago [-]
Of course. On the other hand, is it fair to call it a single point of failure if it being down only affects a shard? Maybe within the shard indeed. Or for a single customer indeed. Not sure if we have terminology to indicate one or the other.
password4321 1 days ago [-]
I lost my Android SMS DB way back in the day (20+ years? dang I'm getting old) because the SMS app's fix when opening the DB detected any issue was to delete it and start fresh.
buggymcbugfix 1 days ago [-]
lol good old ostrich algorithm
JohnBooty 20 hours ago [-]
Excellently-written article. Bravo!
Zenul_Abidin 1 days ago [-]
Similar bug to the one that plagued Codex until 3 months ago.
ameliaquining 1 days ago [-]
Do you happen to have a link?
peter_d_sherman 9 hours ago [-]
>"...and then we discovered an unexpected clue.
We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky). To do this, we built a
transaction logging pipeline.
We streamed every SQL statement that modified the database to a separate log file. Because SQLite is a single-writer database with serializable transactions, our transaction history was completely linear and
deterministic.
(This wouldn’t be true in a multi-writer database like Postgres or MySQL.) Replaying those transactions against the latest known-good backup should restore the database to its most recent state, safely bypassing the corruption.
[...] This pipeline worked, but then it did something even better: it gave us a clue.
[...] To understand what was happening during these faulty checkpoints, the SQLite developers created a new debugging tool for the virtual filesystem layer.
[...] To help diagnose our problem, the SQLite developers created a wrapper around the virtual filesystem that
writes additional tracing information and logs
about changes to the database.
[...] After our next corruption incident, the additional logs from the new tmstmpvfs shim allowed the SQLite developers to find and fix the bug:
a rare data race
in the SQLite source code between a checkpoint and a write transaction."
Great article!
Software Engineering lessons (that repeat in this article!): So called "Heisenbugs" (bugs that make it past developer test harnesses and a company's Quality Assurance (QA) team) that show up post-deployment intermittently and can't be reproduced locally, occur because one or more of the following factors:
1) The lack of Determinism in a software process or processes.
2) The lack of appropriate logging.
3) The lack of the ability to replay a software process, step by exact step, state by exact state, as it has occurred in the field (occurs as an effect of #1 and/or #2).
4) Multi-threaded code; i.e., multiple threads giving rise to race conditions or other very specific intermittent combinatorial/permutational conditions caused by multiple threads and specific sections of code, which due to very large numbers of permutational timing possibilities, were not or could not be exactly tested for in development...
Anyway, great article! A must-read for any Sr. Software Engineer, or any developer that wrestles with hard-to-find-and-fix bugs in the field...
khernandezrt 1 days ago [-]
Show us the $$ numbers! Or an estimate at least.
miki123211 9 hours ago [-]
> Nobody wanted us to spend six months looking for bugs in SQLite
Most companies wouldn't. Instead, they'd fire the weirdo that came up with the idea of using some weird db, and switch to Postgres like God intended.
I'm not saying either is right, this is not a criticism of Tailscale and their approach, paying the Sqlite maintainers to fix a real bug is commendable, but it certainly doesn't inspire confidence in the "Sqlite in production" hype train.
While Sqlite is indeed boring technology for single-user SQL DBs, for traditional CRUD and network services, Postgres seems to be a much better trodden and much safer path.
getnormality 21 hours ago [-]
I wonder what happens when you give Claude the old version and the logs and ask it to find the bug.
nujabe 18 hours ago [-]
What logs though? They had to make multiple deployments to get the right log traces in the first place.
hn3ufz62f7 1 days ago [-]
Learned something new today, thanks
positive-spite 1 days ago [-]
"Tailscale Traces Database Corruption to 16y/o ..."
Would have been a superior headline
*sarcasm
positive-spite 15 hours ago [-]
The title changed...
Now my joke works even worse :(
18 hours ago [-]
XorNot 23 hours ago [-]
I see tailscale also use the pure Go SQLite conversion so I hope this fix will land there soon, it's rapidly become one of my favorite packages for self-contained tools.
pstuart 1 days ago [-]
I imagine the SQLite eschews AI generated code, but using it for testing (vulnerability, performance, etc) would seem like an easy win.
I know their proprietary testing framework is their secret sauce so we may never know...
d-us-vb 1 days ago [-]
Richard Hipp's recent talk at Software Should Work explains that AI agents have been testing SQLite and they've gotten a deluge of new bug reports from the fuzz-like testing they can do. But they do not do this in house; hobbyists and other organizations do this in their own internal agent-driven fuzzing.
1 days ago [-]
Ruca_AI 4 hours ago [-]
[flagged]
five9llc 9 hours ago [-]
[flagged]
jkwang 11 hours ago [-]
[flagged]
sdiazthomas 14 hours ago [-]
[flagged]
richard_gg 9 hours ago [-]
[dead]
fenestella 12 hours ago [-]
[dead]
tangsoupgallery 11 hours ago [-]
[flagged]
RinatNabiev 11 hours ago [-]
[dead]
tizerluo 20 hours ago [-]
[flagged]
madhu_ghalame 16 hours ago [-]
[dead]
1 days ago [-]
dzonga 1 days ago [-]
you gotta admire the power of using json/b and simple KV stores.
so many people sleep on that.
ec109685 1 days ago [-]
While technically true as written, it seems to downplay the significance:
> The bug is a data race with tight timing constraints. It is unlikely to occur in common use.
A large customer did experience this corruption, so it's important for people with tailscale's setup update immediately.
> The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliberately triggers the circumstances of the the bug in order to verify that the issue has been fixed.
Ariarule 1 days ago [-]
Odd not to highlight the sentence where they answer the obvious question "Why Tailscale in particular?":
> They also explained why we were more likely to hit the bug than other SQLite users: we take manual control of the checkpointing process, and we checkpoint very aggressively. Even a bug triggered by a rare condition was bound to hit us eventually.
inigyou 22 hours ago [-]
The bug requires two checkpoints in very quick succession, which presumably isn't something sqlite would do on its own, as it would be a pointless waste of performance.
ec109685 16 hours ago [-]
Tailscale isn't mentioned in the SQLite changelog.
dboreham 1 days ago [-]
Quick note that data corruption bugs that are impossible to reproduce are not uncommon (perhaps they're the norm). So some amount of head scratching trying to figure out a plausible scenario by which the system could get into the state represented by the smoking remains is often required. Then you attempt to force it into the supposed bad state by modifying code paths accordingly. So the approach used in this case is clever, but it's not particularly unusual in the world of data stores.
ec109685 16 hours ago [-]
I agree it's tricky, but Tailscale had downtime by this bug and the SQLite Changelog downplayed it.
This was a great technical writeup and very interesting to read, but it's not clear to me why once the suspected source of the bug was identified, they seemingly didn't build a automated way to trigger the condition? It seems like that could have cut down on the uncertainty of whether the fix worked over a painfully long period of time.
Neywiny 1 days ago [-]
The sqlite dev team did. It's in the article.
> It could exist that long because it was rare—so rare, the SQLite developers had to add code to deliberately trigger it in their testing environments.
Interesting example of a company funding open source - in this case paying for the development of a new and very specific debugging tool.
Unfortunately, HeadScale is not a drop in replacement for TailScale. Most people don't realize this until they start self hosting.
Due to this, I had to migrate from Tailscale to NetBird, which is completely open source.
https://avilpage.com/2026/06/moving-from-tailscale-to-netbir...
https://github.com/juanfont/headscale/issues/1651
For macOS, you need to build the open source tailscale client. Or create file in esoteric file location to opt out using App Store.
For iOS I don’t even think it’s possible to opt out.
> Because this wouldn’t be a quick or easy fix, we reached out to the SQLite developers for a professional support contract. This was a great decision. It gave us direct access to their deep expertise and experience, and we had many detailed technical conversations about our architecture and our incidents.
They were willing to pay to get help solving the problem, and then pay again to make sure that the problem is easier to avoid in the future! That kind of long-term thinking seems pretty rare nowadays...
Meanwhile tailscale or wireguard, by being actually opinionated, avoids needing much configuration at both ends.
I have now mostly switched to Wireguard for this, which is much more sane [2].
[1] https://github.com/jawj/IKEv2-setup [2] https://github.com/jawj/wireguard-setup
On Windows and Mac, VPN config is usually just installing a client, signing in, and then it all basically works. Sometimes you don't even need the client and can set it up in settings.
Don't be the identity provider, have the email host be the identity provider (which it is anyway if you have a forgot password prompt).
Agreed 100% that nobody should still be using passwords in 2026 though.
The only time I like magic links is for services where I am "not really a user". For example, an appointment reminder for my doctor where I need to validate my insurance. Great, send me an email 24-hours before with the reminder and a magic link, as I don't want to think about an account there.
This is an extreme case, yes, but I have similar issues with several services that only support magic links and GitHub login, and it's hard to configure services to use a new catch-all address when they harvest my address from GitHub login and won't let me change it. Magic links are a great tool, but they should _never_ be the only option. They should supplement existing login methods. Password managers exist for a reason. Several services that I have since stopped using for this reason have in the past decided that they should start sending emails to the addresses they harvested from login with other service. Two services have gotten my magic link address stuck in their system because it's not associated with an account (the account has been deleted) and thus I can't manage subscriptions.
A username/password would be perfectly fine for my use-case as an individual user using Tailscale for my homelab and personal devices. But I suspect the majority of real paying customers don't want a separate authentication flow just for Tailscale and would rather use the SSO they already have. I have confidence Tailscale wouldn't half-ass usernames and passwords is they offered it even if it was mostly just for non-paying hobbyist users, but I'm also sort of glad I get to bucket my account security risk with the people paying their bills.
This is Percona's business model. They employ core pg/mysql developers and you can buy a support package from them. Same for enterprisedb. Pretty reasonably priced packages (like maybe $10k-ish/core IIRC) get you 24x7 support. I've only had to escalate issues once but inside 10 hours we got a pg core committer to debug some very strange vacuum behavior.
It seems very similar to Tailscale: immense utility from a free-tier product for the general public, which leads to trust and a large enterprise market.
A shame really, because Keybase was kind of unique in the way the facilitated social proof of identity tied to cryptographic keys. A shame though that they never found a really sustainable business model. When they turned to crypto currency towards the end of their independent run I knew it probably wouldn't last.
I just deleted it as I noticed it was responsible of about 30% of active cpu (and battery drain) usage on my mac, while being logged out due to inactivity. Honestly: I never particularly liked it, and I could not feel the concept especially tempting, just went with the team.
https://sqlite.org/prosupport.html
Looks to me like this is the tool in question, added in January: https://github.com/sqlite/sqlite/commits/master/ext/misc/tms...
> A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used.
This line led me to believe that the writer and checkpointing logic lived on the same database connection, so I was curious to find out how the data race occurred. However, the bug details on the SQLite page[0] outline that it can only ever occur if there are multiple connections open, so the writer and the checkpointer must have been on different threads.
[0] https://sqlite.org/wal.html#the_wal_reset_bug
To be honest, I'm surprised that someone using SQLite would try to access it directly from multiple threads or processes without fear of data racing.
https://sqlite.org/faq.html#q5
One writer, multiple readers is a specifically supported way of using SQLite.
Why should you be worried if it is used as designed?
Well this whole article is about a company discovering a catastrophic corruption bug even though they were using it as designed.
I think the lesson is that if you're ever actually worried about concurrency then just don't use sqlite. We can see here that concurrency is hard and the bugs are old and deep.
I would not think of SQLite as "lite" anything. It's SQL In The Executable.
It has a better security and data-durability track record than both Postgres and MySQL, and often beats them in the sorts of things applications do with databases:
https://sqlite.org/speed.html
> it's often pushed into places where it's being asked to do things it was never really designed for
https://sqlite.org/whentouse.html
https://sqlite.org/hirely.html
Seems like it absolutely is "designed" for this use case.
It"s actually SQL "ite" as in rocks, minerals and fossils. Their version control system is called "Fossil".
https://sqlite.org/testing.html
It's amazing that a bug could exist for 16 years but it is sobering.
https://antithesis.com/blog/2026/wal-reset-bug/
Dijkstra: Tests can only prove the presence of bugs, never their absence!
But I had a funny experience recently where an agent implemented an entire feature completely wrong (exactly backwards, actually, in a way that defeated the purpose, introduced security issues etc.).
It happily supplied tests for the new functionality, and all the tests passed.
What I realized was, even formal verification wouldn't have helped here -- it would have just written a mathematical proof that the incorrect functionality was correctly implemented!
So there's a gap here, where first, the human's intention needs to be formally specified (by the human, or at least the human needs to be able and willing to verify it), and then the slopswarm can hack away at it...
Yes, that has always been the case with formal proofs. Even a mathematical proof with Lean is useless if the proven formal statement doesn’t match what the mathematician meant. And that’s why humans working with formal languages won’t be going away. Natural language is imprecise by nature, even when we have precise ideas in mind. In that case, it’s often better to specify what we mean in a formal language than in natural language.
This never changed with AI; in fact, I think it made this need more visible than it ever had been before. You can't get away with not being able to describe in detail what you want. As with working with humans, any ambiguities will be interpreted, and not always in the way you hoped.
Not by hand in the olden days but with AI to help with it, why not?
This is importantly different from formal code verification, in which the specification that needs to match the programmers' intentions is small enough to review, while the bulk of the LLM's output is proof tactics and such, which don't need human review because the verifier guarantees* that they're right.
I suppose you could just tell Fable "hey try using TLA+ to look for bugs in this code" and see whether it finds any.
* Uh, usually. https://leodemoura.github.io/blog/2026-8-1-postmortem-for-ke...
What do you know?
I think a model checker or something similar that can exhaustively cover a search space would suffice, though at that point I think the boundary between "test" and "formal verification" becomes somewhat fuzzy.
This is the gotcha: in stateful environments you'll have state-space explosion so big you'll have to come up with some compromising measure of "equivalent". You're right that that would constitute a proof by exhaustion of correctness. Unfortunately in practice it's entirely impractical.
[0]: demsky.eecs.uci.edu/publications/c11modelcheck.pdf
[1]: https://docs.rs/loom/latest/loom/
"Simple"/"complex" doesn't necessarily imply "small"/"large", respectively. Testing whether the Collatz conjecture holds is "simple", but its search space is (potentially) infinitely large. Testing correctness of concurrent data structures is (probably) a more "complex" problem, and yet it's still possible to perform exhaustive tests of their functionality [0].
In the end, the practicality of exhausting the 64-bit integer search space is irrelevant since it says nothing about the size of the write + checkpoint search space. I could just as easily say that it's easy to exhaust the 16-bit integer search space, but that doesn't mean exhaustively testing for this particular bug is easy.
[0]: http://demsky.eecs.uci.edu/publications/c11modelcheck.pdf
It's an art to come up with a great test suite that covers just enough and minimizes overlap, not only survives but also helps with refactoring.
The union of a lot of necessary conditions is not a sufficient condition. But it might be good enough for software.
https://en.wikipedia.org/wiki/Probatio_diabolica
https://07th-expansion.fandom.com/wiki/Devil%27s_Proof
A few (very, very, very pedantic) things that stood out:
> We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky).
(Emphasis mine) - it would be "risky", not "potentially risky" - then the "calculated risk period" starts and it's "potentially problematic".
In the SQLite report[0] (11.2) I wish they downplayed this less - a mention of the rarity, then technical details - I'm friendly with a few of the devs/previous-devs, have the utmost respect for their skill and accomplishments (and by extension, faith that the developers I do not personally interact with are also excellent), appreciation and fondness for the huge accomplishment that is SQLite, and on and on... this is world-class work. Maybe section 11.2 wasn't really aimed at me, or I'm too critical. To be fair to all involved, what a minor quibble for such an interesting problem/fix. I hope my comment isn't a fly in the ointment.
Last bugfix point[1] - ugh. What a sinking feeling that must've been to deploy a fix then be flooded with not-green - and a lesson[2] against smuggling other changes in a changeset "just because we're already here"? Happy it turned out non-catastrophic, but did result in a rare (not remembering other instances of top of head) recall[3] from SQLite. That it was throwing errors at the same time SQLite and Tailscale were testing the other WAL-issue bug must've upset some stomachs for a moment.
[0] https://sqlite.org/wal.html#the_wal_reset_bug
[1] https://tailscale.com/blog/sqlite-wal-reset-bug#fixed-with-a...
[2] Nobody conceptually learned anything here - we're all just reminded of what we know: that sometimes "perfect storms" do actually occur.
[3] https://sqlite.org/releaselog/3_52_0.html
I'd have liked to have heard more about the decision to checkpoint so frequently that put them on this path though. Presumably that's to keep the WAL tiny for very fast recovery. Trying to mitigate some of the deleterious effects of inserting a DBMS into your network layer, I suppose? Tricky stuff. Wonder how that compares to typical etcd snapshot frequencies too.
> running boring technology in a non-standard way is a risk.
It was a good read and reminder that the industry is loosing experts gradually. I am not a DBA and yet I have heard about this behavior at least couple times in the past as something to avoid. Its just one of those things which didnt get a chance to be documented cause experts avoided it and regulars didn't get into
Backing up sqlite by copying the file (e.g. rsync) while it's open is a surefire way to eventually get corruption caused by a race condition, but it seems like tailscale wasn't doing that. They were probably using the proper sqlite backup API.
But you don't need checkpoints for consistency and I think the backup API will not copy both the old and new versions of pages just because they're in the WAL, in other words I think checkpointing then backing up should give you the same pages as backing up without checkpointing. So the whole thing seems unnecessary.
On second question of why the tailscale developers did it, its possibly for the same reason why they invested this much into debugging this issue. Some one believed the current behavior did not fit into their architecture, they want to be more performant and take control over things. A big part of me considers this is a required exercise to try, grow and learn. The only thing they could have for improvement would be to have these old hands on architect kind of folks on their team who might have hinted/pointed them to the problem a long before. Challenge/chances are that these older folks would have even stopped them from going in this direction in the design phase itself.
This is just proof that it can't be used in real-world applications.
SCNR
> Abstract: SQLite is a C-language library that implements a self-contained, in-process relational database engine supporting full-featured SQL, an advanced query planner, and ACID transactions. By many estimates, SQLite is the most widely used software library in the world today.
> Over its 26-year history, SQLite has gained a reputation as software that "just works". This talk goes over the design choices and development practices that have, at least in the opinion of the lead developer, resulted in that reputation.
* https://www.youtube.com/watch?v=V_qzqY1bb7I
> Consortium members have the guaranteed, undivided attention of the SQLite developers for 23 staff-days per year and for as much additional time above and beyond that amount that the core developers have available. There are no arbitrary limits on contact time. The consortium will never be over-subscribed. New SQLite developers will be recruited and trained as necessary to cover the 23 day/year support commitment.
The SQLite home page lists five companies that have paid for consortium access. I can easily imagine that there are more who don't want to pay $150K/year but would pay $1.5k/year, proactively, to get "private, expert email advice from the developers of SQLite" when they need it.
Tailscale just moved up in my priorities list. Was going to host my next website with hostinger, but now I'm going to at least try to run a personal server with tailscale to make it public. I might not be able to figure it all out, and may end up going with the VPS route, but this gave me some appreciation for the company that makes me willing to try the less familiar method.
I spent hours empirically trying various scenarios before concluding that no, it was a ProxySQL bug.
[0]: https://github.com/sysown/proxysql/issues/2233
[1]: https://github.com/sysown/proxysql/pull/5385
SQLite has 59,000% ratio [1]
Yet it didn't help for a bug to left unnoticed for 16 years :( I don't know what we can do for the industry. I doubt one can formally verify a project like SQLite, and keep it maintainable.
https://sqlite.org/testing.html
Here they were trying to do a backup by forcing a checkpoint and then copying the file. Systems like postgres let you do online continuous backups.
Online backups/replicas are nice until you’re charged for network traffic or have to recreate the replica from scratch, at which point the initial-restore-then-hook-up-the-WAL-stream dance is prone to all sorts of racy issues. If you’re lucky enough to have only a single process talking to the DB, SQLite seems like a nice way to sidestep that complexity while keeping a simple backup story.
Heck, this is basically the Redis model: a single process/thread coordinates all access to the data, and occasionally forks off a background job to snapshot the state somewhere. From that perspective, the Tailscale controller binary is a database; SQLite is just the data file format.
It's so sometimes just performative. I remember when Tailscale had a similar performative approach with 'just use a JSON file on disk'. Then etcd. Then SQLite. Like sure, you can keep picking the absolite mininum technology for your needs and then change it every couple of years... Or you could just immediately go with a solid Postgres (or Postgres-like setup, eg. yugabyte) and save yourself a bunch of faffing about with weird solutions and migrating between them. But I guess that doesn't drive engagement on your blog.
And this statement is wrong in the article: “ Because SQLite is a single-writer database with serialisable transactions, our transaction history was completely linear and deterministic. (This wouldn’t be true in a multi-writer database like Postgres or MySQL.)”
Actually possible in mpedb to replay multi-writer, and actually better than SQLite3. Try to reply now() in a statement, that is not deterministic in SQLite but is in MPEdb.
If there's a hardware failure, for example a flaky SD card, it's not out of the question. A mobile app with a lot of usage will see it.
(Yes, I know this appears to be a server use case.)
I believe your flaky SD card is category 4, Disk Drive and Flash Memory Failures.
One clue was that during corruption incidents, our metrics showed that SQLite would report copying more pages from the WAL file than were actually available. If there are 10 pages in the WAL file and 20 pages get copied to the database, something is clearly wrong.
vs
it thinks some of the pages have been copied from the WAL into the main database file, but they haven’t. Those pages never get written to the database file, and that data is permanently lost.
The first says "more were copied than existed" but the second says "fewer were copied than should have been."
Like I said, it's probably just me interpreting something incorrectly.
i.e. The checkpoint starts, and a write hits after the modifications to data structures have been done but before the data has actually been put in the database. The process starts over again, but doesn't undo the changes it made to indexes etc. Hence the db thinks it holds pages that don't exist.
That's my interpretation, anyways.
If you have 10 pages and it tries to copy 20, either those 10 pages wouldn't really be copied, or bogus data would be written.
That's how I read at least. Those things are not mutually exclusive.
This is a little like “we shot ourselves in the foot and then performed surgery on our foot, and everything is resolved now.”
Perhaps you or I would make a different decision based on the aims that lead them there. But there isn't enough information to say whether or not their decision was a mistake, even if it has lead to a peculiar bug. It may be perfectly legitimate and we just don't know some of the constraints they had.
This is the feeling I chase as a software engineer. It's the greatest motivator.
1: https://www.usenix.org/legacy/event/fast08/tech/full_papers/...
I feel like they missed a key takeaway from their own argument here, they should not be running a non-standard configuration :)
The team chose SQLite early on (there are some blog posts about this) and then we vertically scaled against the SQLite architecture. There are subtle ways you come to depend on the proximity/latency when you scale with local storage that mean switching requires a lot of non-obvious work - it’s probably the largest hazard for embracing SQLite in a growing saas - but at the same time you can push the vertical scale pretty far, which has great margins.
Had we scaled a different architecture of database there’s little reason to believe it would have been plain sailing as seems to be implied here.
Like this is a great practical takeaway, sure, but eh.
Gotta love single points of failure...
Generally, you do complex distributed systems without on-paper single-points-of-failure anywhere when you absolutely have to, because those systems don't have transient failures. That's not mesh networks like Tailscale at all.
As always: https://how.complexsystems.fail/
It’s problematic because you can’t run connections but it doesn’t stop the world.
But yeah, it's a lot cheaper to build a reliable system than it is to be resilient.
We wanted a way to restore service that didn’t involve rolling back to the last known-good backup (which would lose a lot of data) or repairing the known-corrupted database (which was potentially risky). To do this, we built a
transaction logging pipeline.
We streamed every SQL statement that modified the database to a separate log file. Because SQLite is a single-writer database with serializable transactions, our transaction history was completely linear and
deterministic.
(This wouldn’t be true in a multi-writer database like Postgres or MySQL.) Replaying those transactions against the latest known-good backup should restore the database to its most recent state, safely bypassing the corruption.
[...] This pipeline worked, but then it did something even better: it gave us a clue.
[...] To understand what was happening during these faulty checkpoints, the SQLite developers created a new debugging tool for the virtual filesystem layer.
[...] To help diagnose our problem, the SQLite developers created a wrapper around the virtual filesystem that
writes additional tracing information and logs
about changes to the database.
[...] After our next corruption incident, the additional logs from the new tmstmpvfs shim allowed the SQLite developers to find and fix the bug:
a rare data race
in the SQLite source code between a checkpoint and a write transaction."
Great article!
Software Engineering lessons (that repeat in this article!): So called "Heisenbugs" (bugs that make it past developer test harnesses and a company's Quality Assurance (QA) team) that show up post-deployment intermittently and can't be reproduced locally, occur because one or more of the following factors:
1) The lack of Determinism in a software process or processes.
2) The lack of appropriate logging.
3) The lack of the ability to replay a software process, step by exact step, state by exact state, as it has occurred in the field (occurs as an effect of #1 and/or #2).
4) Multi-threaded code; i.e., multiple threads giving rise to race conditions or other very specific intermittent combinatorial/permutational conditions caused by multiple threads and specific sections of code, which due to very large numbers of permutational timing possibilities, were not or could not be exactly tested for in development...
Anyway, great article! A must-read for any Sr. Software Engineer, or any developer that wrestles with hard-to-find-and-fix bugs in the field...
Most companies wouldn't. Instead, they'd fire the weirdo that came up with the idea of using some weird db, and switch to Postgres like God intended.
I'm not saying either is right, this is not a criticism of Tailscale and their approach, paying the Sqlite maintainers to fix a real bug is commendable, but it certainly doesn't inspire confidence in the "Sqlite in production" hype train.
While Sqlite is indeed boring technology for single-user SQL DBs, for traditional CRUD and network services, Postgres seems to be a much better trodden and much safer path.
Would have been a superior headline
*sarcasm
Now my joke works even worse :(
I know their proprietary testing framework is their secret sauce so we may never know...
so many people sleep on that.
> The bug is a data race with tight timing constraints. It is unlikely to occur in common use.
A large customer did experience this corruption, so it's important for people with tailscale's setup update immediately.
> The developers have never been able to reproduce the bug organically and had to add special testing logic to SQLite that deliberately triggers the circumstances of the the bug in order to verify that the issue has been fixed.
> They also explained why we were more likely to hit the bug than other SQLite users: we take manual control of the checkpointing process, and we checkpoint very aggressively. Even a bug triggered by a rare condition was bound to hit us eventually.
> It could exist that long because it was rare—so rare, the SQLite developers had to add code to deliberately trigger it in their testing environments.