Rendered at 20:25:15 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
tpowell 3 hours ago [-]
It takes a bit of setup and a huge download, but every time I need a good domain I follow this old post from Derek Sivers. I have Claude de-dupe it and turn it into a searchable database (on my machine), then have it search genres and terms I'm looking for. It's a task Claude is very well-suited to, from the technical implementation to back-and-forth about selections.
[link]: https://sive.rs/com
eggbrain 2 hours ago [-]
Note -- if you do this, watch out for requesting access to "all tlds". They send you two emails per TLD -- one for your pending state, and one for your approved/rejected state. I suddenly had 1k+ emails flooding into my inbox, until I found the setting on their website to disable emails.
Your impressions seem about right, but there are a few control steps it seems.
moffkalast 4 hours ago [-]
They really needn't have specified "in a weekend" cause yeah we can tell.
Since when has low effort become a selling point anyhow?
smokel 3 hours ago [-]
I typically interpret it as an excuse, not a selling point.
dreamforever 2 hours ago [-]
it was by no means low effort
eichin 2 hours ago [-]
Reminds me that AltaVista's servers ran in 4G of RAM (there were famous, at the time, pics of the circuit boards - DEC was rightfully proud of this, 30 years ago) and that a modern AltaVista should run on a decent laptop :-)
coredog64 1 hours ago [-]
During AltaVista's prime, 32MB would have been a lot of memory for a typical computer.
Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea.
I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
alightsoul 4 hours ago [-]
I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving them money
There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
burger_moon 11 minutes ago [-]
> There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
Where do you get stats on this info? Not refuting your claims but it's interesting data. I would've thought a wider gap in all biz to active websites since some countries run nearly all their biz on whatsapp/telegram/wechat.
jeromechoo 4 hours ago [-]
It feels like we've hit a point where search engines can become what "todo list apps" were for devs 10 years ago.
What a homebrewed solution lacks in coverage it excels in indexing and serving a small slice of the internet really really well.
marginalia_nu 3 hours ago [-]
To be fair they are a supremely interesting problem to hack away at, and one that will meet you where you are.
Almost anyone can put together a basic search engine in a few thousand lines of code, it's just not very hard to make a program that will index a few million documents better than Confluence.
Then, between that first ansatz and a working scalable internet search engine, you have a pile of interesting problems touching every aspect of computer science and computer hardware and networking, enough so that hundreds of people will have gotten PhDs in narrow sub-problems of those problems you'll be facing.
It's great because you can just tackle the stuff you feel comfortable approaching and leave the rest for later.
headz 6 hours ago [-]
TS;DR: Too Sloppy; Didn't Read.
juleiie 6 hours ago [-]
Yeah it’s ai generated obviously but as I said previously which some people didn’t like - don’t judge the tools, judge the content.
I don’t care if robot hand written this or black or white. It’s useful.
augment_me 35 minutes ago [-]
People don't like this because you move the responsibility of verification to the reader.
Does the program cheat? Does it just query a table is saved in some backend? Maybe, I don't know, you don't know, and since the author only pressed enter once no-one will know until/if someone runs into any issues.
When you can create information for free, and you want people to use this information you need to provide guarantees, proof, and understanding beyond what the autonomous model has done. Otherwise you just move the burden of verification to the reader.
criddell 1 hours ago [-]
If this isn't something worth your time to write, it probably isn't worth my time to read.
Just post the part you did write - the prompt.
order-matters 3 hours ago [-]
did you edit it? did you fully read everything yourself and take out anything extra that didnt need to be in there like unnecessary comparisons and typical AI idioms? tweak the language to be less dramatic or emphatic about things that do not need emphasis?
AI is a great tool, but it is somewhere between a 3D printer and a CNC machine. it can make something smooth and easy to handle and maybe good enough for personal use but you would want to run it through some Finishing steps before giving its output to someone else. In some cases thats using the output as a mold for a full recreation and other times maybe its just sanding it a bit, smoothing out some rough spots and putting paint on it.
youre right not to care about if something was made by a human or a tool, but the full presentation of the content including editing is part of the content when it comes to a write-up.
abc3354 6 hours ago [-]
In the expression "AI slop", "AI" is about the tools, "slop" is about the content
uean 6 hours ago [-]
I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there.
It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."
juleiie 6 hours ago [-]
[flagged]
vivzkestrel 5 hours ago [-]
- you are going to regret offloading this much of your brain's critical thinking to an external source in the long run
- every night a good number of neuron connections in your brain are automatically severed because you did not use that ability
- over long periods of time, it ll take you to a point where you wont be able to write a simple factorial program on your own without gpt telling you
- i am not saying this to incite you or mock you or anything. i am just concerned about how much of your thinking you are offloading
juleiie 1 hours ago [-]
I am not offloading anything. I need way more critical thinking than I ever needed before to review solutions proposed by my digital slaves to evaluate whether they are garbage or good.
If you think that having slaves means you don’t need to be a good administrator, then you are sorely mistaken.
It requires more brainpower than it ever did to manage a project the size what used to do a 100 people team. All on your own.
The fact that I am arguing with all of you here and not blindingly follow the narrative should tell you all you need to know about my critical thinking prowess.
danggggg 5 hours ago [-]
[dead]
fg137 4 hours ago [-]
> because that’s how things will look like from now on.
No it's not.
People are increasingly getting fed up with slops like this -- you can see comments in HN discussions.
Many people just completely skip those articles.
prepend 3 hours ago [-]
More importantly, I’ve started my own little blacklist of people and site I just ignore.
There’s people at work whose messages get completely ignored after too many times of posting verbose, useless, ai slop.
gist 3 hours ago [-]
> People are increasingly getting fed up with slops like this -- you can see comments in HN discussions.
Why claim 'people' (plenty of people get value from satisficing) and even try to use 'see comments in HN discussions' as if there is something that actually documents that ie 'we all think it'.
I skimmed it and even skimming it I got value from (what some others here) call 'slop'. (The value was basically it gave me an idea of something I could implement and again I didn't have to study it or even understand everything (or agree) just seeing someone did that made me think about something I might want to do.
sporedro 4 hours ago [-]
If I want to read AI output, I’ll just ask my own AI. I’m not going to bother reading someone else’s AI.
uean 5 hours ago [-]
Thanks for this. My point is proven.
juleiie 5 hours ago [-]
[flagged]
flyingcoder 5 hours ago [-]
How did web3 and crypto work out for you buddy?
jorisw 4 hours ago [-]
You’re confusing moving on with the times, and using new tools wrong
prepend 3 hours ago [-]
It’s not my problem because I can safely skip ai slop without fomo.
It’s extremely rare that someone smart with something smart to say produces this crap.
Like someone said upthread if the author isn’t willing to spend time writing cogently then I don’t think it’s worth my time to try to parse it.
yieldcrv 5 hours ago [-]
yes but tell your slopywriter to be less verbose, get to the point
juleiie 5 hours ago [-]
[flagged]
nxndjdkdksmsb 4 hours ago [-]
[dead]
fg137 4 hours ago [-]
> judge the content
The content itself is slop.
ImPostingOnHN 5 hours ago [-]
"slop" is a perfectly valid judgement of content
surely if we're expected to read this ourselves, the author can write it themselves?
indeed, it's helpful to the author, too: writing helps you learn
ModernMech 3 hours ago [-]
I dunno, you'll notice several people here saying that got value from it, despite others not engaging with it at all. If your filter is that you don't engage with anything you don't expect a priori to get value from (based on weak signals), you'll miss a lot.
NetOpWibby 26 minutes ago [-]
This is a damn good project. Makes me want to make headway into an idea I've had for quite some time P2P search...we'll see.
cpill 7 minutes ago [-]
I was thinking that a search engine that ignores anything with advertising on it would be useful. This has inspired me to give it a go (on the weekend even)
eggbrain 6 hours ago [-]
This is actually where I see software going in the short term -- cloud moving to local.
A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search.
But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it.
The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally.
dylan604 5 hours ago [-]
The number of times we've gone from cloud/server access via terminal to local compute back and forth is something that always makes me laugh a bit.
an0malous 5 hours ago [-]
It's more of a pipeline than a back-and-forth. New abilities happen in the cloud first because they require specialized, higher capacity resources and then move towards being local as the resource usage gets optimized.
dylan604 1 hours ago [-]
Outside insane GPU appliances, I really see cloud based things as a lock-in to subscription based offerings. The fallacy of always using the latest code isn't worth never ending subscription fees. I'll install it locally. I'll keep my data locally. I don't need to wait for things to upload/download. If there's an update that feels worthy, I'll purchased and install locally.
fg137 4 hours ago [-]
Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.
dreamforever 3 hours ago [-]
It's not for that, sorry, I should have been more specific. It's for people who wanna put in the effort and steer their own crawl to surface their own slice of the web. The article is just a little story of the journey
prepend 3 hours ago [-]
I was wondering the same thing. I’ve wished for just a big blob of the web to grep and regex through, but I don’t think this is that much easier than using duckduckgo or even google.
lagrange77 3 hours ago [-]
Took me a few minutes to realise it's not a domain name search engine.
pimlottc 5 hours ago [-]
Sometimes I think people forget how capable computers are. 500k is not much. You can just slap that in a Lucene instance. This is a solved problem.
marginalia_nu 5 hours ago [-]
Approaching search by just tossing the data in Lucene is how you end up with Confluence's search box though.
orliesaurus 6 hours ago [-]
Like a personal Google? How do you bypass all the captcha, ip bans, cloudflare turnstile antibot stuff etc?
sandeepkd 5 hours ago [-]
Thats the fun part, the user just went with happy path. Javascript, captchas, cloudflare protected content did not made to the catalogue. This sort of use case exists in LLM training data a lot which makes it easier. The data gathered by the user is not really practically useful cause there are way too many gotchas when it comes to web scraping and building a catalogue (source: I have done scraping for a particular domain data and had to do at least 10+ iterations to get it >90 right)
voidUpdate 5 hours ago [-]
They don't: "skips the model entirely if the page is empty, parked, or a bot-challenge wall"
dreamforever 2 hours ago [-]
despite this limitation, there is still some good stuff out there, and with the priority steering, you can focus compute on what you actually want, fast and cheap.
BaudouinVH 4 hours ago [-]
How do you build a list of domains you want to index ? I see there is a fetcher and a spider in the code but so for I haven't found how to build that list.
dreamforever 2 hours ago [-]
Ah, I forgot to mention that anywhere. You have to provide your own. You can start from a small set, like 10 websites you like that have a bit of character, and it will also add any domains it finds from those 10
pavel_lishin 4 hours ago [-]
From the screenshot, it's very funny that one of the indexed sites is www.llresearch.org, which looks like it's run by a crackpot.
dreamforever 2 hours ago [-]
There are all kinds of websites in here lol. There is some gold in here and I'm determined to surface it all. I had to wrap this up without full analysis cos it was dragging on
dewey 6 hours ago [-]
I think Kagi Small Web filter would give you very similar results.
dreamforever 2 hours ago [-]
I'll check them out!
dreamforever 7 hours ago [-]
Check out my latest project! You can fork it, tweak the policy manually or with AI, run the system and watch the data come in! It's engineered to keep a low data footprint, so 500k domains fits into 1GB on disk. If you have local models it's free! You just might not get the best throughput depending on your GPU. My production data is not exposed anywhere yet, and I may never expose it. The point is for you to fork and make your own policy, and thus your own personal search engine! The article covers basic analysis on my data, so it's worth a read if you're interested! A deeper analysis may arrive with V2 if I ever do it
elorant 4 hours ago [-]
Domains are way more than just 40M though.
BaudouinVH 4 hours ago [-]
From what I understand the aim was not to collect all the domains on the web but focus on personal website, etc. and avoid corporate web sites.
1. read each site
2. rent a 4090 with https://vast.ai to run vllm
3. let llm model invent its own category and tag names freely
4. save 1KB of metadata each
5. `code is going up as open source` soon (TM)Your impressions seem about right, but there are a few control steps it seems.
Since when has low effort become a selling point anyhow?
Checks socials and trademark too
I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
Where do you get stats on this info? Not refuting your claims but it's interesting data. I would've thought a wider gap in all biz to active websites since some countries run nearly all their biz on whatsapp/telegram/wechat.
What a homebrewed solution lacks in coverage it excels in indexing and serving a small slice of the internet really really well.
Almost anyone can put together a basic search engine in a few thousand lines of code, it's just not very hard to make a program that will index a few million documents better than Confluence.
Then, between that first ansatz and a working scalable internet search engine, you have a pile of interesting problems touching every aspect of computer science and computer hardware and networking, enough so that hundreds of people will have gotten PhDs in narrow sub-problems of those problems you'll be facing.
It's great because you can just tackle the stuff you feel comfortable approaching and leave the rest for later.
I don’t care if robot hand written this or black or white. It’s useful.
Does the program cheat? Does it just query a table is saved in some backend? Maybe, I don't know, you don't know, and since the author only pressed enter once no-one will know until/if someone runs into any issues.
When you can create information for free, and you want people to use this information you need to provide guarantees, proof, and understanding beyond what the autonomous model has done. Otherwise you just move the burden of verification to the reader.
Just post the part you did write - the prompt.
AI is a great tool, but it is somewhere between a 3D printer and a CNC machine. it can make something smooth and easy to handle and maybe good enough for personal use but you would want to run it through some Finishing steps before giving its output to someone else. In some cases thats using the output as a mold for a full recreation and other times maybe its just sanding it a bit, smoothing out some rough spots and putting paint on it.
youre right not to care about if something was made by a human or a tool, but the full presentation of the content including editing is part of the content when it comes to a write-up.
It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."
- every night a good number of neuron connections in your brain are automatically severed because you did not use that ability
- over long periods of time, it ll take you to a point where you wont be able to write a simple factorial program on your own without gpt telling you
- i am not saying this to incite you or mock you or anything. i am just concerned about how much of your thinking you are offloading
If you think that having slaves means you don’t need to be a good administrator, then you are sorely mistaken.
It requires more brainpower than it ever did to manage a project the size what used to do a 100 people team. All on your own.
The fact that I am arguing with all of you here and not blindingly follow the narrative should tell you all you need to know about my critical thinking prowess.
No it's not.
People are increasingly getting fed up with slops like this -- you can see comments in HN discussions.
Many people just completely skip those articles.
There’s people at work whose messages get completely ignored after too many times of posting verbose, useless, ai slop.
Why claim 'people' (plenty of people get value from satisficing) and even try to use 'see comments in HN discussions' as if there is something that actually documents that ie 'we all think it'.
I skimmed it and even skimming it I got value from (what some others here) call 'slop'. (The value was basically it gave me an idea of something I could implement and again I didn't have to study it or even understand everything (or agree) just seeing someone did that made me think about something I might want to do.
It’s extremely rare that someone smart with something smart to say produces this crap.
Like someone said upthread if the author isn’t willing to spend time writing cogently then I don’t think it’s worth my time to try to parse it.
The content itself is slop.
surely if we're expected to read this ourselves, the author can write it themselves?
indeed, it's helpful to the author, too: writing helps you learn
A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search.
But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it.
The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally.