What are people actually using these hosted search providers for? For personal use I've found using searxng-ai-kit [1] (disclaimer: I'm the author) or similar tools to be far more cost effective (free!). oh-my-pi uses a similar method and will scrape search results for you out of the box, no api key or search provider needed.
For OpenWebUI and also OMP with local models. Searxng is, uh, playing fast and loose[1]. More engines are implementing anubis or other challenges, which searxng attempts to implement/bypass, but they are also making these services inactive by default to limit load[2].
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
I have built my own search engine, which is really great btw. I self-host it.
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
Any organization that wants reliable latency, data processing guarantees (it's a big leak vector for company secrets), etc., will use such a search provider over the solutions you outlined.
Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.
e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]
Hosted search API from your LLM's provider is much easier to use, faster and less error prone. Also I believe that LLM may work better with its provider's search api, as it was definitely trained on it. I mean, it is just more convenient to use your provider's search.
And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.
Ceramic.ai is $0.25/1000 searches. I've been using it as an MCP in OpenCode and it seems adequate. I don't see many people talk about it, and I have not benchmarked it against Exa/Tavilly.
Kagi MCP works really well imo also within DeepSeek Harness. Just looking at cost neglects the possible difference in quality. How good is that DeepSeek web search/fetch?
is the cost of search and scrape provider is impact the quality too? for the search is yes, because there it shows the quality and depth of the search. but for scrape, wht the cons?
How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
I have always blamed Chinese AI operators for the hundreds of thousands of residential proxy requests I receive each day, but there's no way to be sure.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
Yeah, but how these residential proxies get in the residential areas? I would guess popular VPN providers are mostly behind this. They sell you VPN and sell bots residential proxy access on the other end.
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
From what I've seen, when operating in an environment with bash tools, Claude and ChatGPT will react to a site blocking their normal, attributed fetch tool by calling curl or writing a python script with a different user agent to try and bypass the blocks, rather than respecting the site owners' wishes.
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
Claude and ChatGPT are user agents. For obvious reasons, they serve the user, not the websites' owners. (Also, if you don't want people to be able to access some information, then don't put it online.)
they are tools. who do you think is using them? dogs?
your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.
Thank you for escalating the arms race that's currently destroying the internet. I can't look at the Apple user forums, my health insurance company's portal, pay my county taxes, browse Digikey and Mouser, view Github or whatever forge, etc. etc. without having to jump through Cloudflare or Anubis garbage designed to block your abusive behavior.
I'm just overflowing with joy that you're comfortable wasting my time and money.
Even if it's not the GP, if there's profit to be made, somebody will do it. I was reminded of the power of economic motives the other day, when it turned out someone bought an entire trustworthy reseller just so they could compromise the Ledger supply chain by including a tiny eSIM module that OCRed the screen and transmitted the seed.
Here’s what is says on their site:
“Courtesy backoff
Adaptive backoff keyed to target response codes. When a target returns 429 or 503, we back off immediately, so our fleet never applies denial-of-service pressure to a site.”
By the time you get the error you have already crashed the site.
Geez.
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
Thanks for sharing, wasn't aware of this provider. If anyone is wondering about rate limits for the subscription:
>We calculate search rates based on the timestamps of the last searches of a user (just the timestamps, not the queries or anything else!). We keep those timestamps for 12 hours, then discard them. We currently limit searches to 5 per 5 seconds, 15 per minute, and 120 per hour.
In other words, good enough for casual use if you were already going to subscribe. Assuming the rate limits are okay, Uruky’s €5/month becomes cheaper at these search volumes:
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
ScraperAPI and ScrapingDog. The latter also offers Google Search at a fraction of the official Google API price. Wrap their APIs into an agent/MCP tool et voilà.
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
Is anyone actually oneshotting a whole app or even vibecoding one in multiple steps without it being "guided work"? I really wish announcements (like DeepSeek's recent one) would stop focusing on the ability of the model to spit out Minecraft clones on demand.
Someone at work vibe coded a replacement for a entire third party SaaS that we were using in a week. Their version is better.
It wasn't one-shotted of course but I don't think there's much difference in the outcome between one-shot and like 100-shot over a week. That company is still doomed.
Think of something like Trello. Who is going to pay for that when you can easily vibe code something superior (yes you really can now).
I have one-shotted the two most recent relatively complex apps I've tried:
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
With Opus 5.5 you can vibe code a whole app without reading the code at all. Obviously you still have to tell it what to make.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
I am not disputing that the models are getting better for vibe coding. But I think people have very different standards of what a 'big mess' or 'maintainable software' is.
Opus 5.5 + Claude Code (or probably any agent harness; I don't think they make that much difference). I wouldn't say software engineering is done quite yet but yeah it's definitely never going to be the same.
If Anthropic or OpenAI fails, the other wins the US market, no? I’m oversimplifying but…that’s a pretty realistic possibility in the face of people thinking sinking one US AI company would hurt the greater US venture into AI.
> If Anthropic or OpenAI fails, the other wins the US market, no?
I am of the opinion that they both either succeed or they both fail. They are both losing money hand over fist and their model quality is pretty close to each other. There is not a whole lot that sets them apart.
I agree on the analysis, but not on the conclusion. What makes you think that the conditions you describe are stable grounds for a duopoly where both could be successful at the same time?
If they did then Google would take over, and others would form from their competent engineers. Also, they don’t need to be profitable to succeed, they merely need to hype investors enough to ensure that their losses matter less than their growth projections.
Oh come on, this post sounds like one of those twitter bots. No one seriously expected them to just give away billions in R&D for free, and they arent. The chinese model companies are actively moving away from the open source setup. Even state export controls.
> First, we should adhere to the principle of openness and win-win and boost innovation-driven development. As a new engine of world economic growth and an accelerator for the shift of growth drivers, AI is moving from the digital world into the physical world. We should seize this rare, historic opportunity to encourage open source, openness, collaboration and sharing. We should facilitate technological innovation, industrial development and scenario-based application of AI. We should make coordinated advances in the transformation and upgrade of traditional industries, the cultivation and growth of emerging industries and forward-looking planning for future industries, so that all sectors and businesses can benefit from AI
-- Keynote speech by Chinese President Xi Jinping at opening ceremony of 2026 World AI Conference
As a socialist myself, the above comment looks like a prime example of what Mark Fisher would call capitalist realism, the ideological belief that there is no alternative. The fact that somebody would think past quarterly earnings appears unthinkable to the committed ideologue.
They wouldn't give away R&D for free, they would give it away to destroy the capitalist mode of production, as outlined above. If that isn't a worthy goal from China's perspective, then I don't know what would be.
China isn't the US. They don't share the same ideology, strategy or conclusions.
As for the export controls, they explicitly exempt already-open models and set up an automated streamlined process for future ones.
They're about half a gen behind. Very impressive to get this kind of utility from a flash series model... the impact of RL seems to be behind much of the coding and agentic improvements lately.
Most of your local requests will fail (with Python/Curl/whatever) if you are not using a remote-controlled Firefox/Chrome. And even then you will be bitten brutally by captchas, rendering them basically unusable for mass requests. So it's better to outsource your web search to an external service with a stable API.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
That is a really good idea, in fact I’m doing something similar. I am charging anyone reading this comment 100 bucks per letter read, please tell me who I should send the bill to.
How does that work? I'm not a laywer, but I can invoice you $1 and the T.o.S of reading what I have to say are licensed under a $1 per view licence and by continued reading you consent to the terms of this license. Where can I send the bill for this comment?
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
I pay the Exa premium for semantic retrieval with far higher SNR, plus the academic index [1].
A field like objective can be up to 4096 characters [2] versus ori_question which is limited to 128 here.
[1] https://exa.ai/blog/publications-search
[2] https://exa.ai/docs/reference/search
[1]: https://github.com/nikvdp/searxng-ai-kit
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
[1] https://github.com/searx/searx/issues/1237 [2] https://github.com/searxng/searxng/pull/6696/changes#diff-e9...
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.
e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]
[0]: https://github.com/EratoLab/erato
[1]: https://github.com/EratoLab/web-search-mcp
[2]: https://github.com/EratoLab/web-access-mcp
[3]: https://www.linkup.so/
And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
Claude and ChatGPT are not people
your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
The last part is extremely dubious with some recent reveals.
I'm just overflowing with joy that you're comfortable wasting my time and money.
Incentives are crazy powerful.
Not always! With sufficiently powerful barriers it can be blocked at least in part.
Shaming awful people is one of available tools.
Or at least not praising them for making small profit at huge costs to others.
Please stop. The last thing we need in the world is more greed and selfishness.
Only strong regulation can help.
Or at least not praising them for making small profit at huge costs to others.
Also, it is useful as coordination method, enabling passing laws outlawing some behaviors.
(and I am not able to legislate-via-http while posting such comment is easy)
Competitor uruky is more relaxed about api and is cheaper. Creator is active on hn
https://news.ycombinator.com/user?id=BrunoBernardino
>We calculate search rates based on the timestamps of the last searches of a user (just the timestamps, not the queries or anything else!). We keep those timestamps for 12 hours, then discard them. We currently limit searches to 5 per 5 seconds, 15 per minute, and 120 per hour.
Source: https://uruky.com/faq
API: https://uruky.com/docs/api
In other words, good enough for casual use if you were already going to subscribe. Assuming the rate limits are okay, Uruky’s €5/month becomes cheaper at these search volumes:
1. Tavily Basic: 1,704 searches/month
2. Brave Search: 2,126 searches/month
3. Exa Fast / Auto: 2,233 searches/month
4. Exa Instant: 3,907 searches/month
5. DeepSeek Search: 7,503 searches/month
6. Ceramic: 22,507 searches/month
[1] https://github.com/kagisearch/kagimcp
There are plenty of use cases for the ability to quickly through together and refine an application without reviewing the code though.
It wasn't one-shotted of course but I don't think there's much difference in the outcome between one-shot and like 100-shot over a week. That company is still doomed.
Think of something like Trello. Who is going to pay for that when you can easily vibe code something superior (yes you really can now).
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
I am not disputing that the models are getting better for vibe coding. But I think people have very different standards of what a 'big mess' or 'maintainable software' is.
the moment there is a model which can code whole app, we can fairly say software engineering is done.
It's key to name the boundary conditions. Unlimited cost is a very HN assumption.
You are right that there's plenty of use cases where Opus 5.5 is better.
Not everyone wants Anthropic to succeed.
I am of the opinion that they both either succeed or they both fail. They are both losing money hand over fist and their model quality is pretty close to each other. There is not a whole lot that sets them apart.
That is explicitly the Chinese strategy.
Now, what if the largest capitalist economy went all-in on an AI that might no longer be sufficiently monetizable?
Oh come on, this post sounds like one of those twitter bots. No one seriously expected them to just give away billions in R&D for free, and they arent. The chinese model companies are actively moving away from the open source setup. Even state export controls.
> First, we should adhere to the principle of openness and win-win and boost innovation-driven development. As a new engine of world economic growth and an accelerator for the shift of growth drivers, AI is moving from the digital world into the physical world. We should seize this rare, historic opportunity to encourage open source, openness, collaboration and sharing. We should facilitate technological innovation, industrial development and scenario-based application of AI. We should make coordinated advances in the transformation and upgrade of traditional industries, the cultivation and growth of emerging industries and forward-looking planning for future industries, so that all sectors and businesses can benefit from AI
-- Keynote speech by Chinese President Xi Jinping at opening ceremony of 2026 World AI Conference
http://english.scio.gov.cn/topnews/2026-07/18/content_118605...
As a socialist myself, the above comment looks like a prime example of what Mark Fisher would call capitalist realism, the ideological belief that there is no alternative. The fact that somebody would think past quarterly earnings appears unthinkable to the committed ideologue.
They wouldn't give away R&D for free, they would give it away to destroy the capitalist mode of production, as outlined above. If that isn't a worthy goal from China's perspective, then I don't know what would be.
China isn't the US. They don't share the same ideology, strategy or conclusions.
As for the export controls, they explicitly exempt already-open models and set up an automated streamlined process for future ones.
Regardless of the performance metrics... why bad time?
Deepseek is more like a pet project with huge state backing, and they don't seem to compete with Anthropic in terms of profitability.
Anthropic is most likely not. Despite bilions of state funding and contracts.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
That is rarely the case (see sibling comment for reasons). Even open source harnesses will usually guide you to connect a hosted provider for that.
If you are talking about Gemini ( and other AI providers) no they don't they simply steel content.
I am asking to know if DS is any different because I failed to find this information on their website.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
I understand if you pay is a different story.
I mean ... "how does this work?"
I wouldn’t be able to use that API in any app with any significant number of concurrent users.
Or am I reading it wrong?
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.
I suspect they have spent months building caching infrastructure and designing intelligent fetch algorithms to get down to this number.
Or... CCP gave them a big pile of money to run this as a front for an intelligence operation. Honestly, who knows.