LLM generated websites are so instantly recognisable
Vibe coded tools are great. I have loads of them to work for my workflows. AI makes it almost free to make your own tools. Almost none of them are worth sharing or distributing
OK, so, I too have built a local index tool called slopdex [1].
It's very simple, parses the repo code and docs with treesitter, indexes it, vectorizes everything, then allows you do things like create a map of the codebase or do a semantic search.
One cool trick that does work well is the cross-search command, which allows you to do a duplicate code search using the vector index. This gets rid of a blind spot that lots of LLMs have that make it create lots of dupliclate helper functions. This works quite well in practice.
It has a few evals that I'm still working on, showing only modest improvements in token usage when using the map command, maybe around ~10% less. I think this can depend a lot on the type of task though.
https://github.com/huggingface/funes/ has been my go-to here (specifically for conversation search), but I do appreciate that there's a frontend on this one that can recall a session and open the correct agent for it (rather than my opening one agent, then opening another when I've gotten it wrong). Looks nice. Well done.
I feel a lot of these tools are barking up the wrong tree. Solving problems that are just symptoms of a deeper root cause. Index because my codebase is too big to grep. Index to paraphrase hard_decline via card retries. Reduce tool calling to save my agent YOLO search loops. etc. etc.
It feels like trying to overcome bad codebase hygiene by throwing more tools at it, and wallpapering the mold over. I've found that codebases left entirely to LLMs suffer from this gravely, but with even a bit of effort - you'll get outsized returns. Don't let the agent smosh up everything into a giant ball of code. Separate concerns, build services, do the architecture step of the software design with a bit more care than vibe coding.
A bit of a rant, yeah, but I don't see why tools like this exist (and ask you to curl a script into bash, no less). You're doing stuff a bit wrong if grepping is the bottleneck.
I mostly feel the same, but it'd be really interesting to measure it, I think. Like, how many tokens did it cost in a codebase of quality Q1 vs one of quality Q2 to `grep` vs some semantic search. And did you find the correct `DuplicateSymbolName` in the correct namespace?
I don't think most people have control of the code base they're required to work in.
I too have converted a code base I made to typescript and peppered it with AI docs to make visibility easier.
I've converted i18n files to a k:v[] arrays per language because the AI is shit at working with objects. The ability now for it to just find/insert additional languages is well improved.
There definitely are shapes in code that the AI will have trouble with.
But again, I don't think most people have control over the code they maintain.
> Cursor had something like this at some point. They've removed it.
No, they say that they have removed it, but they haven't. Cursor claims they retired local indexing in favor of something they call "instant grep," which, according to their own support, doesn't use a local index, and they removed the tool in Cursor settings to rebuild the index.
But Cursor still uses a local index. They have just hidden it as a subfolder in your ".git" folder. "~/.git/cursor".
Their hidden index often goes stale, and I had to nuke that folder every time I changed many files. This isn't new; when they officially had a local index, it was often stale too. I developed a script that ran on a daily schedule, every few hours, to alert me when Cursor's index went stale.
I constantly had to watch for Cursor suddenly not finding documents that clearly existed, and it often worked on outdated content. Disabling instant grep removes the hidden index, btw and problem gone.
Hah yes. Ill throw my own into the ring too https://github.com/boyter/cs although its a lot older than most other tools having been started 7 years ago.
Cursor was an interesting case where they were maintaining an index. I think everyone has converged on not doing that, which is where I started, with BM25 with some tweaks on top.
Notable link related to this https://www.youtube.com/watch?v=ZnmtJKFonLk where Jo Kristian Bergum (notable in the search space) agrees with BM25 being a good approach for agentic workflows.
I don't know if you've noticed, but when Claude Code, Codex, or any other agent needs to find code, it runs grep. It searches your files for an exact string, reads a few results, and greps again until it finds what it wants. You pay tokens and wait for every round.
That worked fine when repos were small. But codebases are growing fast, partly because agents now write so much of the code. Every grep round costs more than it did last month, whether you notice it or not.
There's a second problem: grep only finds what you already know how to name. Ask it for "where we handle rate limits from the billing provider" and it gets 0% right on the first try, because it needs the exact string.
So I built Pensieve. It's a local index of your code, documents, and past agent sessions. Any agent can query it over MCP and get the few files that matter on the first try. A few cents saved per lookup doesn't sound like much. It adds up over hundreds of lookups a day.
> So I built Pensieve. It's a local index of your code, documents, and past agent sessions. Any agent can query it over MCP and get the few files that matter on the first try
What are the various ways Pensieve is superior to iterative grep? Is it just better "semantic" search instead of literal "grep"?
Cache invalidation is hard - how does Pensieve address it?
Your site says "Pensieve watches your folders, repos and agent sessions and re-indexes what changed. Uncommitted edits count, and every worktree is tracked by branch, so your agents never work from stale context." but you don't define what "stale context" is - if I'm exploring 3 different implmentations of the same concept across 3 different branches, which one is the "stale context"?
Could you take a few minutes to explain the what and the how? Links to writeup to blog posts or files in your repo would be very helpful.
Finally, can I use Pensieve with Paseo? What if I wanted to expose Pensieve's capability to my Paseo setup?
I also built something similar to solve this, but more aimed at inspecting the impact of changes to a concept - think a vertical slice or domain, or even a feature. Often the same thing might be referred to by similar, but slightly different names, or separate pieces of the codebase might not immediately look related when they do share dependencies. So symbols, phrases, and words are attached to concepts for either LLM-based or user-based querying via MCP.
An indexer crawls the codebase and maps slices of code to concepts, then groups related concepts. You can inspect a concept to see things related to it, or ask about the impact of a change and see second-order changes required - things not so easy to see that are related by basic symbol linking (although it does use symbol linking - it heavily relies on Roslyn, since it's all C# focused).
It also returns confidence scores with everything, which adjust based on a dozen or so factors like whether the user defined the concept or the concept was auto-generated during the code crawl. Low confidence score === fall back to grep.
It was a super fun project and ended up being very useful, as much for people as the LLM. But, I'm not convinced it is materially better than grep in some ways and just haven't had the time to really focus on improving it.
I believe HN now has some built-in detection for AI generated comments, which is probably why it was dead (pangram rates it as 100% AI generated, btw).
Vibe coded tools are great. I have loads of them to work for my workflows. AI makes it almost free to make your own tools. Almost none of them are worth sharing or distributing
It's very simple, parses the repo code and docs with treesitter, indexes it, vectorizes everything, then allows you do things like create a map of the codebase or do a semantic search.
One cool trick that does work well is the cross-search command, which allows you to do a duplicate code search using the vector index. This gets rid of a blind spot that lots of LLMs have that make it create lots of dupliclate helper functions. This works quite well in practice.
It has a few evals that I'm still working on, showing only modest improvements in token usage when using the map command, maybe around ~10% less. I think this can depend a lot on the type of task though.
Would be happy to hear any feedback.
[1]: https://github.com/ninjaxtools/slopdex
If not for the rest of the page I would have assumed this was a joke.
It feels like trying to overcome bad codebase hygiene by throwing more tools at it, and wallpapering the mold over. I've found that codebases left entirely to LLMs suffer from this gravely, but with even a bit of effort - you'll get outsized returns. Don't let the agent smosh up everything into a giant ball of code. Separate concerns, build services, do the architecture step of the software design with a bit more care than vibe coding.
A bit of a rant, yeah, but I don't see why tools like this exist (and ask you to curl a script into bash, no less). You're doing stuff a bit wrong if grepping is the bottleneck.
I too have converted a code base I made to typescript and peppered it with AI docs to make visibility easier.
I've converted i18n files to a k:v[] arrays per language because the AI is shit at working with objects. The ability now for it to just find/insert additional languages is well improved.
There definitely are shapes in code that the AI will have trouble with.
But again, I don't think most people have control over the code they maintain.
Referencing other sessions is a cool idea tho
No, they say that they have removed it, but they haven't. Cursor claims they retired local indexing in favor of something they call "instant grep," which, according to their own support, doesn't use a local index, and they removed the tool in Cursor settings to rebuild the index.
But Cursor still uses a local index. They have just hidden it as a subfolder in your ".git" folder. "~/.git/cursor".
Their hidden index often goes stale, and I had to nuke that folder every time I changed many files. This isn't new; when they officially had a local index, it was often stale too. I developed a script that ran on a daily schedule, every few hours, to alert me when Cursor's index went stale.
I constantly had to watch for Cursor suddenly not finding documents that clearly existed, and it often worked on outdated content. Disabling instant grep removes the hidden index, btw and problem gone.
Cursor was an interesting case where they were maintaining an index. I think everyone has converged on not doing that, which is where I started, with BM25 with some tweaks on top.
Notable link related to this https://www.youtube.com/watch?v=ZnmtJKFonLk where Jo Kristian Bergum (notable in the search space) agrees with BM25 being a good approach for agentic workflows.
Any insight into why would be much appreciated because those insights might hold the key to whether this is a usable, sustainable idea to begin with.
That worked fine when repos were small. But codebases are growing fast, partly because agents now write so much of the code. Every grep round costs more than it did last month, whether you notice it or not.
There's a second problem: grep only finds what you already know how to name. Ask it for "where we handle rate limits from the billing provider" and it gets 0% right on the first try, because it needs the exact string.
So I built Pensieve. It's a local index of your code, documents, and past agent sessions. Any agent can query it over MCP and get the few files that matter on the first try. A few cents saved per lookup doesn't sound like much. It adds up over hundreds of lookups a day.
Check out Pensieve!
What are the various ways Pensieve is superior to iterative grep? Is it just better "semantic" search instead of literal "grep"?
Cache invalidation is hard - how does Pensieve address it?
Your site says "Pensieve watches your folders, repos and agent sessions and re-indexes what changed. Uncommitted edits count, and every worktree is tracked by branch, so your agents never work from stale context." but you don't define what "stale context" is - if I'm exploring 3 different implmentations of the same concept across 3 different branches, which one is the "stale context"?
Could you take a few minutes to explain the what and the how? Links to writeup to blog posts or files in your repo would be very helpful.
Finally, can I use Pensieve with Paseo? What if I wanted to expose Pensieve's capability to my Paseo setup?
An indexer crawls the codebase and maps slices of code to concepts, then groups related concepts. You can inspect a concept to see things related to it, or ask about the impact of a change and see second-order changes required - things not so easy to see that are related by basic symbol linking (although it does use symbol linking - it heavily relies on Roslyn, since it's all C# focused).
It also returns confidence scores with everything, which adjust based on a dozen or so factors like whether the user defined the concept or the concept was auto-generated during the code crawl. Low confidence score === fall back to grep.
It was a super fun project and ended up being very useful, as much for people as the LLM. But, I'm not convinced it is materially better than grep in some ways and just haven't had the time to really focus on improving it.