It's hard to put this into words succinctly. But when I was a kid before the Internet, the library was the main source of human knowledge, and it was very organized through the Dewey Decimal System which imposed a kind of tree-like hierarchy across a wide spectrum of topics. And while I imagined that one day this might all wind up served by computers, I thought this organization would survive the transition. But it didn't.
I suppose in the early days of the Internet, they tried? Yahoo was kind of an Internet directory in the beginning, and the Whole Internet catalog was another such effort. But these eventually fell apart and we had to resort to search engines to find anything. Frankly, it's a bit like the early days of personal computing when we used flat file systems instead of hierarchical ones with proper directory trees. When you were limited to what could fit on a floppy, this wasn't such a big burden. But the Internet as it stands might as well be a giant flat file system.
Today, even the search engines are failing us, and we are turning to AI. In the pre-Internet era, we had sort of an equivalent to this also. They were called librarians. But a librarian's job was made easier by the fact that all the books were carefully organized by topic. For AI, it's as though the library were just a massive pile of books tossed around haphazardly and the librarian had to make sense of it enough to pull whatever you're looking for out of the chaos. Is it at all surprising, then, that it takes a huge amount of energy and computing resources to make any of this work?
Apologies if I'm reading too much into what you've said here, but it reads to me as if you're implying search engines couldn't keep up with the volume of the internet and AI is the logical next progression.
Search engines actually worked pretty amazingly at one point, but then were kneecapped because Google et al were more interested in advertising money than useful search results. Then when they needed an excuse to justify ludicrous spending on AI, they shoehorned AI into web search where it is very much the wrong tool for the job if you want accurate, high relevance results.
It's more like the librarian has taken it upon themself (for some reason) to write a book that is the mathematical average of every other book in the library. Then when any patron comes looking for a specific book, the librarian just hands them this slag instead hoping they won't notice the difference.
Now that said, I 100% agree that running a good "old-style" search engine does still take a huge amount of computing resources, but I'd be surprised if the requirements were anywhere near what is being thrown at training LLMs.
save