Cees de Groot

AI sure is a hot topic right now, and I see a lot of people arguing about it. To a lot of people around here, I’m the “computer person” they know and I get asked a lot about AI.

I’m going to suggest a lot of things can be true at once. For instance:

• LLMs are changing how we work and will continue to do so.
• LLMs are vastly over-hyped by vested interests, and may be in a bubble.

Or how about:

• Huge investment in GenAI is having many negative consequences, ranging from environmental to causing affordability problems in many industries that use hardware (ie, everywhere)
• Useful results can be had from models that run on local hardware, even battery-powered hardware, which may have negligible harm or even some benefit

And:

• GenAI is further concentrating wealth and power in megacorps, with the effect of squeezing out the smaller players even more.
• GenAI is lowering the cost of entry for people without a lot of resources already.

I have sympathy for the naysayers; those that say it’s nothing but a stochastic parrot. But I don’t have a lot of sympathy for the naysayers that deny ever using it; you can’t form a credible argument against something without having an understanding of it informed by experience.

I also have sympathy for the cheerleaders. I have seen some impressive things from AI; for instance, a story from an engineer who has a child with a rare disease without a credible cure. The engineer did a lot of research on it, started feeding research papers into AI to analyze, and the AI started finding correlations between different areas of research that humans hadn’t yet found — leading to a positive result for the child.

To be fair, I have rarely seen an AI deliver a 100% correct answer on anything with any real level of complexity. I have seen it both waste more time than it saves, and save a ton of time.

My point here is: It is neither always fantastic nor always terrible.

Let me talk you through an example.

I am a fan of inbox zero for email. That is, the inbox should be empty. Unfortunately, mine has 8000 messages in it. According to the oldest messages in my inbox, I last had inbox zero 8 years ago. But really, only a handful are older than 2020. I guess something must have happened that year…

I’ve been chipping away at this for quite some time now. The problem is, there are certain emails in there that really do still need some action – maybe it’s photos to save off into our photo collection, for instance. But when looking at things sorted by date or thread, there are old shipping confirmations next to phishing attempts and family photos. One can’t just scan down the list.

I’ve tried all the usual tricks, most of which involve selecting groups of message that are easy to bulk erase, or at least easy to scan visually for the occasional thing worth saving. Sort by sender or subject line, for instance. Then I can, for instance, delete all the old messages from the shopping sites I commonly use all at once. But then they start using different senders and different subject lines and that doesn’t get all of them. I’ve tried keyword searches for this sort of thing too. Still, that got me down to about 8000 messages.

So I thought: why not see if an LLM could help me classify these? Maybe it could categorize them, and then I could look at emails grouped by category.

I have one machine with a discrete GPU, an Nvidia RTX 4070. It’s a desktop machine I don’t use all that often. But I set up Ollama on it, running in a Docker container. Ollama runs models locally.

I should also mention at this point that we are solar-powered, and this time of year is a time of peak production of excess solar, because it is sunny and not much heat or AC is required. So that machine is solar-powered and isn’t causing environmental harm. In any case, charging the EV uses much more power than that GPU.

I figured I would do this in two passes. First, ask the LLM to classify each message (or a sampling of them would probably work too), letting it pick its own categories for each. Then, look at the patterns that emerge and give it a single, much smaller, set of broad categories to use and rerun it over that.

Then I can easily select messages from my Maildirs by category and process them in bulk.

I used open-interpreter pointing to that GPU on my network to help me write the scripts for this. It didn’t get things right on its own; for instance, it didn’t call the Ollama API correctly, and insisted on appending “/cur” to the path to the Maildir (which was not going to fly with Python’s maildir module). It took roughly an hour to classify those 8000 messages (or, as I had it do, the first 2000 characters of them), and then the same to do it a second time. I had it output lines in the form of “filename\tcategory” and hand-wrote the shell script that processed those.

In the end, was it useful? Yes, quite. Its classifications weren’t perfect (and it didn’t even follow my prompt perfectly; sometimes it would give me a long discussion on why it picked a certain category rather than just that category, and occasionally it picked categories not on the list). But then, neither were my manual keyword searches. So far I’ve gotten rid of nearly 1000 more messages. Several categories were a “visual scan for sanity and then delete all” sort of thing.

My emails never left my network. I didn’t rely on a cloud AI to process them. I didn’t contribute to global warming (this may have even been a case of saving energy, since it no doubt will offset quite a bit of manual time that would keep screens and room lights energized and so forth). I used about as much energy as watching a movie on a TV.

Did it complete the task for me entirely autonomously? Also no. AI isn’t a mind reader and it can’t possibly evaluate exactly what my thought process would be for a given task. But it can do a decent enough job to save me some time.

Still, this didn’t require hyperscaler datacenters. AI even runs on-phone (Google Translate being one of the most useful AI-driven apps I’ve ever seen, and it can run on-device).