Betting on tools I can control

On one hand I would love to ignore the ai hype. But, currently it is affecting my ways of working as software engineer and engineering manager.
I see many people who experimenting with LLM providers like OpenAI, Anthropic, etc. They provide pretty good tools and models are powerful (until they make them dumb to save money).
And I can offload some things to such tools. Not to increase productivity. I think it is not true. But, to offload some of routine digital tasks somewhere and reduce the context I hold constantly in my head.

While it is technically possible - there is still some worm in my head worries about the strong dependency I could rely on that eventually will get 20x more expensive, or will break, or will even damage me in some way.

I decided to invest more time into tools I could have control. And mostly it is to use open-sourced coding agents (e.g., opencode or aider) and open-sourced (technically open-weighted) models. Sure, these models not that powerful comparing to the latest models from LLM-providers. But, using them would give me consistency in quality, spending, usage.

Another things - I still believe we’re in acquisition phase when BigTech companies paying most of the cost. Eventually it will end. And we will have another AWS-styled tools where you get so locked in into the ecosystem so it is almost impossible to quit.

Recently I started to experiment more with ways to use self-hosted models.
First attempt was an integration of LM Studio with Alfred - https://github.com/RomanVolkov/lms_alfred_workflow It helps me to quickly load a model into local inference and use it. E.g. I am using it to summarise my transcribed voice notes.

Since I have Macbook Pro M1 Pro - it cannot hold bigger models, so I stick on something like 8B or 9B top. Recently I switched to Gemma 4 4B model since it gives pretty good results comparing to previous generation while being small (which means faster).

Unfortunately 4B models are not good for coding tasks. In my desire to still use self-hosted models I tried this: https://github.com/RomanVolkov/runpod_launcher. It’s a tool that helps you to use powerful GPUs on runpod.io and integrate them into opencode. It does it in a way only you can use the model instance. Plus, gives you a quick way to shutdown the instance to avoid big costs.
This changed how you estimate the usage of LLM. While popular providers limit you based on number of tokens used - paying for rented GPU allows you to use it fully and pay per time.
Paying per time is better for me in cases when I have e.g. 1 hour of focused work and I know I won’t hit any limits. I also know how to estimate the cost.

I believe building the workflow around open tools and models will give more benefit long-term than relying on the latest features from openai/etc. It does not mean I will ignore the latest changes in the industry, but my head worm will be calm if I start building my workflow around tools I can control.
This setup is less powerful, but also could be a bit more expensive if you compare with subscriptions. E.g., using NVIDIA A100-SXM4-80GB will give you about 15 hours on runpod.io costing the same as first subscription level of Claude Code. But, it gives you control and privacy. There is always a trade-off.

But again, I would prefer more control and predictability even with such an unpredictable tool like LLM.