This is not really a complaint, I was just trying to glue things together that are not super compatible with each other, all the stuff work nicely on their own. So this just be me rambling and adding in lots of questions. I think I’ll be touching on direnv, nixos, nix package manager, nix-shell, python packaging, poetry, cuda, pytorch, elf and shared libraries, asdf, cmake, linkers and loaders and idk maybe few other things.

This is a small note for myself on how I deal with the mess that is the python packaging ecosystem and development environment. In my experience development environment related to python is almost always fine, package management is where you get odd thoughts like, whether you should join your local sword training academy. To make all of this even more complex, I am doing this on NixOS(which I installed two weeks ago and I am not well versed in it) with Nvidia’s proprietary drivers which are not supported by the window manager of my choice(sway, now that’s a different story altogether, I honestly don’t even want to think about it, but good thing about nixos is that once you get shit working it’s hard for it to break again).

Okay, enough background.

Oh wait, we need to add more context. Other than this, I run arch on my laptop which I’ve been running since last 4 years with differing combinations for python environments and finally settled with `asdf` + `poetry`. Which kind of works, but also breaks sometimes because if the main python version upgrades, asdf can no longer use older version and now you need to download a older version from AUR. Basically I’ve been through hell with this and I still feel like I am in hell because last night I tried installing oobabooga/text-generation-webui and had to spend more than I like amount of time trying to get it up and running, in the ecosystem’s defense I could have used easy workarounds but man I feel like “okay 5 more mins and then i’ll give up”, and that 5 minutes turns into 12hours or something.

Anyway, I spent first couple of hours reading what are the different models, getting context on the whole scene etc. I am a total newcomer into all this LLM jazz, I wanted to explore because I recently started contributing to NGO and they’d probably need some LLM help down the line and also I wanted to explore this stuff for too long now. My PC has a video card that can run the smallest quantized LLMs which would give me an general idea how to play around with these things. And I don’t really want to rent things out just to try stuff out when I have the hardware at home.

Ah, now I think this is enough background. I apologize if things are written in haphazard manner because they are, If you’re even reading this I appreciate your time and would try to be more structured down from this point onwards , I am not thinking I am just writing.

ASDF + Poetry

Read about this here: Primary tool-chest : Python

NixOS and Python

My PC was running windows and I had to run something in WSL+CUDA. Boy that was painful, I decided I cannot live with this. I had to install linux, choices were Arch and NixOS, in hindsight if I picked arch I would easily save about one week worth of time. Sigh, I am not going to talk on the pain points that I faced, I also don’t remember. But good thing is everything is declarative and you can actually fit the whole system in your head this way. Pretty neat. There’s the thing called flakes, avoid it with all of your strength if you’re trying Nix for the first time. Also Nix is 3 things, a package manager, a language, and the distribution using both of those.

Now this is not about NixOS, this is about using Python with nixos. So nix package manager can be run independent of NixOS, the package manager gives you access to all nix packages in your favorite distribution, iirc you also get nix-shell with it which is basically the best thing ever. But when you’re using NixOS, it comes with the package manager with it obviously but the defining behavior of nixos is that it’s does not follow Linux’s FSH. See Nix vs. Linux Standard Base - NixOS Wiki.

So because of this, what happens? A lot of other things, but also happens is me having to take two smokes in a day when I smoke once a week. Stuff stops working, expectations break. Binaries mostly don’t work. Basically it forces you to do everything in the Nix way.

But thankfully there are best practices around this. So you don’t feel like, “ah I did this, but this is such a hack.”, which can be both good or bad depending on who you are.

So following are the setups I’ve tried so far.

No nix-shell, direnv and poetry

nix-shell, direnv and pip

• In this case you can easily just replace pip with poetry

nix-shell, pip, virtualEnvHook, no direnv

Other things about NixOS

Pytorch and CUDA

• Pytorch comes up with its own cuda but if you’re unlucky like me, you’ll try installing cuda just the week the pytorch release had a bug.
• Technically to install and run pytorch stuff, you don’t even need cuda stuff installed.

Resources

• https://gist.github.com/geekodour/683be9808893e86091e224c9cc7d9f28