I completely see the benefits of using guix, transactional package management seems at the very least an interesting alternative to docker in some scenarios, and could be very interesting from a tool automation perspective.
I don't want to sound negative, but I think the biggest problem with getting me or other ruby developers to package our gems to guix would be the following. The bundler/rbenv system I've been using for years is pretty darn good, personally I haven't had problems with it. I've used it in massive and small projects, and admittedly the only inconvenience is having to manually or through orchestration install the dependencies of certain gems, of which there are also only a few (mysql2 with libmysqlclient-dev and his nokogiri mention are probably the most notable). He does point this out but if you're using docker or some automation tool this is a pretty minor point.
Also I already ensure my gems run their tests on mri, jruby, rubinius, etc. and this seems like extra work for little benefit, especially when most in house deployment scenarios are running with docker or chef/puppet on top of the classic rbenv/rvm and bundler setup. My biggest question is after upgrading a gem version how easy is it for me to push the new version of the package to guix?
So essentially, I wish this article had more arguments to show why guix will better the life of ruby devs.
I don't know if it's the same with Guix but in NixOS it's just a matter of running `bundix` and all the Gemfile.lock dependencies are converted to a nix expression. Gem developers can keep developing as usual.
Maybe the only restriction is to make sure the gem has configurable paths if it tries to access system files.
I'm experimenting with it at the moment and converted my blog to be managed by nix on OSX. It's not a walk in the park but it's working. Clone the repo ( https://github.com/zimbatm/zimbatm.com ) and type `nix-shell` (needs to be installed first) and you should get all the blog dependencies available in your environment.
> I don't know if it's the same with Guix but in NixOS it's just a matter of running `bundix` and all the Gemfile.lock dependencies are converted to a nix expression.
I'm glad to hear that the infrastructure is working for you. There are, admittedly, some kinks on master, though I have a PR to make things much, much more robust:
Unfortunately, while Bundler is generally a great, useful piece of software, there are aspects of its design that make it very difficult - borderline antithetical - to cooperate with packaging; if you look at the commit message in the above PR, you can see a thorough description of the challenges.
I have an open PR for Bundler to try to make things a bit more amenable, and recently got the thumbs up with respect to the general idea:
Unfortunately, most Ruby deployments are quite imperative in nature - just push to a git repository, bundle install, and then bundle exec (that's leaving out all of the subtle prereqs: `apt-get install ...`, etc), as with Capistrano. There doesn't appear to be as much of a culture around packaging (yet), and so there hasn't been a demand for the tools to meet those needs.
I hope that we can push through those problems and make it trivial for people to enjoy the ease/dependability/resiliency of package management, as many people have no idea what they're missing out on.
Nice ! That should solve the issue where all gems are re-installed whenever the Gemfile.lock is changed.
I agree that bundler is mainly focusing on the user interactions and is good at that but never look inside. I'm not even sure if the dependency resolution mechanism can be used as a library.
In my opinion bundler should only be used as a dependency-resolution manager. Once gems are installed then they should use good-ol' $LOAD_PATH. Unfortunately there are the git gems and also rails which insists on using bundler.
EDIT: got any thought on adding system dependencies per gem ?
As someone who deploys Ruby software to our servers, I've come up with a formula that's served us well.
We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm.
In other words, the packages are completely self-contained; no gems are used from the environment.
The key to this is the --path option to bundle install, i.e.:
bundle install --path vendor/bundle
This command makes sure all my gems are in a sub directory of my project called vendor/bundle. When creating the debian package using fpm which includes all the source code, vendor/bundle also comes along with the package.
After installing the app on our Linux servers, 'bundle exec rails server' uses the gems from vendor/bundle.
This approach does require that I create the debian package on a machine that closely mirrors the deployed servers, so that compiled gems make any sense.
Most all the issues described in the article have not been problems because of this approach. I used to use rvm as part of my build process; it was very brittle. I'm completely happy now; just wanted to share in case that helps someone else.
>We take all gems associated with the app (like a Rails app, but we do it with others), and stuff them along with the code into a package created by fpm.
That is exactly what I've started doing at work, actually. It's a short-term win and rather simple, but it's a pretty big hack. Package bundling is quite poor form as it leads to file system duplication and for each duplication you have an additional location to patch when a security vulnerability is inevitably discovered. Ideally each Ruby gem would be its own package. I'm also concerned with reproducibility, so I build packages in as isolated of an environment as possible where I run 'bundle install' with a fresh vendor/bundle directory. Builds take longer than they should because of all the re-downloading. Using Guix, where I can easily represent each Ruby component in a package object, I can take advantage of its content-addressable storage to save time on repeated builds by using the cached gem builds from previous times.
We use a 'monorepo', so everytime we update any gem dependency, all of our apps get rebuilt entirely. So, updating all of our packages or not isn't a problem.
And I like the duplication, believe it or not. After having debugged environmental problems with dependencies in various languages over the years, I'm happiest knowing the dependencies in question are in vendor/bundle; end of story. Sometimes, in case of a nasty bug, I have a one-liner fix and I can go to vender/bundle, tweak the gem, and know I only affected the app using it. Those are things I prefer over some lost hard drive space.
Build times are a little long for us, though. That is true. I do a fresh bundle install on every build. That is the second longest part of the build, behind running our dog slow web tests.
The duplication is less of an issue in the corporate, one application per VM environment. However, my focus isn't solely on that. Web applications are really hard for most people to self-host. They are made a bit easier by things like OmniBus packages that bundle absolutely everything, which makes the user dependent on each application author to ship security fixes to software they didn't write.
There's also the issue of stateful package managers that can break in the middle of a package install and you're screwed. Most of the time that doesn't happen, but I've been bitten before. So the fpm approach is a great short-term win and I want to roll it out to the production systems at work soon, but in the long term I think we need functional package management to make our systems more resilient to failure.
What is exactly wrong with piping curl into bash? I'm trusting RVM anyway. As for functional package managers, I'll believe it when I see it. The reason things are brittle is that any time something in the middle of a dependency chain changes the whole chain may break. The more you freeze things, the greater chance there is to freeze up a security hole (see what is going on with Docker). So you have two choices: be ok with the freeze, or update the whole chain. In this respect I prefer the latter, and I especially prefer the way Ruby does it where, due to the global namespace, I'm forced to upgrade the gem everywhere so I end up fixing things and keeping the whole project up to date. While this seems frustrating at first, once you realize that everyone in the Ruby community is doing it, you realize that the reason it is usually painless to update your gems is because other people have fixed those other holes. What is especially nice is that you only have to remember things that are broken at T0, unlike, say, Node, where a popular nested node_module may have 10 different versions in your dependencies and you keep hitting the same bug or security hole over and over again.
I've found all the hate surrounding this practice to be pretty misdirected. As long as it's served over https, it's functionally equivalent to "download and run this random package installer!", which is generally tolerated.
I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications, but I would very much play with it if it were possible to do something like this:
guix virtualenv myapp
cd myapp
guix activate
guix install postgres # installs a local postgres installation available only on this "environment"
guix environment -l myapp.scm # install app dependencies
<do work>
guix deactivate
This would be a killer feature for me and I would ditch virtualenv, pip, npm, perhaps Docker in a heartbeat.
> I'm not sure I want to install guix on my main Archlinux box, as I don't want conflicts with the existing libraries and applications
That's the beauty of functional package management: no global state is modified. Whatever Guix does is restricted to its own namespace, e.g. the /gnu directory. Every package that is built or installed ends up there and is completely isolated from the rest of the system.
To actually use the software you use a link to a profile (which is located in the "local state" directory of Guix).
You can install packages into separate profiles with Guix:
Yes, the 'guix environment' tool is like a language-agnostic virtualenv. Guix does not use or interfere with any components from the host distro, so you can install Guix on top of Arch Linux without fear. We have a binary installation method[0] that isn't too difficult that you can try out. I use Guix on top of an Ubuntu host at work.
For your PostgreSQL example, you'd still need to do things like initialize the DB cluster and start the daemon on your own (the GuixSD distro has a system service for this[1]), but to spawn a shell with the postgres daemon and client programs available you would run:
guix environment --ad-hoc postgresql
Exiting the sub-shell would do essentially what you had in mind with 'guix deactivate': Make postgresql available for garbage collection (via 'guix gc') if/when nothing else is using it.
The point was to install the deps into my project tree, instead of putting everything into /gnu, but to think of it that's not actually much of a problem, it's just personal preference/bikeshedding.
"Installing" a package with Guix really just means that it is referenced through a link. /gnu is just a cache for "evaluated" package recipes. You could create a profile that contains your symlinks to the items in the store wherever you want, including the currenty directory.
guix package -p $(pwd)/.guix-profile -i what ever you want
Programmes in this profile can then be executed like so:
./.guix-profile/bin/what
But that's not necessary when you could just use `guix environment` and enter a shell where everything is already hooked up.
Notice the striking similarities of package management definition files and configuration management, where the latter does more related to state, coordination, attributes and templating files. Packages are just more generic containers of resources and metadata, configuration management makes those resources concrete on specific systems and applies tweaks to converge on the desired catalog state.
Btw, has anyone written a usable Guile-based cfg mgmt tool yet?
>Btw, has anyone written a usable Guile-based cfg mgmt tool yet?
That would be Guix. We use the same primitive utilities for package management and system configuration management. I'm typing this from a laptop running the GuixSD distro, and if a system upgrade were to break things, I can roll-back to a previous, working generation of the system at boot time and keep on going.
Configuration management is slowly coming to Guix. Guix is the foundation of the Guix System Distribution, which is a system instantiated from a declarative system configuration file /etc/config.scm. There is work under way to extend this to managing multiple machines.
I like the approach of using gs (simple gemsets) [1] and dep (simple dependencies) [2]. I've been using this approach of handling dependencies for a while now.
I've written a simple shell script as a replacement for gs (since I constantly forget if I was in a sub-shell or not). All it does is change $GEM_PATH and $GEM_HOME when $PWD changes (while staying in the same shell). If the current $PWD has a directory ".gs", `gem install` will install them there instead of globally. All I have to remember is to create a .gs directory when I start a new project.
It does not address all issues highlighted in the article, nor do I know how well this would work in a production environment with automated deployments. It's probably more of a work around than a solution. I vastly prefer it for personal projects, though. Hate solving gem conflicts with "bundle exec".
Instead of changing environment variables when $PWD changes, I went the other way: make it obvious when you're in a subshell by strapping $VIRTUAL_ENV onto $PS1. Again, simple shell scripts win[0][1].
I'm still interested in a cross-language approach, but I'm suspicious of anything which requires an explicit import step for rubygems that doesn't involve resolving the dependency tree. Unless Gemfile.lock is involved, it's going to need to copy Bundler's resolution algorithm, which seems fraught.
>but I'm suspicious of anything which requires an explicit import step for rubygems that doesn't involve resolving the dependency tree.
The current 'guix import gem' utility doesn't do dependency resolution at all. It just generates some Scheme code for you to start with and you fill in the gaps. It's important to understand that Guix packages specify the exact versions of its dependencies. That includes not only the version number, but its explicit dependencies, the Ruby used during the build, the GCC used for native extensions, the full build script, etc. All of these various inputs to the build function are reflected in the SHA256 hash that uniquely identifies that package.
The loose versioning that Bundler uses is indeed very flawed and prone to strange errors. Guix simply doesn't have this issue.
I do something similar to this. My right prompt shows a small fork symbol when I'm in a sub-shell (based on $SHLVL). However, this does not help me much when I'm several levels deep, which is why eventually I started hooking in on changes to $PWD.
no 'ruby version manager', eg 'chruby, rvm, or rbenv' make any sense in the nix/guix world. the article would have done a disservice to itself to go into detail on each one, when the goal was really to show how to manage ruby projects in guix.
As someone who tried RVM for a while and until it's problems became obvious, and lived (very successfully) with rbenv as an escape for a long time afterwords, I have been liking chruby:
It doesn't have most of the problems of RVM/rbenv, and mainly provides an easy way to switch between rubies while leaving everything else as full-path references instead of the "magic" tht RVM tries to provide.
That said, I really like the local $GEM_HOME method, and guix sounds like an interesting solution.
Thank you. I think the benefits of Guix start to become even more evident when you work in a multi-language environment with god knows how many package managers. We don't yet have all the packages that people need, but that will happen in time if other people see value in Guix. We will see.
I imagine everything said in the article also applies to the Nix package manager[0]. One of its nice features is the Nix expression language (aka Nix) [1]. It has an elegant mixture of purity and pragmatism. I imagine guile scheme would have a lot more integration opportunities outside of just system/build/configuration management. It'd be interesting to see Guix expressions that actually mix other scheme libraries or concepts, rather than just taking Nix expressions and removing the syntax.
I very much like how Guix is just a Guile Scheme library and can itself be integrated with other Scheme code. There is, for example, guix-web, a web interface to the package manager; there also is an Emacs interface using geiser.
Using Guile has turned out to be a great feature in itself and it is still fueling the development of new features that otherwise would be awkward to implement, such as the new "guix graph" command to visualise dependencies.
>It'd be interesting to see Guix expressions that actually mix other scheme libraries or concepts, rather than just taking Nix expressions and removing the syntax.
We already do this. Our build scripts are written in Guile, not Bash like in Nix, and they take advantage of Guile's standard library which has things like a pattern matcher, an HTTP client and server, a POSIX interface, a foreign-function interface, an XML parser, etc. We use third-party libraries such as guile-json and guile-charting in some of our tools, as well.
> I'm not sure how you feel, dear reader, but my Ruby environments feel like one giant, brittle hack
Of course things are not perfect (and guix may have a few cool ideas), but I think we should keep in mind that both rvm and bundler have been huge improvements.
The hacks people needed to come up with before rvm and bundler existed were an order of a magnitude more brittle.
Yeah, while I think the approach outlined in the post makes sense and could be really great, I really haven't had any of the problems outlined since the bad old `config.gem` days. In my experience, everyone defaults `~> X.Y.Z` dependencies, with more lenient patterns justified by actual testing, and I can't remember the last time I saw a blanket `>= X` requirement. This has meant that all, or very nearly so, of the dependency problems I find myself handling are due to actual incompatibility, which I'm glad were caught.
I don't want to sound negative, but I think the biggest problem with getting me or other ruby developers to package our gems to guix would be the following. The bundler/rbenv system I've been using for years is pretty darn good, personally I haven't had problems with it. I've used it in massive and small projects, and admittedly the only inconvenience is having to manually or through orchestration install the dependencies of certain gems, of which there are also only a few (mysql2 with libmysqlclient-dev and his nokogiri mention are probably the most notable). He does point this out but if you're using docker or some automation tool this is a pretty minor point.
Also I already ensure my gems run their tests on mri, jruby, rubinius, etc. and this seems like extra work for little benefit, especially when most in house deployment scenarios are running with docker or chef/puppet on top of the classic rbenv/rvm and bundler setup. My biggest question is after upgrading a gem version how easy is it for me to push the new version of the package to guix?
So essentially, I wish this article had more arguments to show why guix will better the life of ruby devs.