Package / gem system?

[farleyknight]

Haven't worked on anything crystal in a while! Just wondering if you guys have any plans on doing a crystal package / gem system any time soon?

[asterite]

Hi! Yes, we thought some times about doing a package/dependency manager. We'd like it to not need a centralized repository and mostly use github, but we don't know how to do that efficiently, without having to checkout the whole repository. Or where to get the metadata for a package's dependencies. Or what format to use for file (json, yaml, toml?... ini? ... crystal?).

I think this is one of the most important things we need in order to build larger projects, otherwise it would become a pain to develop things in the language.

However, I'm not sure the language is ready to be rolled out. We are still changing core stuff like the IO api, some syntax details, and probably we'll change some semantic details. At this stage I would say it would do more damage than good. For example in Rust almost every project I download doesn't work. Even some samples out there stopped working and it means a lot of effort to go one by one and fix them over and over again, and so would happen with packages. Not that I have anything against Rust, it's just my experience. I also remember the same thing happened to me with Elixir. Again, just empirical evidence of what could happen, nothing more.

Finally, we want to provide a solid base so that code duplication is almost non-existent. We want to have a good http client in the standard library that supports streaming. We want fast json and xml. Now we also have oauth and oauth2 clients (mostly because we are doing some small projects with crystal at work). And all of these integrate very nicely. I don't want the language to come to a point where you have to choose between four different implementation of an oauth client, or an xml parser, etc. Of course we can't provide everything, but what is "very" standard and commonly used should be in the std.

That said, we can use this issue to discuss about how to implement an efficient and elegant package/gem system. Centralized or not? What language to use for the file listing the dependencies? Will you be able to invoke a custom command for a package, like in Rust, so that you can easily include a project like kostya's http_parser.cr which has C code with it?

And most importantly: what name to use? :-)

[vendethiel]

mostly use github, but we don't know how to do that efficiently, without having to checkout the whole repository

component(1) (js) does something like that: it fetches the repo's component.json and downloads (one by one) each file. You need however to be logged on to github not to get rate-limited

[jhass]

And most importantly: what name to use? :-)

Crate? Only conflict I could find after a quick search is https://crate.io/

without having to checkout the whole repository

from git-clone(1):

--depth
Create a shallow clone with a history truncated to the specified number of revisions.

Centralized or not?

I'd vote for gem's style: centralized with configurable, multiple repositories. I think it makes packaging easier, if I look at the "github" style package managers (npm, go get etc.) they quickly are complex enough to force me using them over my system package manager. Another point is that it reduces name clashes, I've seen node libraries with the same name doing a similar thing in a completely different way. Including the github username into the package name again makes system packaging harder and breaks with decentralization since you then depend on github as your central repository.

For the fail-safety part I'd like to see mirroring baked into the core design, ideally setting up a mirror should be as simple as adding a rsync line to your crontab. This would allow us to use existing mirroring infrastructure and for example build a tracking site that also does something like round-robin DNS between them.

A point I'd like to add early to the discussion is providing a package signing mechanism. Basing something upon things like https://keybase.io could yield interesting results.

[farleyknight]

I'm personally a decentralized kind of guy. A centralized server means a core set of people have to maintain it, and I don't think Crystal is quite there yet, as @asterite has said.

I'd rather have something simple that we can throw away easily if it doesn't work. Something simple like a git clone into vendor/ as @jhass suggested. I just feel kind of awkward trying to write code that will become a dependency without some concept of package to tell people to download.

That would give me the most motivation to jump back into my C API generator and a more robust SDL interface, as I would have clear instructions on the README on how to use that package in other people's projects.

Including the github username into the package name again makes system packaging harder and breaks with decentralization since you then depend on github as your central repository.

Git itself is decentralized. People can use bitbucket if they so please. There's no reason why we couldn't use the full URL as the unique package name, with a shorthand name for referencing it in a dependency file. Consider how Gemfile works.

[farleyknight]

BTW @asterite if you're down with a minimalist package system like I described, I'd be more than happy to write the code for it. As long as you're comfortable with explaining what files would be changed.

[jhass]

Git itself is decentralized. People can use bitbucket if they so please. There's no reason why we couldn't use the full URL as the unique package name, with a shorthand name for referencing it in a dependency file. Consider how Gemfile works.

That's rather quoted out of context. My point is that a more centralized approach encourages more unique library names. Since Crystal like Ruby has no caller defined namespaces, this prevents name clashes in case you want to use two libraries with the same name, which happens faster in a fully decentralized approach. It would also reduce confusion about which library you talk about in these cases. For this reason a few git based package systems include things like the Github username into the package name, which is no better than a central repository.

[farleyknight]

Since Crystal like Ruby has no caller defined namespaces, this prevents name clashes in case you want to use two libraries with the same name, which happens faster in a fully decentralized approach.

I've honestly never had this problem! Maybe it is a problem in practice for others. I dunno.

[Inkybro]

I'd just like to weigh in to say that I agree with @jhass on most of his points, although you made some very fair points as well, @farleyknight, and @asterite.

@asterite, nothing is ever perfect, and there are some STD libs in Ruby that have alternative implementations available as gems. For instance, I know that many people don't very much like Ruby's option parser, and opt for some alternative library. I think a selection is a good thing, not a bad one. It might be more confusing to a new-comer, but ultimately will mean that a richer ecosystem results. I still get what you mean, though. I think I may be a more "build it and they will come" kind of guy, as in I don't entirely agree with providing a ton of STD implementations. If someone needs it, they can always build it, right? Obviously, I understand providing the basic core things, HTTP, etc., but if too much time and energy is focused here, the language itself cannot evolve as much, or at least, as quickly. I think focus should be spent on expanding the functionality of Crystal itself, and all but the most common implementations should be left to the community, at least at this point in its development.

I can see exactly why the language is not ready to be rolled out, but that should not hinder progress on some kind of package management system. This would be extremely beneficial for the community around Crystal, in my opinion. Even if it is implemented one way (decentralized) now, and another (centralized) later on. Even if things break down the line (because they inevitably will, one way or the other), that's okay. Ruby has these problems. Many languages do, as @asterite noted in his post.

With all of that said, I can absolutely see how using a decentralized approach might be a better idea for now (it is probably much simpler to implement, much less maintenance required, etc.), but if my opinion had any weight, which it very well may not as this is my first post ever here, I'd say that I'd implement such a system with the eventual goal being to move to something centralized.

I hope this has helped and that I am understanding this discussion correctly.

[waj]

@Inkybro I was about to make a very similar comment. We probably will start with a decentralised approach that would still be useful later (like when you use git sourced gems in a Gemfile).

I'm not sure about the advantages of having a central repository anyway. Maintaining it would require a big effort and it must be really secure and reliable. I also prefer having namespaces (user names?). Otherwise, anyone can reserve an obvious name for a library and maybe later that is not the preferred implementation in the community.

On the other hand for example, I commonly use Puppet modules. The full name for each of these modules is the common "user name / module name". Some users have very good reputation so that's normally taken into account when choosing one module to use.

Still a centralised directory might be useful to make the discovery more easy, but this one can have just pointers to the original repositories and some metadata.

[Inkybro]

@waj Good points.

I actually think I may be slightly misunderstanding the whole centralized vs. decentralized deal. I just think that it should be as easy as running a command (bundle install in w/ Ruby and Bundler). The repositories are automatically fetched and setup.

I like the idea of Puppet modules, I think it makes quite a bit of sense (although not sure if it'd help with name conflicts in the end).

So my end question is, can we really call Ruby's package management "centralized"? I know you can specify more than 1 source, although for any typically released gem you'd simply specify rubygems.org. So while I see how that is centralized to a degree, the option and capability for it to be less so is there.

Can someone help me understand the distinction a little more, or does it sound like I am understanding it correctly?

EDIT: Although I do agree that a centralized repo like RubyGems would need to be very secure, reliable, and would require much maintenance, I also have to say that, who knows, some party/ies may come along at some point in Crystal's development and be more than happy to sacrifice their time and effort to such a task. I certainly think that it should not be ruled out, at the least, but first I think I need to get a better handle on what we mean when we say centralized vs. decentralized.

[waj]

With "centralized" I understand that I must push the library to some central place in order to be available to others. That doesn't necessarily means there cannot be multiple "centralized" locations.

It's true that one can find some overlapping in the semantics of using rubygems or github. In either website you must create an account and you're free to push everything you want inside your space. I think the key point here is using standard and existing infrastructure. Using github also means one could easily replace it with any other git repository.

[Inkybro]

Well, in any case, the library must be pushed to some central place. Even if that were to be FTP or some alternative, it is still going to be in some central place. All the same, I see the benefits in and have no good argument against using GitHub for this, and I like the fact that any git repo can be used. Plus, GitHub is so pervasive these days, this should be a breeze for most.

Ultimately, in my mind, as long as one can easily install dependencies and get a project "ready to run", then the endpoint (GitHub, etc.) is of absolutely no concern to me, personally. Perhaps supporting a variety of options would be nice, but in the end, GitHub is a very good choice, since it is so widely used and understood among the developer community.

I think I get the idea a little more. Thanks, @waj.

[trans]

The main advantage of a centralized approach is that a package can't just up and disappear if its maintainer decides to discontinue development. Also, a centralized system can have safe guards to ensure packages are clean of viruses and such. Plus it provides a central place for people to find and learn about packages.

On the other hand, a decentralized approach gives more freedom to the developer and requires less work --which is very beneficial to a language like Crystal which is in the early stages yet.

I think a good compromise would be to start with a decentralized approach, then later when Crystal is popular enough to support a centralized system, it can be created in such a way as to piggyback off the previous decentralized system. For example, a git plugin can push a release to the central server whenever one creates a release tag. In fact, the central system might just be a collection of git repositories itself.

[Inkybro]

@trans I think something to that effect seems a good approach, too.

[weskinner]

+1 for the name "geode"

[0x1eef]

Why "geode" ? I don't get it.

[weskinner]

Because they package a bunch of crystals together.

[bcardiff]

I thought it was because they have crystals inside :-). I like it.

There is also something called nodules that are geodes without the hollow cavity.
And nodules and modules are ridiculous similar. Sadly as a medical term it is not so nice. But could work also.

[asterite]

@zamith suggested the name "shard", and he even built a website (in Crystal!!) that lists GitHub projects that use Crystal.

I think I like it 😄

In this case shard would mean "a piece or fragment of a crystal", so the definition fits nicely with "a piece of functionality in crystal". It's also just one syllable, so it's faster to pronounce ("gem" has also one syllable).

Maybe "geode" would be something like rvm or rbenv, if a geode packages a bunch of crystals together (let's hope we never need a tool like that 😉)

[kostya]

what about gem -> crem

[ysbaddaden]

I like shard too!

[jhass]

+1 to shard

[zamith]

I just wanted to point out that the original idea for shard was from @naps62, so credit where credit is due. :)

But as you mentioned, I also like it a lot.

On a sidenote (and we have discussed this before), what about moving crystalshards to shards.crystal-lang.org? Do you feel it is too soon?

/cc @asterite @jhass @waj

[JacobUb]

@zamith If we're playing this game... https://groups.google.com/forum/?fromgroups#!searchin/crystal-lang/shard/crystal-lang/0wurVGB4JMU/yqeMYPCSrAkJ
Unless @naps62 suggested it before 20/10/2014, ofc :)

[zamith]

@exilor Hahaha. Had no idea. Sorry about that, then.

[weskinner]

@asterite I see what you mean. I like shard as well.

[naps62]

@exilor oh damn :( so close

[sardaukar]

+1 to shard

[Inkybro]

Hey, guys. Been a while since I piped up here.

So, I still love the idea of a decentralized approach for dependency management. But I have what I think may be a few good ideas, and eagerly await to hear what everyone else thinks of them.

I was thinking it'd be very cool if we settled on some decentralized backend (e.g. git). Users who want to distribute a "Crystal shard" (or whatever they will be called) can use some included binary, perhaps shard push or some command, to "push" their shard up. In reality, this binary and its associated commands would simply wrap git/git-core.

Alternatively, when a user wants to install a library, he or she could use something like shard pull [git-address]. Eventually, we could extend this to have some Shardfile syntax and what-have-you, and an entire package management system can be built around this architecture, which remains decentralized, yet still offers all the really important functionality.

So,

A. Is this a good idea? If so, why do you think? Do you have anything you'd do differently? If not, similarly, why? -- and what do you propose?
B. Is this idea even really new, so to speak? In other words, is what I mentioned basically the eventual plan, or is the whole thing still just "up in the air"?
C. I heard talk about some piece of current, internal code that handles some very basic dep management. Is the implementation of this code similar to what I'm describing? Where is that code, so I can check it out?

Those are my basic questions and thoughts as of right now. I'm very curious to expand the discussion beyond simple answers. I'd love to hash these ideas out a little further and move this along because I really still think that Crystal'd get along a lot nicer/faster with more support, and I think a good package management implementation is really what is missing to garner some good interest.

Look forward to your responses. Thanks.

LAST MINUTE EDIT: I apologize if I have missed out on any important development or news regarding these matters. I keep as close a watch as I can, but I really have very little time.

[refi64]

@jhass Oh.

Well, the rest of the idea would still work...

[asterite]

Nimble or Cargo could work fine, I think. We just didn't have time to implement this yet...

One thing I'd really like to have is recursive dependencies and dependencies checks, like bundler. Do Cargo or Nimble do this?

By the way, I think the current Projectfile, although nice, is not very handy for this type of things because it's a Crystal file so it has to be compiled. Imagine doing that for every dependency (well, shouldn't be that slow, but parsing JSON or TOML should be faster). And also, we already have a TOML parser, but I'm not sure I'd use it. I'd prefer JSON (even it's a bit more verbose). I don't think TOML can be mapped as easily to a data structure as JSON (i.e, like json_mapping).

[refi64]

@asterite What do you mean by dependency checks? I know Nimble downloads any needed dependencies.

[refi64]

Also, I can toy with writing a package manager over the week, but I'm not giving any guarantees...

[asterite]

@kirbyfan64 I mean something like this

Of course, if anyone wants to slowly write a package manager we can later integrate it in the compiler if it's good enough. And right now almost anything is better than the current crystal deps :-)

One thing we do like is to have a project's dependencies in that project's directory, without a global directory where all dependencies go. That makes it easy to remove a project's dependencies when you delete a project (just delete the project's dir) and also to browse a project's dependencies source code (it's right in that directory).

[trans]

Why do I need to trust you? I know all about Toml. It's not at all perfect.
Hipster is what it is -- all the new kids are doing it. Blah. It's junk.

[bararchy]

I also do like TOML, its easy, its readable, and its hashes :)

Tough, I really don't care which parser ends up managing the packages TBH

[trans]

Ryan + Ary gave me an idea. What if a crystal project had a directory
in it, call it dep or deps (or whatever), and in this directory
you added a file for each dependency. e.g.

dep/
    json.dep
    clik.dep

Then each file contains a single line with a git repo reference and a
tag or branch ref constraint. A command line tool can look at these
files and check each out right into the dep directory. It then
recursively checks the dep directory of each of those and does the
same.

There are two ways that recursive behavior can go down, and it depends
on what Crystal is capable of handling (or made to handle).

In the first case, if Crystal can only handle one version of a given
package, no exceptions, the tool has to look for version conflicts. It
might be able to do this by always checking out the repos to the top
level dep directory and checking out the most recent compatible
version (MRCV). If there is no possible MRCV, then of course a version
conflict error arises.

In the other case, perhaps Crystal can handle multiple versions of the
same package? If my package uses version 4 of some library, but
another dependency of mine use the same package but version 3, is
there any reason that they can't just use their own respective
versions and compile without conflict? (Of course it's a good idea for
the tool to tell me about the version difference, at least.)

One other thing this setup would allow: build and test dependencies,
or any other group for that matter could be in a subdirectory. e.g.

dep/
  json.dep
  test/
    cryunit.dep

[ysbaddaden]

I started designing something this morning. Let's see where it goes.

• I chose YAML because JSON is too noisy and TOML's spec is unstable (sic). I apologise for trolling ;-)
• I'll clone bare repositories into .shards and install dependencies into the libs folder (ie. just like crystal deps), because we don't need a global packages folder and I never know where Bower, NPM and alikes put their global caches, and I hate that.
• I'm starting decentralised but with the long term goal of having metadata registries (name -> URL + versions + dependencies) to speed things up (no need to clone everything upfront), and simplify usage (I prefer to remember a package name, not a repository URL) and have a searchable list of packages.

[bararchy]

I would really like if the dir will be an external dir, like in gem, where for each "require" crystal can link the needed deps from the global directory, this way I dont need download all the deps each time for each project, and It's easier to keep all the deps up to date.

[trans]

On Wed, May 20, 2015 at 7:50 AM, Trans [email protected] wrote:

Why do I need to trust you? I know all about Toml. It's not at all perfect.
Hipster is what it is -- all the new kids are doing it. Blah. It's junk.

Okay. That's a bit harsh. Toml is okay. I just think it is
overrated, and espoused for the wrong reasons, i.e. new and simple
--- new and simple does not necessarily equate to better.

[trans]

If I were able to get this to compile to Crystal would it be helpful to you? https://github.com/rubyworks/versus Note it has a version resolver.

[cuzbog]

@asterite Do you mean like if package a depends on b, and b on a, then throw an error? Yes, Cargo does that. Also, here's an example of Servo's (in case you didn't know, Servo is like Gecko written in Rust) Cargo.toml. It's very readable by humans. One thing I wouldn't like is Github being compulsory, because there's Bitbucket, GitLab etc.

[trans]

On Wed, May 20, 2015 at 8:24 AM, Trans [email protected] wrote:

In the other case, perhaps Crystal can handle multiple versions of the
same package? If my package uses version 4 of some library, but
another dependency of mine use the same package but version 3, is
there any reason that they can't just use their own respective
versions and compile without conflict? (Of course it's a good idea for
the tool to tell me about the version difference, at least.)

So can anyone tell me if this is possible (or can be made possible) or not?

[asterite]

@trans Dependencies aren't compiled to separate binaries that are later linked. When you compile your program, all of the dependencies are load up and from them and the main program an executable is created. If version 3 defines a method foo that returns 1 and version 4 defined a method foo that returns 2, one of them will win: they can't coexist.

[trans]

@asterite That's helpful. Thanks.

Theoretically, couldn't the two versions be "namespaced" so as not to be confused? So code using version 4 would call v4::foo and code using version 3 would call v3::foo. I once developed such a system for Ruby. With the exception of global variables and core extensions it worked. In short, instead of require 'foo', one used a more javascript styled foo = require('foo', '>4'), so a namespace automatically wrapped the library, which was then reachable via foo variable.

[bcardiff]

@trans In theory for a simple code it can be done, but as @asterite said, right now the compiler loads all the required sources and compiles it as a single unit. Class/method lookups are run over the whole code.

If the lookups are changed in order to use some sort of scoped lookup based on project dependencies each shard would be able to use just the code it's advertised depends on. But then we need to see how to deal with blocks calls, inversion of control, delegation. Monkey patch will be left out (despite not being the something to encourage). Top level funs will definitely clash. I'm not sure if for something more than simple code that will end up working.

I don't see something like that coming. Yet, it seems an interesting problem.

[clord]

I know most languages have their own package management solution and so the temptation is there to invent yet another one from whole cloth, but I'd like to propose adopting a slightly modified version of nix as a way to distribute code and dependencies. And if not that, at least base a new crystal-specific manager on it.

Why?

Most language package managers end up needing other things too. I'm thinking of having a local imagemagick install, or libxml, etc. and everyone always ends up rolling their own solutions to having these installed, usually half-assed and buggy for a decade or more.

Using a real package manager as the foundation means you can easily port these sorts of dependencies without having to reinvent the wheel.

The reason I suggest nix specifically is that it is really well thought out, and solves a lot of problems that a source-code package manager will also have to solve. Aforementioned binary dependencies is one. Also, it's really good at versioning. One project might depend on v3.1 of some library, whereas one of it's dependencies might depend on 2.0. Now the ideal way to deal with this is to allow both to have their own dependent versions used, without conflict. Nix makes that sort of stuff trivial.

Because of the way nix works, it becomes really easy to 'switch' projects. With ruby, RVM and other tools try to solve this problem (badly, imo). nix inspired tooling would allow the whole stack to be specified declaratively at the project level.

Also, because of the way nix lays out packages, it makes it really easy to build stuff. It puts literally everything into one tree (kind of like a mini chroot) and then builds it. So one need only set the appropriate LOAD_PATH environment variable and run the normal compiler without much cooperation from the package manager (the compiler could also be a dependency of the project btw)

So my proposal then is to:

1. Either fork be inspired by nix (at the very least the idea of immutable self-contained packages)
2. Create a crystal-specific front-end to the nix store
3. use the resulting tool to manage source of dependencies
4. use it to publish end-user binaries and compiled libraries as well, ideally with an easy way to port from elsewhere.

This is really a big opportunity to leap-frog gems and RVM. At least studying the design of nix will certainly produce a much stronger package manager that will solve some of the worst problems with the other systems out there, and save some work in the process.

[rishavs]

Hi

Let me provide the perspective of a beginner and a not too serious coder.

coming from node and python, i personally prefer a centralized approach. I personally believe that npm was among the most important factors in node getting so big.

Also, if crystal does implements a package manager, i'd really want it to be part of crystal itself, rather than a separate project/product.

I'd love to use commands like `crystal install [email protected]'

For me all the complex and dirty bits should be under the hood. Global or local namespace is sufficient for me. Heck, even a global default is great. the less i have to fiddle with package mangement, it would be better for me.

Finally, while i love YAML, i really dont think it should be used in any public project with lots of contributors. It is very contentious and a lot of people find its lack of adherence to its own specs to be off putting. JSON is what I'll recommend.

[vyp]

@rishavs You may be interested in https://github.com/manastech/crystal/wiki/Roadmap#package-manager if you haven't already seen it.

@clord I agree, but I don't see the reason for using a fork of nix, why not just nix itself? And besides, unless I'm misunderstanding something, crystal install is not supposed to be a package manager, but just more like a build helper, ala rust's cargo install command, so really you should be able to use whatever you want.

[clord]

@vyp the problems a 'build helper' solves and those that a 'package manager' solve overlap significantly, especially if crystal wants to have something resembling ruby gems.

[jhass]

Shall we close this in favour of #1357?

[asterite]

@jhass Good idea!