Start article about the history of conda-forge #2298
Conversation
✅ Deploy Preview for conda-forge-previews ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
Great article @jaimergp! It felt like it ended with cliffhanger and make me want for more. Are you planning on part 2/∞? |
Yes! This is just the beginning, and not ready for publication yet. Was hoping to gather some interest here and get comments from the "old guard" while I cover the very beginnings. Then I'll need a looot of help with the 2016-2021 period, and after that I think I can recollect a few things. Even basic bullet items with a rough chronology would help so I can research git histories, archive.org, etc. |
|
Here are some big events to track / mention. I don't have all of the details:
|
|
There were several bots
Some other things to note
|
|
|
||
| conda-forge's origins cannot be explained without understanding the context of Python packaging back in the early 2010s. Back then, the installation of Python packages across operating systems was very challenging, specially on Windows, as it often meant compiling dependencies from source. | ||
|
|
||
| Python 2.x was the norm, the community was transitioning from `easy_install` to `pip`, and there wouldn't be an alternative for Python eggs [^eggs] until 2012, when wheels are introduced [^wheels]. To get Python, you'd get the official installers from Python.org, stick to the system provided one in Linux, or resort to ActiveState's or Enthought's distributions in macOS and Windows [^legacy-python-downloads]. |
There was a problem hiding this comment.
Python(x,y) was a popular Windows distribution back then, especially for science users. I recall it and Enthought's basically being the two main options for getting things working on Windows.
| Python 2.x was the norm, the community was transitioning from `easy_install` to `pip`, and there wouldn't be an alternative for Python eggs [^eggs] until 2012, when wheels are introduced [^wheels]. To get Python, you'd get the official installers from Python.org, stick to the system provided one in Linux, or resort to ActiveState's or Enthought's distributions in macOS and Windows [^legacy-python-downloads]. | |
| Python 2.x was the norm, the community was transitioning from `easy_install` to `pip`, and there wouldn't be an alternative for Python eggs [^eggs] until 2012, when wheels are introduced [^wheels]. To get Python, you'd get the official installers from Python.org, stick to the system provided one in Linux, or resort to options like ActiveState's or Enthought's distributions in macOS and Windows [^legacy-python-downloads]. |
There was a problem hiding this comment.
Added "options like" to not disregard Python(x,y) and others.
|
I remember being in a birds of a feather session at SciPy around 2013 or 2014 where the momentum to make conda forge real seemed to solidify and it was very soon after that conference that it took material form. |
|
The content is accurate as far as I can remember (which doesn't necessarily mean much). I would suggest doing a full checkup for grammar, and in particular, being consistent across the post with tense. |
It was 2015 that the BoF happened and the soft launch on 2016 if I'm not mistaken. |
|
I would like it if there were a paragraph that mentions the deep collaborative period between Anaconda's default channel and conda-forge. Where we would often trade recipes collaboration was intricately linked. I really look back fondly at the times where I was learning alot from msarahan, mingwandroid, jcrist, mrocklin (not sure if he worked for Anaconda at the time). For me, the availability of Qt, Pillow, and OpenCV on windows/osx/linux were what brought me to Anaconda/conda/conda-forge. |
|
|
||
| By 2015, several institutes and groups were using Binstar to distribute software packages they used daily: the [Omnia Molecular Dynamics](https://github.com/omnia-md) project started as early as March 2014 [^binstar-omnia], UK's [Scitools](https://scitools.org.uk/) joined in June 2014 [^binstar-scitools], the [US Integrated Ocean Observing System (IOOS)](http://www.ioos.noaa.gov/) started using it in July 2014 [^binstar-ioos]. The channel for conda-forge was not created until April 2015 [^binstar-conda-forge], and [Bioconda](https://anaconda.org/bioconda) waited until September of the same year. | ||
|
|
||
| In 2015, Continuum Analytics rebranded as Anaconda Inc, and Binstar.org became Anaconda.org. |
There was a problem hiding this comment.
Continuum Analytics did not change its name to Anaconda Inc until 2017-2018. Before that "Anaconda Powered by Continuum Analytics" was a common slogan to highlight the connection between Anaconda and Continuum.
There was a problem hiding this comment.
From https://www.anaconda.com/blog/continuum-analytics-officially-becomes-anaconda is looks like the name change was in 2017
|
My recollection might be off but I recall a meeting at SciPy 2015 which included @pelson, @ocefpaf, @scopatz, myself and likely others where some initial details of what became This timeline aligns with the first commit in conda-forge/staged-recipe from the Fall of 2015 and the history recorded in the talks documentation page I recall |
|
Another possible highlight to include in the history is SciPy 2013 when Travis Oliphant announced binstar.org which became anaconda.org during the Thusday lightning talks |
|
|
||
| conda-forge's origins cannot be explained without understanding the context of Python packaging back in the early 2010s. Back then, the installation of Python packages across operating systems was very challenging, specially on Windows, as it often meant compiling dependencies from source. | ||
|
|
||
| Python 2.x was the norm, the community was transitioning from `easy_install` to `pip`, and there wouldn't be an alternative for Python eggs [^eggs] until 2012, when wheels are introduced [^wheels]. To get Python, you'd get the official installers from Python.org, stick to the system provided one in Linux, or resort to ActiveState's or Enthought's distributions in macOS and Windows [^legacy-python-downloads]. |
There was a problem hiding this comment.
I think there is an important context being missed here - wheels were for pure python packages. Wheels with vendored shared libraries didn't come until much later - and they referenced conda packages to highlight that it is possible to have a "manylinux" concept.
|
|
||
| ## The origins of `conda` | ||
|
|
||
| In 2012, Continuum Analytics announces Anaconda 0.8 in the SciPy conference [^anaconda-history]. Later that year, in September, Continuum would release `conda` 1.0, the cross-platform, language-agnostic package manager for pre-compiled artifacts [^conda-changelog-1.0]. The motivation behind these efforts was to provide an easy way to ship all the compiled libraries and Python packages that users of the SciPy and numpy stacks needed [^packaging-and-deployment-with-conda] [^lex-fridman-podcast]. |
There was a problem hiding this comment.
The history goes even further I think - the knowledge and know-how to build conda came from the direct experience of Ilan Schnell building the Enthought python distribution, if I understood correctly.
|
|
||
| In 2012, Continuum Analytics announces Anaconda 0.8 in the SciPy conference [^anaconda-history]. Later that year, in September, Continuum would release `conda` 1.0, the cross-platform, language-agnostic package manager for pre-compiled artifacts [^conda-changelog-1.0]. The motivation behind these efforts was to provide an easy way to ship all the compiled libraries and Python packages that users of the SciPy and numpy stacks needed [^packaging-and-deployment-with-conda] [^lex-fridman-podcast]. | ||
|
|
||
| In constrast with Python eggs and wheels, conda packages were agnostic enough to ship Python itself, as well as the underlying shared libraries without having to statically vendor them under each Python package. This was particularly convenient for projects that relied on both compiled dependencies (e.g. C++ or Fortran libraries) and Python "glue code". |
There was a problem hiding this comment.
As above - this is jumping the gun. There was no binary shipping mechanism for Python - conda was the first environment manager to do so AFAIK.
Before this, on windows you used to go to Christoph Gohlk's website. On linux, I think you had to build from source.
There was a problem hiding this comment.
Howdy @pelson! Long time.
Back around say 2005-2010 I remember it working out OK on platforms where you had strong package manager interfaces at the OS tier — like Debian/Ubuntu based distributions, MacPorts on OS X, or ports on FreeBSD. But of course trying to do so on Windows, or anywhere where you didn't have a well maintained compiler stack trivially installable was still a nightmare, especially if you needed to work with a broad collection of packages. In terms of Python specific package managers which did deal with the scientific Python distribution problem, I believe that Enthought had something even before Canopy, and definitely ActiveState did, but both of these were licensed products which hampered their use, particularly in the academic community.
|
|
||
| By June 2013, conda is using a SAT solver and includes the `conda build` subcommand [^new-advances-in-conda], along with the concept of recipes [^conda-recipes-repo] [^early-conda-build-docs]. This is also when the first Miniconda release is announced. By the end of the year, Continuum Analytics announces Binstar.org, the predecessor of the Anaconda.org channels. This meant that now any user could build their software stack as conda packages and redistribute them online at no cost. | ||
|
|
||
| By 2015, several institutes and groups were using Binstar to distribute software packages they used daily: the [Omnia Molecular Dynamics](https://github.com/omnia-md) project started as early as March 2014 [^binstar-omnia], UK's [Scitools](https://scitools.org.uk/) joined in June 2014 [^binstar-scitools], the [US Integrated Ocean Observing System (IOOS)](http://www.ioos.noaa.gov/) started using it in July 2014 [^binstar-ioos]. The channel for conda-forge was not created until April 2015 [^binstar-conda-forge], and [Bioconda](https://anaconda.org/bioconda) waited until September of the same year. |
There was a problem hiding this comment.
UK's SciTools -> the UK Met Office supported "SciTools" project
| ## How conda-forge came to be | ||
|
|
||
| In 2014, Filipe Fernandes ([@ocefpaf](https://github.com/ocefpaf)) and Phil Elson ([@pelson](https://github.com/pelson)) get in touch [^chatting-ocefpaf]. They are maintaining the Binstar channels for IOOS and Scitools, respectively. | ||
|
|
There was a problem hiding this comment.
Think there is a lot Filipe and I can add here 😉
We should give some context of the geospatial world in particular (hi gdal).
I would also get to the point of acknowledging both Christophe Gohlke and David Cournapeau, who especially helped me with the Windows builds of the whole SciPy stack (a topic on which I had no knowledge at all, yet needed to get building in a CI context on appveyor. In those days you had to pick particular compilers for particular Python versions, and this was a bit of a dark art to me).
|
|
||
| In constrast with Python eggs and wheels, conda packages were agnostic enough to ship Python itself, as well as the underlying shared libraries without having to statically vendor them under each Python package. This was particularly convenient for projects that relied on both compiled dependencies (e.g. C++ or Fortran libraries) and Python "glue code". | ||
|
|
||
| By June 2013, conda is using a SAT solver and includes the `conda build` subcommand [^new-advances-in-conda], along with the concept of recipes [^conda-recipes-repo] [^early-conda-build-docs]. This is also when the first Miniconda release is announced. By the end of the year, Continuum Analytics announces Binstar.org, the predecessor of the Anaconda.org channels. This meant that now any user could build their software stack as conda packages and redistribute them online at no cost. |
There was a problem hiding this comment.
I think conda-recipes is worth more detail - it was the place that people would contribute their recipes.
It was really successful, but the recipes were of various quality, and typically only worked on one or two platforms. There was a high chance that a recipe you found there would no longer build, and you had to tweak it to get it to work.
To my knowledge, SciTools was the first repo to do CI based builds. Filipe borrowed the technical infra for IOOS to do the same. I borrowed some of the harder to build recipes back. It was a successful collaborative effort, but it was inefficient since we were working in separate repos. We often had duplicate recipes etc.
conda-forge came about because I could see that the conda-recipes repo was popular, there was demand for having high-quality recipes, and we needed a way to build them in a consistent way. I even built a tool (conda-build-all) to try to do this from our repositories in an efficient way. In the end, it got to the point where we wanted to de-centralise the responsibilities, and the one-repo-per-recipe concept fell out.
PR Checklist:
docs/orcommunity/, you have added it to the sidebar in the corresponding_sidebar.jsonfileStill work in progress, but I wanted to capture the momentum started by Wolf and Filipe's podcast episode.
Tagging some folks for awareness, visibility, and hopefully a review, comments or even contributions if they are feeling generous 🙏 @ocefpaf @jakirkham @pelson @dholth @bryevdv @msarahan @asmeurer @ilanschnell. Feel free to tag others as well if you feel they can add more context into the early days!
🔍 Preview article link 🔍