add support for nightly boards manager packages

[igrr]

Arduino IDE has support for “nightly” versions of boards manager packages (arduino/Arduino#3449). These versions are distinguished by the lack of “size” and “checksum” fields in json file. IDE always uninstalls previously installed nightly version before installing the new one.

This change adds support for generation of nightly package builds.

TODO:

• auto-patch core version in platform.txt
• drop "staging" package links
• update readme with instructions for installing nightly packages
• update travis build script to generate nightly packages and upload them to arduino.esp8266.com

[igrr]

This is ready for review and testing.

Switching between stable and nightly packages seems to work for me (i.e. I'm not seeing any issues when compiling). It seems that the IDE will not remove old nightly package archives (that's currently 18 megabytes per archive, can be potentially slimmed down if we strip documentation when building the archive). For this to happen, all nightly packages need to have the same name (e.g. esp8266-nightly.zip, without version/date info included). I'm inclined to add a caveat that Arduino download directory needs to be cleaned manually once in a while, if nightly packages are used, instead of removing version info from the nightly packages.

Also only the latest nightly package is listed in the package json file, so rolling back to the previous nightly package is not possible. This can be implemented later if requested, just needs some tweaks to the python script used to merge packages.

[davisonja]

Is cleaning the ide downloads from the package a possibility? Though in practice this is an issue for anything that's frequently updated 😊 (feels like the ide really needs an option to trim the historical downloads)

…

On 19/02/2017 7:12 pm, "Ivan Grokhotkov" ***@***.***> wrote: This is ready for review and testing. Switching between stable and nightly packages seems to work for me (i.e. I'm not seeing any issues when compiling). It seems that the IDE will not remove old nightly package archives (that's currently 18 megabytes per archive, can be potentially slimmed down if we strip documentation when building the archive). For this to happen, all nightly packages need to have the same name (e.g. esp8266-nightly.zip, without version/date info included). I'm inclined to add a caveat that Arduino download directory needs to be cleaned manually once in a while, if nightly packages are used, instead of removing version info from the nightly packages. Also only the latest nightly package is listed in the package json file, so rolling back to the previous nightly package is not possible. This can be implemented later if requested, just needs some tweaks to the python script used to merge packages. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2936 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN_A5wZtth1DUFcKbH97ihXjSF4MTSPks5rd91ggaJpZM4L4ABA> .

[igrr]

Running some shell/batch script from the build step is possible, but I wouldn't risk accidentally screwing up in this script and wiping something useful.

If the IDE could match downloaded files and specific versions in the package.json file, it could delete all downloaded files which no longer have a corresponding version in the package.json file. However the IDE doesn't do this, and looking at the IDE code this would require some major changes.

One possible way would be to store all files related to a downloadable core in a directory named as a qualified name of the core (e.g. esp8266com_esp8266/). Then one could iterate over files in this directory and remove every file which isn't listed in the latest downloaded package.json file for any version.

[davisonja]

Even if it wasn't part of the build, but a provided utility that could be manually triggered by a user wanting to reclaim space. (rather than them working out what to remove by hand)

…

-- On 19/02/2017 7:25 pm, "Ivan Grokhotkov" <[email protected]> wrote: Running some shell/batch script from the build step is possible, but I wouldn't risk accidentally screwing up in this script and wiping something useful. If the IDE could match downloaded files and specific versions in the package.json file, it could delete all downloaded files which no longer have a corresponding version in the package.json file. However the IDE doesn't do this, and looking at the IDE code this would require some major changes. One possible way would be to store all files related to a downloadable core in a directory named as a qualified name of the core (e.g. esp8266com_esp8266/). Then one could iterate over files in this directory and remove every file which isn't listed in the latest downloaded package.json file for any version. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2936 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN_A0nyDfZajJV5c9QSehYW6G38aujkks5rd-BegaJpZM4L4ABA> .

[igrr]

I suppose this will work for Mac OS:

find $HOME/Library/Arduino15/staging/packages -name 'esp8266-*-nightly*.zip' | xargs rm

Linux version will differ only with the path prefix:

find $HOME/.arduino15/staging/packages -name 'esp8266-*-nightly*.zip' | xargs rm

As for Windows version, there's probably a similar one-liner, but my batch skills are too rusty to remember this.

[me-no-dev]

nice! Will shamelessly steal this once done ;)

[igrr]

Okay, I've learned that this approach doesn't really work.

Each day the boards manager package json file is updated. Old nightly version is removed, new one is added. Suppose you have installed the nightly package and day later you've opened the boards manager in the IDE to update the package. New json file will not have the last day's version listed, so the IDE will not find any installed version and will assume that the esp8266 package is not installed. You get the combobox with the list of versions which can be installed (which now includes new nightly version).

If this new version is installed, IDE doesn't remove the old package, and installs the new one side by side. So in packages/esp8266/hardware/esp8266, we end up with two versions (subdirectories):

2.4.0-nightly+20170221
2.4.0-nightly+20170222

If we try to compile the sketch now, we get a "board esp8266 platform esp8266 is unknown" error from arduino-builder. Manually removing any of the two directories solves the problem.

If the old package is kept in the json file, IDE finds it and figures out that it's upgrading the package, and successfully removes the old directory. But keeping nightly packages in the json file indefinitely will not scale well, so some other solution is needed.

So far I'm out of ideas here, any pointers very welcome.

[me-no-dev]

Is the folder naming conversion something that is constant and standard throughout the boards? Maybe we can "fix" this into the source (ArduinoIDE) and detect earlier version by the file name, rather than expecting it to be in the json.

[igrr]

Names of folders are basically version numbers. Boards manager uses semver, so I would say that we can parse these names. We probably can't impose a strict order on the names, because AFAICT semver doesn't say which of 2.4.0-nightly+20170221 and 2.4.0-nightly+20170222 should be considered earlier version. But at the very least we can figure out that there is an installed directory for a version which isn't listed in the package manager, and uninstall it when the new version is installed.

[me-no-dev]

Yes. Basically any other version that might be there should be considered earlier :) and should be removed.

[me-no-dev]

I have not looked recently (and have never used package manager) but as far as I remember there is a drop-down with list of versions available. It would make sense to have a single "nightly" option above all tagged ones. Using separate json looks kinda like workaround rather than how things should be.
What do you think about the above?

[igrr]

This is how it is done currently, nightly package has one nightly version plus all releases.

The reason for making it a separate package is because I don't want causal users to grab the nightly version just because it is the latest one.

Edit: If you want to try yourself, use http://arduino.esp8266.com/nightly/package_esp8266com_index.json link.

[me-no-dev]

But it could really help in situations like the current one, where the release is so behind the nightly and would not require the user uninstall/reinstall the package and have more issues cause of that

[igrr]

The issue described above will happen anyway, as soon as the nightly version is updated, even if I add the nightly version to the "main" json file.

[me-no-dev]

Yes I get that :) But I'll be looking towards fixing that problem

[luc-github]

I do not know if it can help but on my side before installing a nightbuild I launch a simple clean batch to avoid error mentioned earlier (previous install directory present leading to "esp8266 is unknown" and remove previous zip package to save space)

forfiles /P %USERPROFILE%\AppData\Local\Arduino15\staging\packages\ /M esp8266-*-nightly*.zip /C "cmd /c del @file"
forfiles /P %USERPROFILE%\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\ /C "cmd /c if @isdir==TRUE rmdir /s /q @file"

then I install latest nightbuild in board manager - so far it works for me

[luc-github]

Hi is nightly build is still frozen to 2.4.0-nightly+20170222 ? or it will be updated ?
Just to know if I need to pull full git to get latest changes
Thank you

[igrr]

Unfortunately the nightly build concept doesn't seem to work in the IDE the way I have expected (see arduino/Arduino#6012) so i have stopped working on this until some other solution is found. Install via git for now.

[luc-github]

Ok thank you