[release-0.5] backports for 0.5.2 #21684

tkelman · 2017-05-03T01:50:22Z

No description provided.

(cherry picked from commit d42eac9) PR: #19054

(cherry picked from commit 7705088) PR: #19054

Check that test_broken and test_skip do not result in an exception. Already fixed in 8982605 - "Rework test framework", but present in julia-0.5 (see #21008). See also #21086 (cherry picked from commit 3ce8ab0)

Minimal fix backported from 8982605. Fixes #21008

Backport of #21069. (cherry picked from commit f65f273)

Backport of #21069. (cherry picked from commit 038390b)

Backport of #21069. (cherry picked from commit 1594692)

(cherry picked from commit cf7d049) ref #19197

causing duplicate files and including directories it shouldn't be (cherry picked from commit 4521e6b)

(cherry picked from commit 8fb132f) ref #20914

(cherry picked from commit 1632473) ref #20914

from OpenMathLib/OpenBLAS#1098 (cherry picked from commit 26beab3) ref #21091 fix Makefile.system dependency for release-0.5

Fixes #21122 (cherry picked from commit 3d126b8) remove testset for release-0.5

See 30ba746#commitcomment-21430419 for more details (cherry picked from commit 7ce4925) ref #21134

(cherry picked from commit 3bdfeef) ref #21127

(cherry picked from commit 1d678da) ref #20950

* do checkout before changing HEAD 541d94b * add test for change to file permissions 42493ae * add test for symlinks a3fab6b * add issue numbers to comments 1b7d47a change close(repo) to finalize(repo) for release-0.5

* updated fix for #19892; initialize FFTW threads the first time the planner is called (#21127 incorrectly prevented threads from being used at all) * add test for #21163 (cherry picked from commit 378ed8a)

The DirtyCOW exploit mitigation patch in Linux (torvalds/linux@19be0eaff), introduced a kernel bug, that would cause the kernel to hang if an application tries to use /proc/<pid>/mem (or ptrace) to bypass page protections on pages that are backed by transparent huge pages (though only if the write causes a COW resolution). We make use of this feature in our cg memory manager on Linux, but since we can't predict whether or not our mappings will be backed by transparent huge pages, the only safe thing to do is to fall back to dual maps on kernels that potentially have this issue. Since the problematic commit is a fix for a high-profile exploit, it is quite likely that it is included in most stable kernels by now. I have a patch pending at http://marc.info/?l=linux-mm&m=148359462417378&w=2, which I expect will fix this in the kernel for 4.10. However, we'll have to disable the use of /proc/self/mem for all prior kernel versions to avoid locking up the kernel. (cherry picked from commit c8312d3) ref #19887

(cherry picked from commit 4776d9a) ref #21251

this ensures that another library calling _setmode or _dup2 does not affect our library output fix #21264 (cherry picked from commit 261c716) ref #21380

This makes sure that atomic load and stores have the correct alignment as guaranteed by the gc. (cherry picked from commit f8e3e12) ref #19482

tkelman · 2017-05-03T01:54:57Z

@jlbuild !nuke

versioninfo()
Base.runtests()
Base.runtests("pkg libgit2-online")

tkelman · 2017-05-03T02:55:21Z

Looks like that's not running.

More importantly, @staticfloat the centos buildbots are having certificate issues and aren't able to download some of the deps: https://build.julialang.org/builders/package_tarball64/builds/247/steps/make/logs/stdio
Any ideas on what we should do about that? Given less urgency I'd say we should drop centos 5 and use centos 6 as a baseline now that 5 is EOL and having a bunch of download issues, but maybe we want to put in place more of a band-aid right this second?

(cherry picked from commit 9baadbf)

…#21431). (#21432) (cherry picked from commit 8a3b6c1)

(cherry picked from commit 8d27c72) ref #20480 go back to vec of matrix comprehension for release-0.5 avoids ERROR: LoadError: LoadError: ArgumentError: argument to Flatten must contain at least one iterator in start at .\iterator.jl:599 in grow_to! at .\array.jl:357 in intersect at .\pkg\types.jl:44 in push! at .\pkg\types.jl:182 in requirements at .\pkg\query.jl:34 in resolve at .\pkg\entry.jl:476

prepares the way to keep track of the propagation of requirements (cherry picked from commit d47f6e8) ref #20480 expand out Pair{<:AbstractString,ResolveBacktraceItem} for 0.5

Add a pre-pruning step in which requirements are propagated along the dependency graph, keeping a backtrace of the process. This has two benefits: 1. Make things easier for the solver, by simplifying the graph and reducing the number of versions 2. It should detect most common problems with requirements that introduce implicit contradictions, and it provides a useful error message to fix the situation. (cherry picked from commit d054d36) ref #20480

This reverts commit c250310.

This reverts commit 8a4e1be.

This reverts commit e1c9a4b.

This reverts commit 39cd805.

tkelman · 2017-05-04T21:17:34Z

after reverts: @nanosoldier runbenchmarks(ALL, vs = "@6445c82d0060dbe82b88436f0f4371a4ee64d918")

tkelman · 2017-05-04T22:53:53Z

Set of binaries that's starting hopefully the last run of pkgeval here:

nanosoldier · 2017-05-05T00:37:40Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

staticfloat · 2017-05-05T01:13:39Z

@tkelman I was testing this branch out for fun on my new buildbots, and I ran into the problem of libLTO.so not being included within tarballs on armv7l. I've opened a PR on this, but we should probably sneak it into 0.5.2 and 0.6.0 so that binary-dist works from scratch on both. Luckily it's not hard to fix manually.

tkelman · 2017-05-05T03:13:32Z

It may be a difference in buildbot configuration? Can someone with access to arm and power test the binaries linked above, that were built from the older existing buildbots?

staticfloat · 2017-05-05T03:25:01Z

I just tested them, they both work fine. Time to dig into why they don't link against libLTO I guess.

tkelman · 2017-05-05T03:39:07Z

GCC 7 was released this week

ugh so gcc 7 bumped the libgfortran somajor from 3 to 4, breaking a lot of assumptions our scripts were making, and the mac nightlies are now missing libgfortran

tkelman · 2017-05-05T03:56:18Z

can we downgrade the mac buildbot to gcc@6 for the purposes of getting this release and 0.6 rc out now, then fix the gcc 7 issues without having to rush it?

timholy · 2017-05-05T06:34:11Z

@timholy is #21251 worth a 35% slowdown in growing arrays?

I'd say no. The reverse changes would probably be fine, but at this point just dropping it is OK.

staticfloat · 2017-05-05T07:33:13Z

I've installed gcc@6, I've changed the buildbot to try to install that before installing, and restarted the buildbot. Go ahead and try it now.

tkelman · 2017-05-05T09:15:46Z

rerun to see whether remaining regressions are noise : @nanosoldier runbenchmarks(ALL, vs = "@6445c82d0060dbe82b88436f0f4371a4ee64d918")

tkelman · 2017-05-05T09:45:56Z

@staticfloat is there a link or alias step we can do to have that get used as gfortran ? looks like it only installed under a versioned name by default?

tkelman · 2017-05-05T10:29:35Z

https://gist.github.com/b44bbbac3697bcf715ba1c464298f95b

BioEnergeticFoodwebs is a flaky numerical tolerance
Immerse same as above
Pajarito and RingArrays timed out
SeqMaker timed out on the baseline 0.5.1 run
Sims log says it passed, maybe the exit status was wrong for some reason?

If the benchmark and buildbots are okay I want to tag this, #21708 can wait, I'd rather not have to go through pkgeval another time

nanosoldier · 2017-05-05T14:24:18Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

staticfloat · 2017-05-05T16:36:49Z

#21708 can wait, I'd rather not have to go through pkgeval another time

Since 0.6.0 is just around the proverbial corner, I agree; non-NEON architectures can just wait until 0.6.0 is released, which shouldn't be too far off (knock on wood)

martinholters · 2017-05-05T19:35:41Z

I might have lost track here, but have the regressions we had attributed to #21319 just occurred again although it was reverted?

tkelman · 2017-05-05T20:54:22Z

or potentially #21685 ? or #21425 or #21206 ?

vchuravy and others added 22 commits March 10, 2017 23:55

add patch to make shlib builds with cmake work

7d2d211

(cherry picked from commit d42eac9) PR: #19054

use cmake for llvm 3.8+

4d41845

(cherry picked from commit 7705088) PR: #19054

DefaultTestSet tests for test_broken and test_skip

9127cca

Check that test_broken and test_skip do not result in an exception. Already fixed in 8982605 - "Rework test framework", but present in julia-0.5 (see #21008). See also #21086 (cherry picked from commit 3ce8ab0)

Don't throw for tests which are explicitly marked broken

aa3907c

Minimal fix backported from 8982605. Fixes #21008

Introduce temporary if necessary to inline _apply

8a4e1be

Backport of #21069. (cherry picked from commit f65f273)

use newvar! instead of add_slot!

c250310

Backport of #21069. (cherry picked from commit 038390b)

add test

e9fc37a

Backport of #21069. (cherry picked from commit 1594692)

Force inlining of varargs-setindex!

70e5b56

(cherry picked from commit cf7d049) ref #19197

Work around bug in tar 1.29 with --no-recursion and -T (#20942)

5c1972b

causing duplicate files and including directories it shouldn't be (cherry picked from commit 4521e6b)

work-around joinpath error when cloning git ssh urls (fix #20695)

633993f

(cherry picked from commit 8fb132f) ref #20914

pkg: factor url_or_pkg logic into its own function and test it

32e26e1

(cherry picked from commit 1632473) ref #20914

Carry openblas power assembly fixes

b004efa

from OpenMathLib/OpenBLAS#1098 (cherry picked from commit 26beab3) ref #21091 fix Makefile.system dependency for release-0.5

Fix stride for vectors in BLAS.ger! and BLAS.trsv! (#21125)

ea015b8

Fixes #21122 (cherry picked from commit 3d126b8) remove testset for release-0.5

OSX: Add both RPATH entries to julia, don't overwrite

b044f74

See 30ba746#commitcomment-21430419 for more details (cherry picked from commit 7ce4925) ref #21134

fix FFTW crash after set_num_threads

0f9d440

(cherry picked from commit 3bdfeef) ref #21127

Identify ppc64le as a 64-bit architecture

468974c

(cherry picked from commit 1d678da) ref #20950

do checkout before changing HEAD (#20916)

8619307

* do checkout before changing HEAD 541d94b * add test for change to file permissions 42493ae * add test for symlinks a3fab6b * add issue numbers to comments 1b7d47a change close(repo) to finalize(repo) for release-0.5

updated fix for #19892 (FFTW threads initialization) (#21169)

4be8a3f

* updated fix for #19892; initialize FFTW threads the first time the planner is called (#21127 incorrectly prevented threads from being used at all) * add test for #21163 (cherry picked from commit 378ed8a)

More fixes for non-1 arrays

39cd805

(cherry picked from commit 4776d9a) ref #21251

add stdio dup call for fd on windows

a676636

this ensures that another library calling _setmode or _dup2 does not affect our library output fix #21264 (cherry picked from commit 261c716) ref #21380

Correct alignment of atomic load and stores

2a5dd2a

This makes sure that atomic load and stores have the correct alignment as guaranteed by the gc. (cherry picked from commit f8e3e12) ref #19482

tkelman force-pushed the tk/backports-0.5.2 branch from a2aaed6 to 172ff0d Compare May 3, 2017 04:53

ranjanan and others added 5 commits May 2, 2017 22:15

Fix bug in backsolve of sparse hermitian matrices (#21165)

b428e47

(cherry picked from commit 9baadbf)

Fix sparse right multiplication of Symmetric and Hermitian matrices (…

d13cb5f

…#21431). (#21432) (cherry picked from commit 8a3b6c1)

Add ResolveBacktrace types in Pkg

f907b57

prepares the way to keep track of the propagation of requirements (cherry picked from commit d47f6e8) ref #20480 expand out Pair{<:AbstractString,ResolveBacktraceItem} for 0.5

tkelman added 4 commits May 4, 2017 14:15

Revert "use newvar! instead of add_slot!"

4504215

This reverts commit c250310.

Revert "Introduce temporary if necessary to inline _apply"

86bdc39

This reverts commit 8a4e1be.

Revert "Ensure linearindices completely inlines (through _length)"

3a873fe

This reverts commit e1c9a4b.

Revert "More fixes for non-1 arrays"

6accda3

This reverts commit 39cd805.

fix line numbers in a doctest

87448f2

staticfloat mentioned this pull request May 5, 2017

[release-0.5] Backport #19022 #21708

Closed

fredrikekre mentioned this pull request May 5, 2017

Promote hcat vcat JuliaArrays/StaticArrays.jl#158

Merged

tkelman merged commit 80ee8b7 into release-0.5 May 6, 2017

tkelman deleted the tk/backports-0.5.2 branch May 6, 2017 15:47

This was referenced May 6, 2017

[release-0.5] Backport of #21069 #21080

Merged

fix #21311, inlining _apply reordering evaluations #21319

Merged

More fixes for non-1 arrays #21251

Merged

bjarthur mentioned this pull request May 8, 2017

splitting out cairo backend GiovineItalia/Compose.jl#250

Closed

Uh oh!

[release-0.5] backports for 0.5.2 #21684

[release-0.5] backports for 0.5.2 #21684

Uh oh!

Conversation

tkelman commented May 3, 2017

Uh oh!

tkelman commented May 3, 2017

Uh oh!

tkelman commented May 3, 2017

Uh oh!

tkelman commented May 4, 2017

Uh oh!

tkelman commented May 4, 2017

Uh oh!

nanosoldier commented May 5, 2017

Uh oh!

staticfloat commented May 5, 2017

Uh oh!

tkelman commented May 5, 2017

Uh oh!

staticfloat commented May 5, 2017

Uh oh!

tkelman commented May 5, 2017

Uh oh!

tkelman commented May 5, 2017

Uh oh!

timholy commented May 5, 2017

Uh oh!

staticfloat commented May 5, 2017

Uh oh!

tkelman commented May 5, 2017

Uh oh!

tkelman commented May 5, 2017

Uh oh!

tkelman commented May 5, 2017

Uh oh!

nanosoldier commented May 5, 2017

Uh oh!

staticfloat commented May 5, 2017

Uh oh!

martinholters commented May 5, 2017

Uh oh!

tkelman commented May 5, 2017

Uh oh!

Uh oh!