Complex64 not working

[vchuravy]

as discussed in #21 Complex64 is currently not working and we are getting a seqfault when passing it to clblas, whereas Complex128 works without issue.

Complex64 maps to cl_float2 and Complex128 maps to cl_double2.

The definitions of both types in cl_platform.h is:

typedef union
{
    cl_float  CL_ALIGNED(8) s[2];
#if __CL_HAS_ANON_STRUCT__
   __CL_ANON_STRUCT__ struct{ cl_float  x, y; };
   __CL_ANON_STRUCT__ struct{ cl_float  s0, s1; };
   __CL_ANON_STRUCT__ struct{ cl_float  lo, hi; };
#endif
#if defined( __CL_FLOAT2__) 
    __cl_float2     v2;
#endif
}cl_float2;

typedef union
{
    cl_double  CL_ALIGNED(16) s[2];
#if __CL_HAS_ANON_STRUCT__
   __CL_ANON_STRUCT__ struct{ cl_double  x, y; };
   __CL_ANON_STRUCT__ struct{ cl_double s0, s1; };
   __CL_ANON_STRUCT__ struct{ cl_double lo, hi; };
#endif
#if defined( __CL_DOUBLE2__) 
    __cl_double2     v2;
#endif
}cl_double2;

The only difference I can see is that cl_float2 is using 8bit alignment and cl_double2 is using 16bit alignment and if I remember correctly Julia uses 16bit alignment for nearly everything.

Complex is defined here:
https://github.com/JuliaLang/julia/blob/02aeb44299d090d50d2c58e004f58a8b8d4f3da6/base/complex.jl

[vchuravy]

On v0.4 https://travis-ci.org/JuliaGPU/CLBLAS.jl/jobs/103056335
on nightly: 0.5 https://travis-ci.org/JuliaGPU/CLBLAS.jl/jobs/103056337

Trace of calls
https://github.com/clMathLibraries/clBLAS/blob/8b5f7a0e6f800b9597319d71f70fbf67e410b004/src/library/blas/xaxpy.c#L116

https://github.com/clMathLibraries/clBLAS/blob/8b5f7a0e6f800b9597319d71f70fbf67e410b004/src/library/blas/generic/solution_seq_make.c#L332

[vchuravy]

And the only function that really calls clGetCommandQueueInfo https://github.com/clMathLibraries/clBLAS/blob/8b5f7a0e6f800b9597319d71f70fbf67e410b004/src/library/blas/generic/solution_seq_make.c#L374 and https://github.com/clMathLibraries/clBLAS/blob/8b5f7a0e6f800b9597319d71f70fbf67e410b004/src/library/blas/generic/solution_seq_make.c#L503-L504

in https://github.com/clMathLibraries/clBLAS/blob/8b5f7a0e6f800b9597319d71f70fbf67e410b004/src/library/blas/generic/common.c#L311 and
https://github.com/clMathLibraries/clBLAS/blob/8b5f7a0e6f800b9597319d71f70fbf67e410b004/src/library/blas/generic/common.c#L296

so my idea is that it probably is not related to alignment, but something funky going on with regards to queues (or that because of misaligment, we are overriding queue information).

[vchuravy]

@dfdx When building with debug information I got it to happen with Float64 and since we also get an invalid value error it seems to have more to do with the interaction between OpenCL.jl and CLBLAS runtime. Do you know if there is any asynchronous operations happening?

[dfdx]

Currently all high-level functions in CLBLAS add task to a queue and return corresponding event. However, in tests we immediately call cl.wait(), so there's always only one task in the queue. jl.

BTW and coming a little bit back, I think sizeof() allows to "guess" alignment pretty accurately :

type T1 x::Int8 end; sizeof(T1)                # => 1
type T2 x::Int8; y::Int8 end; sizeof(T2)    # => 2
type T3 x::Int8; y::Int16 end; sizeof(T3)  # => 4

So any type larger than 8 bytes is aligned to 16 bytes. Complex64 has size 8 bytes, i.e. it's maximally packed. I don't really know what aligned(8) means for struct { float; float}, but I'm pretty much sure it has the same 8 bytes, which confirms your analysis.

[vchuravy]

@dfdx Here is a travis build that build clBLAS from scratch with debug information enabled and it clearly tells us where the seqfault comes from https://travis-ci.org/JuliaGPU/CLBLAS.jl/jobs/103249068#L1672

Thats on the branch https://github.com/JuliaGPU/CLBLAS.jl/tree/vc/complex32

[vchuravy]

Maybe we are encountering something similar to clMathLibraries/clBLAS#187

[vchuravy]

@dfdx I won't have time this week to look into this, feel free to continue the bug hunt or ping me in a week or so.

[dfdx]

Got it. I think I'll have some spare time later this week to debug it.

[dfdx]

For convenience of debugging, here's a short test calling clBLAS directly without all CLBLAS.jl's wrappers:

import OpenCL: CLArray, CL_float
const cl = OpenCL
import CLBLAS: CL_float2

const libCLBLAS = "libclBLAS"

dev, ctx, q = cl.create_compute_context()
CLBLAS.setup()

num_queues = cl.CL_uint(1)
queues = Ptr{Void}[q.id]
num_events = cl.cl_uint(0)
events = Ptr{Void}[]       
ret_event = Array(cl.CL_event,1)

N = 10
DX = Complex64(2.0)
X = cl.ones(Complex64, q, 10)

err = ccall((:clblasCscal, libCLBLAS), cl.CL_int,
            (Csize_t, CL_float, Ptr{Void}, Csize_t, Cint,
             cl.CL_uint, Ptr{Ptr{Void}}, cl.CL_uint, Ptr{Ptr{Void}},
             Ptr{Ptr{Void}}),
            N, DX, pointer(X), Csize_t(0), Cint(1),
            cl.CL_uint(1), pointer(queues), cl.CL_uint(0),
            pointer(events), pointer(ret_event))

if err != cl.CL_SUCCESS
    throw(cl.CLError(err))
end

Which gives CL_INVALID_EVENT_WAIT_LIST (code=-57), while same thing for Complex128 and clblasZscal produces no error.

[vchuravy]

If the event list is empty you need to pass in C_NULL and not a pointer to
an empty list.

--edit:
Never mind: You are hitting https://github.com/clMathLibraries/clBLAS/blob/master/src/library/blas/xaxpy.c#L91-L94 while numEventsInWaitList should be equal to zero but it isn't...

[vchuravy]

so running the script under lldb

lldb julia -- clblas.jl
(lldb) breakpoint set -n clblasCscal
(lldb) r
Process 10392 launched: '/usr/bin/julia' (x86_64)
1 location added to breakpoint 1
error: libclBLAS.so 0x00059975: DW_TAG_member 's' refers to type 0x0005999a which extends beyond the bounds of 0x00059904
Process 10392 stopped
* thread #1: tid = 10392, 0x00007ffdc891cf31 libclBLAS.so`::clblasCscal(N=10, alpha=cl_float2 at 0x00007fffffffc510, X=0x0000000001ac3530, offx=0, incx=1, numCommandQueues=1, commandQueues=0x00007ffdf4920d10, numEventsInWaitList=4103052928, eventWaitList=0x00007ffdf49213f0, events=0x00007ffd40000000) + 52 at xscal.cc:156, name = 'julia', stop reason = breakpoint 1.1
    frame #0: 0x00007ffdc891cf31 libclBLAS.so`::clblasCscal(N=10, alpha=cl_float2 at 0x00007fffffffc510, X=0x0000000001ac3530, offx=0, incx=1, numCommandQueues=1, commandQueues=0x00007ffdf4920d10, numEventsInWaitList=4103052928, eventWaitList=0x00007ffdf49213f0, events=0x00007ffd40000000) + 52 at xscal.cc:156
   153      const cl_event *eventWaitList,
   154      cl_event *events)
   155  {
-> 156    CHECK_QUEUES(numCommandQueues, commandQueues);
   157    CHECK_EVENTS(numEventsInWaitList, eventWaitList);
   158    CHECK_VECTOR_X(TYPE_COMPLEX_FLOAT, N, X, offx, incx);
   159  
(lldb) po numCommandQueues
1

(lldb) po numEventsInWaitList
4103052928

First thing I noticed is that you have an error in your ccall, Second argument should be cl_float2 and not cl_float.

err = ccall((:clblasCscal, libCLBLAS), cl.CL_int,
            (Csize_t, CL_float2, Ptr{Void}, Csize_t, Cint,
             cl.CL_uint, Ptr{Ptr{Void}}, cl.CL_uint, Ptr{Ptr{Void}},
             Ptr{Ptr{Void}}),
            N, DX, pointer(X), Csize_t(0), Cint(1),
            cl.CL_uint(1), pointer(queues), cl.CL_uint(0),
            pointer(events), pointer(ret_event))

But even after that:

(lldb) p alpha
(cl_float2) $2 = {
  s = {}
   = (x = 0, y = 0)
   = (s0 = 0, s1 = 0)
   = (lo = 0, hi = 0)
  v2 = (0, 0)
} 

(lldb) po numEventsInWaitList
4103052928

[dfdx]

I'm not really familiar with lldb (or gdb), do I understand it correctly that just after ccalling clblasCscal value of input parameter numEventsInWaitList is equal 4103052928?

[vchuravy]

Yes and alpha e.g. DX is not correctly passed through. So I am back
thinking about misalignment or misrepresentation of cl_float2

On Fri, 22 Jan 2016, 06:47 Andrei Zhabinski notifications@github.com
wrote:

I'm not really familiar with lldb (or gdb), do I understand it correctly
that just after ccalling clblasCscal value of input parameter
numEventsInWaitList is equal 4103052928?

—
Reply to this email directly or view it on GitHub
#23 (comment).

[vchuravy]

I just created a small c-example to see if I could figure out the correct way of working with Complex64 and cl_float2 and if I compile the library with gcc and the test program with clang I get the same problem on the C level.

Right now the only way of solving this for users is to recommend to them to compile clBLAS with clang/clang++

[dfdx]

Sounds unpleasant, but reasonable. I'll add corresponding note to the README.