Inspiration from PETSc ODE/DAE solvers for API and algorithms

[mauro3]

This is to propose to support the API PETSc uses for its ODE/DAE solvers.
I think the PETSc framework is the nicest around and the most modern (afaik).
(Other issues related are #20, #18, JuliaLang/julia#75)

The PETSc ODE/DAE solvers have been implemented over the last few years with modern developments in mind (they are citing papers from 2003, 2009), which is more than can be said for most other suites of ODE solvers which carry decades old ballast around. It supports, explicit, implicit and IMEX methods for ODEs and DAEs all within the same API. The API is based around formulating the problem like

F(t, u, u') = G(t, u)

For problems using implicit methods this is used as F(t, u, u') =0 for
explicit ones to u'=G(t, u) (i.e. F=u'), IMEX mixes the two at once.

Best have a look at the PETSc manual which gives a nice concise intro (Chapter 6):
http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf

PETSc has a liberal licence so code could probably be translated
from PETSc into Julia. The problem with that approach is that
one needs to understand the whole PETSc framework, otherwise
the code looks quite obfuscated.

Further, some cool solvers PETSc has:

• General linear methods:
  http://www.mcs.anl.gov/petsc/petsc-3.4/docs/manualpages/TS/TSGL.html#TSGL
  These methods contain Runge-Kutta and multistep
  schemes (e.g. BDF) as special cases. As far as I can tell with
  my limited ODE/DAE solver knowledge, these are the best
  currently available.
• IMEX: implicit explicit methods
  http://www.mcs.anl.gov/petsc/petsc-3.4/docs/manualpages/TS/TSARKIMEX.html
  "These methods are intended for problems with well-separated
  time scales, especially when a slow scale is strongly nonlinear
  such that it is expensive to solve with a fully implicit
  method."

Disclaimer: I have not actually used the PETSc solvers, just read
the docs and wished I could use them in Matlab...

[fsaporito]

I read over the project list in the GSOC page on the julia site (julialang.org/gsoc/2014/)
There is this project proposal:

"Project: PETSc integration for scalable technical computing

PETSc is a widely used framework of data structures and computational routines suitable for massively scaling scientific computations. Many of these algorithms are also ideally suited for big data applications such as computing principal components of very large sparse matrices and solving complicated forecasting models with distributed methods for solving partial differential equations.

This project proposal is to develop a new Julia package to interface with PETsc, thus allowing users access to state of the art scalable algorithms for optimization, eigenproblem solvers, finite element mesh computations, and hyperbolic partial differential equation solvers. The more mathematically oriented student may choose to study the performance of these various algorithms as compared to other libraries and naïve implementations. Alternatively, students may also be interested in working on the LLVM BlueGene port for deploying Julia with PetSc integration in an actual supercomputing environment.

Expected Results: Wrapper package for the PETSc suite of libraries."

It's a good idea to recreate the API in native Julia when, if someone does this project, the PETCs framework can be called directly from Julia?
In my opinion it's worthwhile to do so, for two main reasons, but others may think otherwise:

1. Speed
   While surely the PETSc implementation is very optmised, at such level that it would be difficult to reach within a reasonable time, calling over an external library over and over is computationally heavy, and for Julia would means to grow slower.

2. Glue Language
   If we start from creating interfaces to other libraries instead of coding ours implementations, and the process continues, in a few time Julia will slowly became a glue languages for linking external libraries in a common framework. While I don't believe that a language like that is a total garbage, since it could have a few uses, I don't think that this is the goal for the Julia language that anybody had in mind.

What is your opinion on this?

[jiahao]

You're quite right to be concerned at the possibility of overhead because of calling an external library. In fact, benchmarking the PETSc interface vs one or a few particular algorithms reimplemented in native Julia could make for a very nice component of a summer project. The usual counterargument is that computing the functions and its derivatives will probably be expensive anyway, but it's an assertion that can be tested.

[stevengj]

The main reason to implement this in pure Julia is to support ODE integration with arbitrary types; basically, you want to support integrating ODE functions over any normed vector space, i.e. any type that supports at least +, -, * by scalar, and a norm. (Not to mention support for different scalar types, e.g. BigFloat.)

[ivarne]

Another reason to implement in Julia, is that many more people will be able to dive into the source code for the solvers they are using if it is written in Julia.

This topic for this issue is anyway to draw inspiration from the API's for the PETSc solvers, into our own API discussions, not reimplementing everything in Julia. If someone are going to create wrappers for PETSc in Julia, it would be nice if they could make it effortlessly as a new backend for the solvers in ODE.jl

[mauro3]

Here a quick comparison of the PETSc and the matlab API for stiff solvers, e.g. Matlab's ode15s and ode15i.

ode15s can be applied to problems of the form M(t,u) u'=G(t,u) where M is called the mass matrix. Thus it can be applied to nearly as general problems as above PETCs setup. The mass matrix needs to be set in the options structure with Mass, MStateDependence, MvPattern and MassSingular. This is a bit clunky, considering that the mass matrix is a central part of the equation and not an option.

ode15i can be applied to problems of the form F(t, u, u') = 0, i.e. as the LHS of the PETSc setup. Here there is no need to set the mass matrix. (however, ode15i seems to be a lot less efficient that ode15s)

The PETSc interface unifies the two matlab interfaces quite nicely. Also solvers which only support the F or the G term would only implement that. Also in PETSc the difference between F and G is that for F a Jacobian is needed (either provided of by finite difference) whereas for G non Jacobian is needed as an explicit method is applied to that.

(PS: anther reason to re-implement some of the PETSc-solvers in Julia is that PETSc is a big dependency which involves a lot of compilation. Too much to ask of someone just wanting to solve a ODE.)

[tomasaschan]

If we decided to generalize the API to the form PETSc uses, i.e. ode(F, G, y0, tspan; kwargs...), the solvers we use today would only be able to use one of F and G to define the system. If I understand the PETSc API correctly, that's the case in their solvers as well.

What is a good way to handle the case when the user calls a solver with both F and G specified, but the solver cannot handle that type of systems? How do we detect it, and how do we let the user know they're doing something that won't work?

I think the idea of the API is really nice, and perhaps it's possible to create a wrapper function which, much like e.g. the \ operator, inspects the input arguments and chooses a solver based on the properties of the problem. The user would simply call ode and let the library (try to) work it out. If one needs more direct control over what solver method is used (e.g. specific performance/accuracy requirements), the specific solver methods are of course also available.

(This comment somehow disappeared when I posted it about an hour ago...)

[stevengj]

Regarding API, I think we should prescribe both:

ts,ys = odeXX(G, y0, tspan; kws...)  # solve y' = G(t,y)
ts,ys = odeXX(F, G, y0, tspan; kws...) # solve F(t,y,y') = G(t,y)

Not all solvers will support the second form, which is fine; they will just throw a MethodError.

All solvers which support the second form should also define the first form. Most simply, they may of course define odeXX(G, y0, tspan; kws...) = odeXX((t,y,yp) -> yp, G, y0, tspan; kws...), but I expect that they may want to have special-case optimizations for F = y'.

[mauro3]

This sounds good and would also be compatible with the API already defined.

[stevengj]

On second thought, although we should definitely have a simple interface odeXX(G, y0, tspan), I'm not sure that odeXX(F, G, y0, tspan) is sufficient for the complex interface, because you potentially want to specify other pieces of information, such as the Jacobian of F (and for efficiency you really want to allow a single function to compute and return both F and the Jacobian dF/dy' at the same time), whether F or G is linear, and so on.

(PETSc has you create a problem "object" and provides various methods to modify its properties. I suppose we could have something in a similar spirit but more Julian.)

[mauro3]

Matlab also sets the Jacobian or its sparsity structure in the ode-options.

We could have a keyword argument, say FJac&GJac, to either (1) pass a Jacobian function to, or (2) to provide a sparsity pattern (for finite differencing), or (3) to indicate to use automatic differentiation, or (4) to indicate that F or G return both F or Gand its Jacobian. If we wanted to use dispatch instead, then maybe a trick like #31 may work (until the Function-type contains its signature...if ever).