Eliminate HAVE_MPI conditional compilation

[markcmiller86]

We made the MPI code conditionally compiled originally on the idea that maybe multiple forms of parallelism would be supported here. But, do we expect to be doing anything useful at large scale without MPI? If not, having MPI conditionally compiled only complicates the code without much benefit and it should be removed.

[markcmiller86]

So, my last commit just made this problem a bit worse. I added a bunch of conditional compilaton to make the code compile in serial. That seemed useful for debugging vaarious issues.