Consider how we can handle emptyDir

[SaaldjorMike]

We generally won't recommend emptyDir for storage nodes. We should probably do something to make it clear that this is not recommended to use, unless in very specific use-cases.

[SaaldjorMike]

Perhaps we can check for certain cases, like httponly nodes & ephemeral disks, which would allow emptyDir.
But if we can't know for certain if a given configuration is safe to use emptyDir with, then we may want to introduce a flag to opt-in to emptyDir, with documentation etc. describing when it may make sense to use emptyDir - and also when it does not.

[bderrly]

This might have saved us a lot of heartache recently. I set up our production cluster with emptyDir not understanding the full implications of it. Now we are paying the price when there are node pool upgrades on our GKE cluster. I just filed #680 which I think can help us even after we make the move to a local PVC.