partisan_plumtree_broadcast crashing

[mcesaro]

Likely related to #251, when nodes go down and up a few times (usually when debugging apps!), dynamic node names appear apparently randomly, and for some reason this leads to a crash of partisan_plumtree_broadcast:

=ERROR REPORT==== 25-Oct-2023::15:00:14.183849 ===
** Generic server partisan_plumtree_broadcast terminating 
** Last message in was {'$gen_cast',
                           {broadcast,
                               {{{erleans_pm,registry},
                                 {'pebble@max-a5',
                                     {<<"01HDH23JRRXK9RYQC8K43ETEV9">>,
                                      pebble}}},
                                [{{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  4},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  6},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  6},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  4},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  2},
                                 {{12,
                                   '[email protected]'},
                                  1}]},
                               {object,
                                   {[{{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      4,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      6,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,
                                      [{'$deleted',{1698,238814,182073}}]},
                                     {{12,
                                       '[email protected]'},
                                      6,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      4,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      2,[]},
                                     {{12,
                                       '[email protected]'},
                                      1,[]}],
                                    []}},
                               plum_db,0,'pebble@max-a5','pebble@max-a5'}}
** When Server state == {state,['clockwork@max-a5','pebble@max-a5',
                                'test@max-a5'],
                               [],#{},#{},
                               [partisan_plumtree_backend],
                               [],
                               ['clockwork@max-a5','pebble@max-a5',
                                'ska@max-a5','test@max-a5'],
                               1000,10000}
** Reason for termination ==
** {'module could not be loaded',
       [{plum_db,merge,
            [{{{erleans_pm,registry},
               {'pebble@max-a5',{<<"01HDH23JRRXK9RYQC8K43ETEV9">>,pebble}}},
              [{{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},4},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},6},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},6},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},4},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},2},
               {{12,'[email protected]'},1}]},
             {object,
                 {[{{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    4,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    6,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,
                    [{'$deleted',{1698,238814,182073}}]},
                   {{12,'[email protected]'},
                    6,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    4,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    2,[]},
                   {{12,'[email protected]'},
                    1,[]}],
                  []}}],
            []},
        {partisan_plumtree_broadcast,handle_cast,2,
            [{file,
                 "/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
             {line,549}]},
        {gen_server,try_handle_cast,3,[{file,"gen_server.erl"},{line,1103}]},
        {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1165}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}

=CRASH REPORT==== 25-Oct-2023::15:00:14.192338 ===
  crasher:
    initial call: partisan_plumtree_broadcast:init/1
    pid: <0.18755.0>
    registered_name: partisan_plumtree_broadcast
    exception error: undefined function plum_db:merge/2
      in function  partisan_plumtree_broadcast:handle_cast/2 (/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl, line 549)
      in call from gen_server:try_handle_cast/3 (gen_server.erl, line 1103)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1165)
    ancestors: [partisan_sup,<0.1077.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1078.0>,<0.1086.0>]
    dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
                                bits => 58,uniform => #Fun<rand.1.65977474>,
                                uniform_n => #Fun<rand.2.65977474>,
                                jump => #Fun<rand.3.65977474>},
                              [213968340301438561|119831794556142432]}}]
    trap_exit: false
    status: running
    heap_size: 46422
    stack_size: 28
    reductions: 77660
  neighbours:

=SUPERVISOR REPORT==== 25-Oct-2023::15:00:14.192677 ===
    supervisor: {local,partisan_sup}
    errorContext: child_terminated
    reason: {undef,
                [{plum_db,merge,
                     [{{{erleans_pm,registry},
                        {'pebble@max-a5',
                            {<<"01HDH23JRRXK9RYQC8K43ETEV9">>,pebble}}},
                       [{{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         4},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         6},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         6},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         4},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         2},
                        {{12,'[email protected]'},
                         1}]},
                      {object,
                          {[{{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             4,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             6,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,
                             [{'$deleted',{1698,238814,182073}}]},
                            {{12,
                              '[email protected]'},
                             6,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             4,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             2,[]},
                            {{12,
                              '[email protected]'},
                             1,[]}],
                           []}}],
                     []},
                 {partisan_plumtree_broadcast,handle_cast,2,
                     [{file,
                          "/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
                      {line,549}]},
                 {gen_server,try_handle_cast,3,
                     [{file,"gen_server.erl"},{line,1103}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1165}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.18755.0>},
               {id,partisan_plumtree_broadcast},
               {mfargs,{partisan_plumtree_broadcast,start_link,[]}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,5000},
               {child_type,worker}]

It seems that the plum_db:merge function is not available because the module is not loaded.

[aramallo]

@mcesaro I need to take a look at this to see whether the issue is in Partisan, PlumDB or Erleans (our fork).

[mcesaro]

@aramallo Hi,
it's happening also on non-erlans applications:

=CRASH REPORT==== 26-Oct-2023::08:28:08.676084 ===
  crasher:
    initial call: partisan_plumtree_broadcast:init/1
    pid: <0.18966.0>
    registered_name: partisan_plumtree_broadcast
    exception error: undefined function plum_db:merge/2
      in function  partisan_plumtree_broadcast:handle_cast/2 (/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl, line 549)
      in call from gen_server:try_handle_cast/3 (gen_server.erl, line 1103)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1165)
    ancestors: [partisan_sup,<0.1007.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1008.0>,<0.1019.0>]
    dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
                                bits => 58,uniform => #Fun<rand.1.65977474>,
                                uniform_n => #Fun<rand.2.65977474>,
                                jump => #Fun<rand.3.65977474>},
                              [125007732914024471|257206006587947701]}}]
    trap_exit: false
    status: running
    heap_size: 610
    stack_size: 28
    reductions: 40368
  neighbours:

=SUPERVISOR REPORT==== 26-Oct-2023::08:28:08.676618 ===
    supervisor: {local,partisan_sup}
    errorContext: child_terminated
    reason: {undef,
                [{plum_db,merge,
                     [{{{erleans_pm,registry},
                        {'pebble@max-a5',
                            {<<"01HDNAMM2S470TFV43NFD2GMNE">>,pebble}}},
                       [{{12,'e8a18d8a-73c7-11ee-bbf0-1d718ad5fce5@max-a5'},
                         2}]},
                      {object,
                          {[{{12,
                              'e8a18d8a-73c7-11ee-bbf0-1d718ad5fce5@max-a5'},
                             2,
                             [{'$deleted',{1698,301688,672946}}]}],
                           []}}],
                     []},
                 {partisan_plumtree_broadcast,handle_cast,2,
                     [{file,
                          "/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
                      {line,549}]},
                 {gen_server,try_handle_cast,3,
                     [{file,"gen_server.erl"},{line,1103}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1165}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.18966.0>},
               {id,partisan_plumtree_broadcast},
               {mfargs,{partisan_plumtree_broadcast,start_link,[]}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,5000},
               {child_type,worker}]

I think I can trigger this behaviour in my laptop by suspending it on Wifi1 and waking it on Wifi2.
It seems that when there is a network change, partisan still try to use the last known network.
Admittely not a common use case, but still it might be a nuisance. A potential workaround would be to have a function/API to signal the network change and reset all the connections; otherwise, the effect is that the running nodes are stuck in the try-to-reconnect loop, but also that new nodes cannot join the cluster.

[aramallo]

@mcesaro Thanks for the extra info. However, notice that since plum_db is in your logs, you are not running just Partisan, or you running plum_db directly? Otherwise, you are still running erleans (plum_db is a dependency in our fork).

[mcesaro]

@aramallo Hi, my bad. One of the servers also includes erleans as a dependency. So, I re-checked the dependencies for all the applications; I'm not explictly using erleans in a distributed mode as only one server is currently running erleans, but the network swtich issue now is limited to the host running erleans with plum_db.