Skip to content

Monitor queues when used for rebuilds - #166

Open
martinsumner wants to merge 1 commit into
openriak-3.4from
nhse-o34-orkv.i165-rebuildcrash
Open

Monitor queues when used for rebuilds#166
martinsumner wants to merge 1 commit into
openriak-3.4from
nhse-o34-orkv.i165-rebuildcrash

Conversation

@martinsumner

Copy link
Copy Markdown
Contributor

if a queue is used for rebuild, it is important to detect the failure of the pool, and not leave the AAE system in limbo.

Pools are monitored during a rebuild, and AAE is restarted should the pool fail before the rebuild completes.

The state has been extended on riak_kv_vnode:

  • tictacaae_rebuilding now has {TS, Ref} not just TS as the state during rebuild, whereby Ref is a reference of a monitor of the pool;
  • there is a a list of {reference(), Fun/1} pairs, if a 'DOWN' message is received for reference(), then the riak_kv_vnode state is updated with Fun(State) (and in this case this will restart AAE and update the controller pid on State).

Changes made to riak_kv_tictacaae_cli:

  • Export of function for use in testing;
  • Longer pause on sys:get_state (as the aae_controller restart in parallel mode may take up to 10s as there is an outstanding snapshot on the parallel store for the rebuild).

if a queue is used for rebuild, it is important to detect the failure of the pool, and not leave the AAE system in limbo.

Pools are monitored during a rebuild, and AAE is restarted should the pool fail before the rebuild completes.
@martinsumner

Copy link
Copy Markdown
Contributor Author

#165

@martinsumner

Copy link
Copy Markdown
Contributor Author

Tested in OpenRiak/riak_test#32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant