The currently unused Mmapper::protect

[wks]

Mmapper::protect was designed to do two things:

1. Use the mprotect system call to prevent access to a memory range, and
2. record in the array of MapState that those chunks are in the MapState::Protected state.

It works at the granularity of Chunk, as mentioned in JikesRVM's doc.

Back in JikesRVM, it was used for one purpose: when the Options.protectOnRelease option is enabled, it will protect the memory allocated by CopySpace when the CopySpace is released.

It is never used in the Rust MMTk. But some places in the Rust MMTk bypass the Mmapper and call mprotect directly. Here is a summary.

• CopySpace::protect and CopySpace::unprotect: These two methods were intended to implement the Options.protectOnRelease in JikesRVM, but not fully implemented, and remain dead code.
• FreeListPageResource::mprotect: It is used by the LOS to implement the PageProtect plan.

The Mmapper::protect method is not helpful

The chunk-grained Mmapper::protect method works for CopySpace, but it is not necessary. CopySpace resets the whole space at a time, and its memory is either contiguous (Map64) or obtained in whole chunks from VMMap::allocate_contiguous_chunks. But if we just want to protect the memory, we can call mprotect directly, too.

But it doesn't work for LOS. An object in the LOS occupies whole pages, not whole chunks. That's finer than the chunk-grained Mmapper.

The MapState::Protected state is not helpful, either

The MapState::Protected state is also not helpful. It gives a chunk a special state of Protected. But in both of the use cases shown above, mprotect was intended to catch use-after-free bugs. If we make Protected a proper state, it will allow the user to check for it (for example, addr.is_mapped() will return false) and choose not to access the memory if it is protected. We don't have this need, and segmentation fault is actually the expected behavior.

The Protected state is used in MapState::transition_to_mapped. If the memory is already protected, we just call munprotect instead of dzmmap. But this is also not necessary. If we call mmap on an already mapped range with the MAP_FIXED flag and a new permission, it will overwrite the existing mmap, and set it to the new permission.

The Protected state is also used in MapState::transition_to_quarantined so that it panics if we try to quarantine protected memory. In a normally executing program, it should not happen. The map state transition should be like the following:

Unmapped -> (Quarantined) -> Mapped <--> Protected

The state only monotonically change from Unmapped to Mapped, optionally via Quarantined, and we only ever protect mapped memory. We can remove the Protected state, and leave the state as Mapped when calling mprotect (which is the actual behaviour of the current PageProtect plan). If we try to quarantine memory that is already Mapped, it should be an error. But note that we currently have the following comment in MapState::transition_to_quarantined:

            MapState::Mapped => {
                // If a chunk is mapped by us and we try to quarantine it, we simply don't do anything.
                // We allow this as it is possible to have a situation like this:
                // we have global side metadata S, and space A and B. We quarantine memory X for S for A, then map
                // X for A, and then we quarantine memory Y for S for B. It is possible that X and Y is the same chunk,
                // so the chunk is already mapped for A, and we try quarantine it for B. We simply allow this transition.
                return Ok(());
            }

This will only happen if we lazily quarantine metadata memory. This doesn't make sense. Actually it beats the purpose of quarantining, i.e. preventing mmap(NULL, ...) invocations initiated by third-party libraries from accidentally allocating memory in MMTk's heap range. If we implement #1346 (or even just quarantine the spaces of the plan and the side metadata memory instead of the entire heap), the situation described in this comment will not happen.

Conclusion

In conclusion, neither the Mmapper::protect method nor the MapState::Protected state is helpful. We can simply remove them. Then we can simplify the state transition graph to just

Unmapped -> (Quarantined) -> Mapped

where the transition Unmapped -> Quarantined only happens at start-up time.

Debugging needs

When it triggers segmentation fault, we may want to know the reason why it happens, e.g. whether it is because mmtk-core protected it, or because some third-party code protected it and what was the reason. We have the MmapAnnotation type which is already used by the mmapper. It will display a description in /proc/pid/maps. We can use the same mechanism (PR_SET_VMA_ANON_NAME) for mprotect, too.