That is how browsers work, when the URL does not change, when you open the modal window, it will call history.back().

You can work around that, by doing something like this:

HTML

<button>Click to view the video</button>

JavaScript

(function(window, document){
$(window).on('popstate', function() {
  if(window.location.hash === '#kylefoxrocks') {
    $('.jquery-modal').remove();
  }
});
$('button').on('click', function(evt) {
	evt.preventDefault();
	window.location.href = '#kylefoxrocks'
	window.location.href = '#modalopened'
	var video = document.createElement('video');
	var src = document.createElement('source');
	video.setAttribute('controls', '');
	src.setAttribute('src', 'https://cdn.emkae.net/magic-mike-xxl-i-want-it-that-way.mp4');
	src.setAttribute('type', 'video/mp4');
	video.appendChild(src);
	$(video).appendTo('body').modal();
});
})(window, document);

See the Demo
Demo Source Code

I have the same issue using Kyle Fox modal in my single page website where I load external content in full screen modal. The user leaves website when he taps browser back button instead of modal close button.

I try to resolving it by manipulating the history API with Jquery.
In the example code bellow, "window.history.pushstate" and "window.history.back" methods works well when I click the modal open and close button, but I can't figure out how to handle browser back and forward button to make modal reopen on forward and close on back. It's something to do with "popstate".

Any help ?

Here is an example code:

Index page:

<a href="ajax-content.html" data-modal>Load ajax content</a>

Content load in modal (ajax-content.html):

<p>I'm ajax content.</p>
<a href="#" id="closemodal">Close</a>

Jquery:

<script>

// Open modal and history pushstate
$('a[data-modal]').click(function(event) {
    $(this).modal();
    window.history.pushState(null, null, this.href);
    return false;
});

// Close modal and history back
$("body").on('click', '#closemodal', function () {
    $.modal.close();
    window.history.back();
    return false;
});

// Browser back and forward button   
$(window).on('popstate', function () {
    ... ???
});

</script>

Any help would be most welcome ?