Added tests and updated READMD.md

Author: jxy-s

Diff for: README.md

@@ -134,23 +134,36 @@ integers                 : { size=10 } [Type: std::vector<int,jxy::details::allo
 
 Below is table of functionality under the `jxy` namespace:
 
-| jxylib | STL equivalent | Notes |
-| ------ | -------------- | ----- |
-| `jxy::allocator` | `std::allocator` | |
-| `jxy::default_delete` | `std::default_delete` | |
-| `jxy::unique_ptr` | `std::unique_ptr` | |
-| `jxy::shared_ptr` | `std::shared_ptr` | |
-| `jxy::basic_string` | `std::basic_string` | |
-| `jxy::string` | `std::string` | |
-| `jxy::wstring` | `std::wstring` | |
-| `jxy::vector` | `std::vector` | |
-| `jxy::map` | `std::map` | |
-| `jxy::mutex` | `std::mutex` | Uses `KGUARDED_MUTEX` |
-| `jxy::shared_mutex` | `std::shared_mutex` | Uses `EX_PUSH_LOCK` |
-| `jxy::unique_lock` | `std::unique_lock` | |
-| `jxy::shared_lock` | `std::shared_lock` | |
-| `jxy::scope_resource` | None | Similar to `std::experimental::unique_resource` |
-| `jxy::scope_exit` | None | Similar to `std::experimental::scope_exit` |
+| jxylib | STL equivalent | Include | Notes |
+| ------ | -------------- | ------- | ----- |
+| `jxy::allocator` | `std::allocator` | `<jxy/memory.hpp>` | |
+| `jxy::default_delete` | `std::default_delete` | `<jxy/memory.hpp>` | |
+| `jxy::unique_ptr` | `std::unique_ptr` | `<jxy/memory.hpp>` | |
+| `jxy::shared_ptr` | `std::shared_ptr` | `<jxy/memory.hpp>` | |
+| `jxy::basic_string` | `std::basic_string` | `<jxy/string.hpp>`| |
+| `jxy::string` | `std::string` | `<jxy/string.hpp>` | |
+| `jxy::wstring` | `std::wstring` | `<jxy/string.hpp>` | |
+| `jxy::vector` | `std::vector` | `<jxy/vector.hpp>` | |
+| `jxy::map` | `std::map` | `<jxy/map.hpp>` | |
+| `jxy::multimap` | `std::miltimap` | `<jxy/map.hpp>` | |
+| `jxy::mutex` | `std::mutex` | `<jxy/locks.hpp>` | Uses `KGUARDED_MUTEX` |
+| `jxy::shared_mutex` | `std::shared_mutex` | `<jxy/locks.hpp>` | Uses `EX_PUSH_LOCK` |
+| `jxy::unique_lock` | `std::unique_lock` | `<jxy/locks.hpp>` | |
+| `jxy::shared_lock` | `std::shared_lock` | `<jxy/locks.hpp>` | |
+| `jxy::scope_resource` | None | `<jxy/scope.hpp>` | Similar to `std::experimental::unique_resource` |
+| `jxy::scope_exit` | None | `<jxy/scope.hpp>` | Similar to `std::experimental::scope_exit` |
+| `jxy::thread` | `std::thread` | `<jxy/thread.hpp>` | |
+| `jxy::deque` | `std::deque` | `<jxy/deque.hpp>` | |
+| `jxy::queue` | `std:queue` | `<jxy/queue.hpp>` | |
+| `jxy::priority_queue` | `std::priority_queue` | `<jxy/queue.hpp>` | |
+| `jxy::set` | `std::set` | `<jxy/set.hpp>` | |
+| `jxy::multiset` | `std::multiset` | `<jxy/set.hpp>` | |
+| `jxy::stack` | `std::stack` | `<jxy/stack.hpp>` | |
+
+## Tests - `stltest.sys`
+
+The `stltest` project implements a driver that runs some tests against jxystl, 
+usage of STL, and exceptions in the Windows Kernel.
 
 ## Practical Usage - `stlkrn.sys` 
 The `stlkrn` project is a Windows Driver that uses `jxylib` to implement 
@@ -227,6 +240,75 @@ use cases. `x86` has *not* been tested.  There is functionality under the
 I would like to continue this work over time, if any issues/bugs are found 
 feel free to open issues against this repo.
 
+## Related Work
+
+This project provides STL support in the Windows Kernel by using as much of the 
+STL facility as possible. There are other solutions for use of STL in kernel 
+development. This section will outline alternatives, first I will summarize 
+this work:
+
+This Project:
+ - Uses the STL directly. Does **not** reimplement any STL functionality unless absolutely necessary.
+ - Requires pool types and tags. No global `new` or `delete` is implemented.
+ - Forbid moving data between objects of different pools or tags.
+ - Avoids CRT initialization and `atexit` functionality. CRT initialization order 
+   is non-obvious, driver initialization and teardown *should be obvious*. `atexit` functionality 
+   may introduce data races for kernel code, `atexit` is not implemented.
+
+[Bareflank Hypervisor][github.bareflank]:
+
+Bareflank implements support for running C++ in their hypervisor. They have full STL and CRT 
+support. This is a comprehensive project that enables a plethora features of the standard in 
+kernel mode (including exceptions). As I understand their solution forces `NonPagedPool` on global 
+`new`/`delete` allocations. I have to commend Bareflank with their implementation, it's well 
+thought out and cross platform. However the Windows implementation builds through cygwin and 
+"shims" in support for the Windows kernel. In comparison, this project aims to be considerate to 
+the Windows kernel. It enables specifying pool tags and types (paged vs non-paged) and hopes 
+to minimize "sharp edges" associated with using C++ and the STL in kernel mode. All that said, 
+Bareflank is impressive for what is does. For an excellent presentation on Bareflank's support of 
+C++ I highly recommend watching [Dr. Rian Quinn's presentation at cppcon 2016][channel9.bareflank].
+
+[Win32KernelSTL][github.Win32KernelSTL]:
+
+The Win32KernelSTL project does allow you to use STL functionality directly in the kernel. The project 
+implements global `new`/`delete` and forces `NonPagedPool`, it implements CRT initialization support, 
+and bugchecks when a cpp exception is thrown. It makes no attempt to do cpp exception unwinding. Due 
+to the assumptions it makes I find it unpractical for any serious use cases. The code is reasonably 
+clear and documented, I recommend giving this project a browse for educating around C++ support in the 
+kernel. One note, the CRT code in Win32KernelSTL does implement `atexit` but keep in mind there is no 
+synchronization emitted by the compiler here (as opposed to user mode). So a local static requiring 
+insertion of an entry in the `atexit` list may race causing a double-init or double-free.
+
+[Driver Plus Plus][github.dxx]:
+
+This project implements necessary C++ facility for pulling in a number of C++ solutions into 
+kernel mode (`EASTL`, `msgpack`, etc.). Driver Plus Plus implements CRT initialization and global 
+`new`/`delete` support (which forces `NonPagedPool`). Again this is counter to the goals of this 
+project. However, this project does enable a lot of great C++ facility for use in kernel mode. It 
+does make modifications to the C++ solutions it pulls in to shim in support for it's use cases. 
+Driver Plus Plus also makes the assumption around `atexit` as mentioned previously.
+
+[KTL][github.ktl]:
+
+KTL (Windows Kernel Template Library) reimplements a good amount of modern C++ functionality for 
+use in the Windows Kernel. It also implements global `new`/`delete` but does a decent job 
+at providing facility for specifying pool tags and types where possible. However this does mean 
+the global allocator might hide an allocation in a non-obvious pool. Further the template 
+allocators in this project carry the cost of two points for an allocator and deallocator object, 
+I am also concerned that conversion between the allocator types may allow for cross pool/tag 
+allocs/frees. Overall I'm impressed by the amount of facility that is implemented here. 
+Reimplementation of STL functionality and the global allocators are counter to the ideologies of 
+this project.
+
+[Kernel-Bridge][github.KernelBridge]:
+
+Kernel-Bridge implements some great facility for Windows Kernel development. The library provides 
+wrappers for registering for Windows callbacks using C++ objects. I would like to find more time 
+to use and investigate this solution. It does implement CRT support. The `atexit` functionality 
+implemented is not dynamic - it uses a static array, if it runs out of slots, it fails. The 
+default `new`/`delete` forces `NonPagedPool`. It does not have full exception support, it will 
+bugcheck if a cpp exception is thrown - it will not unwind objects on the stack.
+
 ## Credits
 This repository draws from some preexisting work. Credits to their authors.
 
@@ -242,3 +324,9 @@ symbol files, as well as a lot of reverse engineering and guessing.
 [github.vcrtl]: https://github.com/avakar/vcrtl
 [github.vcrtl.x64]: https://github.com/avakar/vcrtl/tree/master/src/x64
 [github.phnt]: https://github.com/processhacker/phnt
+[github.bareflank]: https://github.com/Bareflank/hypervisor
+[channel9.bareflank]: https://channel9.msdn.com/Events/CPP/CppCon-2016/CppCon-2016-Rian-Quinn-Making-C-and-the-STL-Work-in-the-Linux--Windows-Kernels
+[github.Win32KernelSTL]: https://github.com/DragonQuestHero/Win32KernelSTL
+[github.dxx]: https://github.com/sidyhe/dxx
+[github.ktl]: https://github.com/MeeSong/KTL
+[github.KernelBridge]: https://github.com/HoShiMin/Kernel-Bridge

Diff for: include/jxy/alloc.hpp

@@ -9,7 +9,7 @@
 // the pool type and tags for all allocations.
 //
 #pragma once
-#include <wdm.h>
+#include <fltKernel.h>
 #include <cstddef>
 
 void* __cdecl operator new(size_t Size, POOL_TYPE PoolType, ULONG PoolTag) noexcept(false);

Diff for: include/jxy/locks.hpp

@@ -27,6 +27,8 @@ class shared_mutex
 {
 public:
 
+    using native_handle_type = PEX_PUSH_LOCK;
+
     shared_mutex() noexcept;
     ~shared_mutex() noexcept;
     void lock() noexcept;
@@ -35,6 +37,7 @@ class shared_mutex
     void lock_shared() noexcept;
     bool try_lock_shared() noexcept;
     void unlock_shared() noexcept;
+    native_handle_type native_handle() noexcept;
 
 private:
 
@@ -47,11 +50,14 @@ class mutex
 {
 public:
 
+    using native_handle_type = PKGUARDED_MUTEX;
+
     mutex() noexcept(false)
     {
-        m_GuardedMutex = ExAllocatePoolWithTag(NonPagedPoolNx,
-                                               sizeof(*m_GuardedMutex),
-                                               t_PoolTag);
+        m_GuardedMutex = static_cast<PKGUARDED_MUTEX>(
+            ExAllocatePoolWithTag(NonPagedPoolNx,
+                                  sizeof(*m_GuardedMutex),
+                                  t_PoolTag));
         if (!m_GuardedMutex)
         {
             throw std::bad_alloc();
@@ -83,6 +89,11 @@ class mutex
         KeReleaseGuardedMutex(m_GuardedMutex);
     }
 
+    native_handle_type native_handle() noexcept
+    {
+        return m_GuardedMutex;
+    }
+
 private:
 
     PKGUARDED_MUTEX m_GuardedMutex = nullptr;

Diff for: include/jxy/memory.hpp

@@ -20,7 +20,7 @@
 // jxy::shared_ptr      std::shared_ptr
 //
 #pragma once
-#include <wdm.h>
+#include <fltKernel.h>
 #include <jxy/alloc.hpp>
 #include <memory>
 

Diff for: include/jxy/scope.hpp

@@ -28,13 +28,17 @@ class scope_resource
     using this_type = scope_resource<T, TReleaseFunctor>;
     using value_type = T;
 
-    scope_resource() = default;
-
     ~scope_resource() noexcept
     {
         TReleaseFunctor()(m_Resource);
     }
 
+    scope_resource() = default;
+
+    scope_resource(T Resource) : m_Resource(Resource)
+    {
+    }
+
     value_type get() const noexcept
     {
         return m_Resource;

Diff for: include/jxy/stach.hpp renamed to include/jxy/stack.hpp

@@ -10,7 +10,7 @@
 // jxy::thread          std::thread
 //
 #pragma once
-#include <ntddk.h>
+#include <fltKernel.h>
 #include <thread>
 #include <jxy/memory.hpp>
 
@@ -38,6 +38,8 @@ class thread
     static constexpr POOL_TYPE pool_type = PagedPool;
     static constexpr ULONG pool_tag = 'TYXJ';
 
+    ~thread();
+
     thread() noexcept = default;
 
     template <typename TFunc, 
@@ -94,7 +96,7 @@ class thread
         std::invoke(std::move(std::get<t_Indices>(tuple))...);
 
         //
-        // Check that no matter what the invoked routine does that it returned
+        // Check that no matter what the invoked routine does it returns
         // to passive.
         //
         NT_ASSERT(KeGetCurrentIrql() == PASSIVE_LEVEL);
@@ -134,13 +136,19 @@ class thread
         }
         else
         {
-            NT_VERIFY(ObReferenceObjectByHandle(threadHandle,
-                                                THREAD_ALL_ACCESS,
-                                                *PsThreadType,
-                                                KernelMode,
-                                                reinterpret_cast<void**>(&m_Thread),
-                                                nullptr));
-            NT_VERIFY(ObCloseHandle(threadHandle, KernelMode));
+            NT_VERIFY(NT_SUCCESS(
+                ObReferenceObjectByHandle(threadHandle,
+                                          THREAD_ALL_ACCESS,
+                                          *PsThreadType,
+                                          KernelMode,
+                                          reinterpret_cast<void**>(&m_Thread),
+                                          nullptr)));
+            NT_VERIFY(NT_SUCCESS(ObCloseHandle(threadHandle, KernelMode)));
+
+            //
+            // Release the memory to the invoke routine.
+            //
+            params.release();
         }
     }
 

Diff for: include/jxy/thread.hpp

@@ -22,4 +22,14 @@ template <typename T,
           typename TAllocator = jxy::allocator<T, t_PoolType, t_PoolTag>> 
 using vector = std::vector<T, TAllocator>;
 
+template <typename T,
+          ULONG t_PoolTag,
+          typename TAllocator = jxy::allocator<T, PagedPool, t_PoolTag>>
+using paged_vector = std::vector<T, TAllocator>;
+
+template <typename T,
+          ULONG t_PoolTag,
+          typename TAllocator = jxy::allocator<T, NonPagedPoolNx, t_PoolTag>>
+using non_paged_vector = std::vector<T, TAllocator>;
+
 }

Diff for: include/jxy/vector.hpp

@@ -144,7 +144,7 @@
     <ClInclude Include="..\include\jxy\queue.hpp" />
     <ClInclude Include="..\include\jxy\scope.hpp" />
     <ClInclude Include="..\include\jxy\set.hpp" />
-    <ClInclude Include="..\include\jxy\stach.hpp" />
+    <ClInclude Include="..\include\jxy\stack.hpp" />
     <ClInclude Include="..\include\jxy\string.hpp" />
     <ClInclude Include="..\include\jxy\thread.hpp" />
     <ClInclude Include="..\include\jxy\unordered_map.hpp" />

Diff for: jxystl/jxystl.vcxproj

@@ -18,7 +18,7 @@
     <ClInclude Include="..\include\jxy\list.hpp" />
     <ClInclude Include="..\include\jxy\deque.hpp" />
     <ClInclude Include="..\include\jxy\set.hpp" />
-    <ClInclude Include="..\include\jxy\stach.hpp" />
+    <ClInclude Include="..\include\jxy\stack.hpp" />
     <ClInclude Include="..\include\jxy\queue.hpp" />
     <ClInclude Include="..\include\jxy\unordered_map.hpp" />
     <ClInclude Include="..\include\jxy\unordered_set.hpp" />

Diff for: jxystl/jxystl.vcxproj.filters

@@ -48,3 +48,8 @@ void jxy::shared_mutex::unlock_shared() noexcept
 {
     FltReleasePushLockEx(&m_PushLock, 0);
 }
+
+jxy::shared_mutex::native_handle_type jxy::shared_mutex::native_handle() noexcept
+{
+    return &m_PushLock;
+}

Diff for: jxystl/locks.cpp

@@ -12,7 +12,7 @@
 // This is done intentionally as to not introduce a path which may be unsafe
 // for the kernel.
 //
-#include <wdm.h>
+#include <fltKernel.h>
 #include <stdexcept>
 #include <system_error>
 #include <intrin.h>
@@ -56,7 +56,6 @@ void __cdecl _Throw_Cpp_error(int Code)
     // FIXME: This should throw a std::system_error but it comes with global
     // allocator requirements. For now we'll throw a faked "system error" as
     // a runtime error. 
-    // Maybe need a non-standard jxy::error_code and jxy::system_error?
     //
     UNREFERENCED_PARAMETER(Code);
     throw std::runtime_error("jxy: Faked system error!");

Diff for: jxystl/msvcfill.cpp

@@ -7,6 +7,11 @@
 //
 #include <jxy/thread.hpp>
 
+jxy::thread::~thread()
+{
+    NT_ASSERT(m_Thread == nullptr);
+}
+
 jxy::thread::thread(thread&& Other) noexcept
     : m_Thread(std::exchange(Other.m_Thread, nullptr))
 {
@@ -53,6 +58,8 @@ void jxy::thread::join()
     {
         std::_Throw_Cpp_error(std::_NO_SUCH_PROCESS);
     }
+
+    ObDereferenceObject(m_Thread);
     m_Thread = nullptr;
 }
 
@@ -63,6 +70,7 @@ void jxy::thread::detach()
         std::_Throw_Cpp_error(std::_INVALID_ARGUMENT);
     }
 
+    ObDereferenceObject(m_Thread);
     m_Thread = nullptr;
 }
 

Diff for: jxystl/thread.cpp

@@ -10,6 +10,11 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "stlkrn", "stlkrn\stlkrn.vcx
 EndProject
 Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "jxystl", "jxystl\jxystl.vcxproj", "{D7E96B42-E040-46B7-A15C-29F0CC9DD469}"
 EndProject
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "stltest", "stltest\stltest.vcxproj", "{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}"
+	ProjectSection(ProjectDependencies) = postProject
+		{D7E96B42-E040-46B7-A15C-29F0CC9DD469} = {D7E96B42-E040-46B7-A15C-29F0CC9DD469}
+	EndProjectSection
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|x64 = Debug|x64
@@ -42,6 +47,18 @@ Global
 		{D7E96B42-E040-46B7-A15C-29F0CC9DD469}.Release|x86.ActiveCfg = Release|Win32
 		{D7E96B42-E040-46B7-A15C-29F0CC9DD469}.Release|x86.Build.0 = Release|Win32
 		{D7E96B42-E040-46B7-A15C-29F0CC9DD469}.Release|x86.Deploy.0 = Release|Win32
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Debug|x64.ActiveCfg = Debug|x64
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Debug|x64.Build.0 = Debug|x64
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Debug|x64.Deploy.0 = Debug|x64
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Debug|x86.ActiveCfg = Debug|Win32
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Debug|x86.Build.0 = Debug|Win32
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Debug|x86.Deploy.0 = Debug|Win32
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Release|x64.ActiveCfg = Release|x64
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Release|x64.Build.0 = Release|x64
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Release|x64.Deploy.0 = Release|x64
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Release|x86.ActiveCfg = Release|Win32
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Release|x86.Build.0 = Release|Win32
+		{19BE23CA-600E-4B26-A1A9-1E4878BF9AD1}.Release|x86.Deploy.0 = Release|Win32
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE