Merge pull request #604 from network-intelligence/dev

Dev -> trunk for version 2.10.0

Author: andrewchi

.github/workflows/build-wheels.yml

@@ -60,7 +60,7 @@ jobs:
           CIBW_TEST_COMMAND: cd {project} && python mercury_python_test.py
 
       - name: Upload to S3
-        env: 
+        env:
           AWS_ACCESS_KEY_ID: ${{secrets.ACCESS_KEY_ID}}
           AWS_SECRET_ACCESS_KEY: ${{secrets.ACCESS_KEY_SECRET}}
           AWS_DEFAULT_REGION: us-east-1

COPYING

@@ -66,7 +66,7 @@ License for src/rapidjson
 
 Tencent is pleased to support the open source community by making
 RapidJSON available.
- 
+
 Copyright (C) 2015 THL A29 Limited, a Tencent company, and Milo Yip.
 All rights reserved.
 
@@ -88,8 +88,8 @@ Other dependencies and licenses:
 Open Source Software Licensed Under the BSD License:
 --------------------------------------------------------------------
 
-The msinttypes r29 
-Copyright (c) 2006-2013 Alexander Chemeris 
+The msinttypes r29
+Copyright (c) 2006-2013 Alexander Chemeris
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
@@ -120,7 +120,7 @@ IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 Open Source Software Licensed Under the JSON License:
 --------------------------------------------------------------------
 
-json.org 
+json.org
 Copyright (c) 2002 JSON.org
 All Rights Reserved.
 

LICENSE

@@ -29,5 +29,3 @@ HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
 STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
 ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
 OF THE POSSIBILITY OF SUCH DAMAGE.
-
-

VERSION

@@ -1 +1 @@
-2.9.0
+2.10.0

doc/CHANGELOG.md

@@ -1,5 +1,16 @@
 # CHANGELOG for Mercury
 
+## VERSION 2.10.0
+* Added defensive code around memcpy operations in QUIC reassembly.
+* Exempted private IP addresses from Domain Faking check.
+* Enhanced protocol matcher and SMTP enhancements to support more commands.
+* Fixed compilation of experimental tool intercept.so.
+* Integrated STUN classifier into Mercury's analysis path.
+* Extended DHCP to report on all message types, not just responses.
+* Bugfix: removed user_agent reset code from do_observation struct.
+* STUN fingerprints are generated for client requests but not server responses.
+
+
 ## VERSION 2.9.0
 * Added a new configuration option, network-behavioral-detections,
   which enables behavioral detections without a resources file.

doc/autogen.md

@@ -7,12 +7,12 @@ To minimize the amount of C++ code that must be written in order to handle assig
 The goals of the auto-generation system are:
 
 * to minimize the amount of tedious and error-prone manual work involved in implementing protocols,
-  
+
 * to provide a uniform programmatic interface and JSON output syntax for different assigned numbers, and
-  
+
 * to have tooling that can be re-run in the future so that new assigned number values can be easily brought into the software
   once they are registered with IANA,
-  
+
 * to track assigned numbers that are not registered with IANA, and be able to combine this information with IANA data.
 
 We used the following approach.  Each assigned number is represented by a single C++ class, which is generated from one or more CSV files, using the `csv` utility.  A Makefile controls the downloading of CSV files from IANA files, and the running of `csv`, to create a single C++ header-only library that includes all of the assigned number classes for a particular protocol.  The `wget` utility is used for downloading files.  The resulting protocol library header file is written into the `src/tables/` subdirectory; it must be manually copied into the `src/libmerc` subdirectory to be used in mercury.  Any protocol library used in mercury must be copied into that directory and committed into the mercury git repo, so that it doesn’t need to be auto-generated for each build.
@@ -63,7 +63,7 @@ Value,Name,Reference
 0x03-0xff,Unassigned,
 ```
 
-As with the IANA conventions, the cells in the first line describes the data in their columns.  The last cell in the other lines are empty, which is acceptable, although each line must have the same number of cells. 
+As with the IANA conventions, the cells in the first line describes the data in their columns.  The last cell in the other lines are empty, which is acceptable, although each line must have the same number of cells.
 
 Let's run `csv` on that file, from the `src/tables` directory, and write the output into `example.h`:
 
@@ -133,11 +133,11 @@ As implemented, the `csv` utility provides minimal error reporting.  Users deser
 
 When working with additional protocols, it may be necessary to tweak the code to handle slightly different CSV file conventions.
 
-One important limitation of the current system is that ranges (such as 0x03-0xff) are not handled.  A future version could add provide the information about what range a number is in through the `get_name()`member function, or a new member function. 
+One important limitation of the current system is that ranges (such as 0x03-0xff) are not handled.  A future version could add provide the information about what range a number is in through the `get_name()`member function, or a new member function.
 
-Additional information about a number could be provided through additional member functions.  The RFC numbers associated with most IANA registrations could be used to determine the year that a protocol option was standardized, and this information could be provided through a member function.   The range of years associated with the assigned numbers in a protocol message give a strong indication of the year that the implementation was completed.   Another useful bit of information is the RFC reference itself, which could be provided as a URL to facilitate quick lookups.  
+Additional information about a number could be provided through additional member functions.  The RFC numbers associated with most IANA registrations could be used to determine the year that a protocol option was standardized, and this information could be provided through a member function.   The range of years associated with the assigned numbers in a protocol message give a strong indication of the year that the implementation was completed.   Another useful bit of information is the RFC reference itself, which could be provided as a URL to facilitate quick lookups.
 
-There are well-known addresses that are managed by IANA, which could be handled as assigned numbers.  
+There are well-known addresses that are managed by IANA, which could be handled as assigned numbers.
 
 Well-known ports could also be handled as assigned numbers, though implementations often use nonstandard destination ports, and IANA registrations should be taken with a grain of salt.
 

doc/fdc.md

@@ -2,16 +2,16 @@
 
 
 
-This note documents the data formats used in the Mercury Fingerprint and Destination Context (FDC).  The FDC is encoded using the Concise Binary Object Representation ([CBOR](https://datatracker.ietf.org/doc/html/rfc8949)), an IETF standard data format that is extensible, consice, and trivially mappable to the common JavaScript Object Notation ([JSON](https://datatracker.ietf.org/doc/html/rfc8259)).  The formats are formally defined using the Concise Data Definition Language ([CDDL](https://datatracker.ietf.org/doc/html/rfc8610)), an IETF standard notational convention for unamiguously expressing CBOR and JSON data formats. 
+This note documents the data formats used in the Mercury Fingerprint and Destination Context (FDC).  The FDC is encoded using the Concise Binary Object Representation ([CBOR](https://datatracker.ietf.org/doc/html/rfc8949)), an IETF standard data format that is extensible, consice, and trivially mappable to the common JavaScript Object Notation ([JSON](https://datatracker.ietf.org/doc/html/rfc8259)).  The formats are formally defined using the Concise Data Definition Language ([CDDL](https://datatracker.ietf.org/doc/html/rfc8610)), an IETF standard notational convention for unamiguously expressing CBOR and JSON data formats.
 
 A Fingerprint and Destination Context (FDC) object contains a Network Protocol Fingerprint ([NPF](https://github.com/cisco/mercury/blob/main/doc/npf.md)) and other data features, all of which are metadata observed in a single network session.  An NPF fingerprint is a set of data features formed by selecting and normalizing some elements of a protocol message, so that they are correlated with the sending application or library implementation.  A fingerprint by itself sometimes uniquely identifies an application, but often does not.   In the latter case, the other data features are valuable for indentifying the sending application.
 
-- An NPF fingerprint in CBOR encoding, as defined in the [NPF CDDL specification](https://github.com/cisco/mercury/blob/main/doc/npf.cddl). 
+- An NPF fingerprint in CBOR encoding, as defined in the [NPF CDDL specification](https://github.com/cisco/mercury/blob/main/doc/npf.cddl).
 - The server name, which corresponds to the TLS or QUIC Server Name field or the HTTP Host field.
 - The destination IP address, as a string containing a textual representation.
 - The destination port number, as an unsigned integer less than 64,535.
 - The user agent as a string, which corresponds to the value of the User-Agent header for HTTP, the value of the SOFTWARE attribute for STUN, and the concatenation of the Protocol and Comment strings for SSH.
-- Optionally, an unsigned integer corresponding to the truncation code, which indicates whether reassembly was required in order to obtain a complete fingerprint, and whether or not the fingerprint was truncated due to a missing packet.  Its values are 
+- Optionally, an unsigned integer corresponding to the truncation code, which indicates whether reassembly was required in order to obtain a complete fingerprint, and whether or not the fingerprint was truncated due to a missing packet.  Its values are
   - none = 0,
   - reassembled = 1
   - truncated = 2
@@ -35,5 +35,4 @@ fdc = [
 ]
 ```
 
-​	
-
+​

doc/intercept.md

@@ -7,7 +7,7 @@ The interceptor library provides deep visibility into the communication of the p
 To build interceptor:
 
    1.  Check out the mercury using git.
-   
+
    2.  Install prerequisites as needed.  On Debian/Ubuntu:
 
      $ sudo apt install libssl-dev libnss3-dev libgnutls28-dev
@@ -52,14 +52,13 @@ The test program [test_intercept.sh](../test/test_intercept.sh) shows how the li
 
 Function interception is currently implemented for these libraries (listed with their Debian/Ubuntu package names):
 
-* openssl (libssl-dev)            
-* NSS (libnss3-dev)           
-* GNUtls (libgnutls28-dev)       
-  
+* openssl (libssl-dev)
+* NSS (libnss3-dev)
+* GNUtls (libgnutls28-dev)
+
 
 
 
 ## Disclaimer
 
 The interceptor library is experimental, and will continue to evolve.  Please do not use it in mission-critical environments.  Feedback is welcome; please send to [[email protected].]([email protected]).
-

doc/libmerc_config.md

@@ -51,7 +51,7 @@ analysis_context_get_process_info(c, &probable_process, &probability_score);
 bool probable_process_is_malware = 0;
 double probability_malware = 0.0;
 analysis_context_get_malware_info(analysis_ctx, &probable_process_is_malware, &probability_malware);
-        
+
 const struct os_information *os_info = NULL;
 size_t os_info_len = 0;
 analysis_context_get_os_info(analysis_ctx, &os_info, &os_info_len);

doc/npf.cddl

@@ -96,4 +96,3 @@ ipv6_flow_key = [
   dst_port: uint,
   protocol: uint
 ]
-