Skip to content

Commit a30779e

Browse files
mrutkowsmrutkows
authored
Trim() JSON document keys only under paths supplied using the '--from' flag (#63)
* support parsing the 'paths' flag from command line Signed-off-by: Matt Rutkowski <[email protected]> * Assure BOM hashing works with Trim changes Signed-off-by: Matt Rutkowski <[email protected]> * Use a custom JSON encoder to assure Trim() does not use HTML encoding this preserving fidelity Signed-off-by: Matt Rutkowski <[email protected]> * Move QueryRequest and QueryResult structs to common package Signed-off-by: Matt Rutkowski <[email protected]> * Move LicenseInfo struct to schema package as an abstract type Signed-off-by: Matt Rutkowski <[email protected]> * Prevent cmd package from accessing QueryRequest fields directly Signed-off-by: Matt Rutkowski <[email protected]> * Complete work to insulate QueryRequest and its parsing from the cmd package Signed-off-by: Matt Rutkowski <[email protected]> * Initial support for the Trim() --from flag Signed-off-by: Matt Rutkowski <[email protected]> * Improve temp file creation for Temp() tests Signed-off-by: Matt Rutkowski <[email protected]> * Improve temp file creation for Temp() tests Signed-off-by: Matt Rutkowski <[email protected]> * Improve query command to allow FROM to be used to select entire slices Signed-off-by: Matt Rutkowski <[email protected]> * Trim() should use shared encode JSON method from utils package Signed-off-by: Matt Rutkowski <[email protected]> * Remove unused query request flags Signed-off-by: Matt Rutkowski <[email protected]> * Streamline error handline in Query() internal methods Signed-off-by: Matt Rutkowski <[email protected]> * Restructure internal Query() methods to allow for reuse of JSON maps Signed-off-by: Matt Rutkowski <[email protected]> * Restructure internal Query() methods to allow for reuse of JSON maps Signed-off-by: Matt Rutkowski <[email protected]> * Add additional Query() tests for new logic that can return slices of FROM objects Signed-off-by: Matt Rutkowski <[email protected]> * Add additional Query() tests for new logic that can return slices of FROM objects Signed-off-by: Matt Rutkowski <[email protected]> * Add new README example for the improved Query() of slices Signed-off-by: Matt Rutkowski <[email protected]> * Add basic information on Trim command to README Signed-off-by: Matt Rutkowski <[email protected]> * Add flag and example or the Trim command to README Signed-off-by: Matt Rutkowski <[email protected]> * Add 2 more examples of Trim() command Signed-off-by: Matt Rutkowski <[email protected]> * Add additional tests to validate output bytes Signed-off-by: Matt Rutkowski <[email protected]> * Add additional buffer test and document command completion Signed-off-by: Matt Rutkowski <[email protected]> * Ensure all Go source files include a SPDX-License-Identifier Signed-off-by: Matt Rutkowski <[email protected]> --------- Signed-off-by: Matt Rutkowski <[email protected]>
1 parent a3b51ee commit a30779e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

78 files changed

+1887
-926
lines changed

‎Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ lint: format
103103
golint .
104104

105105
# install: go install github.com/golangci/golangci-lint/cmd/[email protected]
106-
ci-lint:
106+
golangci_lint:
107107
@echo "golangci-lint run"
108108
golangci-lint run
109109

‎README.md

Lines changed: 306 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ The utility has now grown to include a rich set of commands, listed below, that
88

99
## Command Overview
1010

11-
The utility supports the following commands:
11+
The utility supports the following BOM-related commands:
1212

1313
- **[license](#license)**
1414
- **[list](#license-list-subcommand)** produce listings or summarized reports of license data contained in a BOM along with license "usage policy" determinations using the policies declared in the `license.json` file.
@@ -27,6 +27,14 @@ The utility supports the following commands:
2727

2828
- **[vulnerability](#vulnerability)** produce filterable listings or summarized reports of vulnerabilities from BOM data (i.e., CycloneDX Vulnerability Exploitability eXchange (**VEX**)) data or independently stored CycloneDX Vulnerability Disclosure Report (**VDR**) data.
2929

30+
**Experimental commands**:
31+
32+
Feedback and helpful commits appreciated on the following commands which will be moved to non-experimental after two point releases:
33+
34+
- **[diff](#diff)** : Shows the delta between two similar BOM versions in
35+
36+
- **[trim](#trim)** provide the ability to remove JSON information, by field key and limited query syntax, from the input JSON BOM document.
37+
3038
---
3139

3240
## Index
@@ -45,7 +53,9 @@ The utility supports the following commands:
4553
- [`schema` command](#schema): list supported BOM formats, versions, variants
4654
- [`validate` command](#validate): BOM against declared or required schema
4755
- [`vulnerability` command](#vulnerability): lists vulnerability summary information included in the BOM or VEX
48-
- [`diff` command](#diff): *experimental*: shows the delta between two BOM versions
56+
- [`diff` command](#diff): *experimental*: shows the delta between two similar BOM versions
57+
- [`trim` command](#diff): *experimental*: remove specified fields from JSON BOM documents and output smaller BOMs that are appropriate sized for different use cases and analysis
58+
- [`completion` command](#completion): generates command-line completion scripts for the utility
4959
- [Design considerations](#design-considerations)
5060
- [Development](#development)
5161
- [Prerequisites](#prerequisites)
@@ -110,10 +120,9 @@ For convenience, links to each command's section are here:
110120
- [schema](#schema)
111121
- [vulnerability](#vulnerability)
112122
- [validate](#validate)
123+
- [completion](#completion)
113124
- [help](#help)
114125

115-
### General command information
116-
117126
#### Exit codes
118127

119128
All commands return a numeric exit code (i.e., a POSIX exit code) for use in automated processing where `0` indicates success and a non-zero value indicates failure of some kind designated by the number.
@@ -552,10 +561,18 @@ allow Artistic Artistic-2.0 Artistic License 2.0 true tru
552561

553562
This command allows you to perform SQL-like queries into JSON format SBOMs. Currently, the command recognizes the `--select` and `--from` as well as the `--where` filter.
554563

564+
#### Query flags
565+
566+
##### Query `--from` flag
567+
555568
The `--from` clause value is applied to the JSON document object model and can return either a singleton JSON object or an array of JSON objects as a result. This is determined by the last property value's type as declared in the schema.
556569

570+
##### Query `--select` flag
571+
557572
The `--select` clause is then applied to the `--from` result set to only return the specified properties (names and their values).
558573

574+
##### Query `--where` flag
575+
559576
If the result set is an array, the array entries can be reduced by applying the `--where` filter to ony return those entries whose specified field names match the supplied regular expression (regex).
560577

561578
**Note**: All `query` command results are returned as valid JSON documents. This includes a `null` value for empty result sets.
@@ -564,9 +581,11 @@ If the result set is an array, the array entries can be reduced by applying the
564581

565582
The `query` command only supports JSON output.
566583

584+
- `json` (default)
585+
567586
#### Query result sorting
568587

569-
The `query` command does not support output results.
588+
The `query` command does not support formatting of output results as JSON format is always returned.
570589

571590
#### Query examples
572591

@@ -650,6 +669,56 @@ In this example, the `--from` clause references the singleton JSON object `compo
650669
}
651670
```
652671

672+
##### Example: Return the JSON array of components
673+
674+
In this example, the `--from` filter will return the entire JSON components array.
675+
676+
```bash
677+
./sbom-utility query -i test/cyclonedx/cdx-1-4-mature-example-1.json --from components --quiet
678+
```
679+
680+
```json
681+
[
682+
{
683+
"bom-ref": "pkg:npm/[email protected]",
684+
"description": "Node.js Sampler package",
685+
"licenses": [
686+
{
687+
"license": {
688+
"id": "MIT"
689+
}
690+
}
691+
],
692+
"name": "sample",
693+
"purl": "pkg:npm/[email protected]",
694+
"type": "library",
695+
"version": "2.0.0"
696+
},
697+
{
698+
"bom-ref": "pkg:npm/[email protected]",
699+
"description": "Node.js body parsing middleware",
700+
"hashes": [
701+
{
702+
...
703+
}
704+
],
705+
"licenses": [
706+
{
707+
"license": {
708+
"id": "MIT"
709+
}
710+
}
711+
],
712+
"name": "body-parser",
713+
"purl": "pkg:npm/[email protected]",
714+
"type": "library",
715+
"version": "1.19.0"
716+
}
717+
]
718+
```
719+
720+
**Note**: The command for this example only used the `--from` flag and did not need to supply `--select '*'` as this us the default.
721+
653722
##### Example: Filter result entries with a specified value
654723

655724
In this example, the `--where` filter will be applied to a set of `properties` results to only include entries that match the specified regex.
@@ -807,6 +876,218 @@ For details see the "[Adding SBOM formats, schema versions and variants](#adding
807876

808877
If you wish to have the new schema *embedded in the executable*, simply add it to the project's `resources` subdirectory following the format and version-based directory structure.
809878

879+
### Trim
880+
881+
This command is able to "trim" one or more JSON keys (fields) from specified JSON BOM documents effectively "pruning" the JSON document. This functionality helps consumers of large-sized BOMs that need to analyze specific types of data in large BOMs in reducing the BOM data to just what is needed for their use cases or needs.
882+
883+
#### Trim supported output formats
884+
885+
This command is used to output, using the [`--output-file` flag](#output-flag), a "trimmed" BOM in JSON format.
886+
887+
- `json` (default)
888+
889+
#### Trim flags
890+
891+
Trim operates on a JSON BOM input file (see [`--input-file` flag](#input-flag)) and produces a trimmed JSON BOM output file using the following flags:
892+
893+
##### Trim `--keys` flag
894+
895+
A comma-separated list of JSON map keys. Similar to the [query command's `--select` flag](#query---select-flag) syntax.
896+
897+
##### Trim `--from` flag
898+
899+
A comma-separated list of JSON document paths using the same syntax as the [query command's `--from` flag](#query---from-flag).
900+
901+
#### Trim examples
902+
903+
The original BOM used for these examples can be found here:
904+
905+
- [test/trim/trim-cdx-1-5-sample-small-components-only.sbom.json](test/trim/trim-cdx-1-5-sample-small-components-only.sbom.json)
906+
907+
##### Example: Trim `properties` from entire JSON BOM
908+
909+
Validating the "juice shop" SBOM (CycloneDX 1.2) example provided in this repository.
910+
911+
```bash
912+
./sbom-utility trim -i ./sbom-utility trim -i test/trim/trim-cdx-1-5-sample-small-components-only.sbom.json --keys=properties
913+
```
914+
915+
Original BOM with properties:
916+
917+
```json
918+
{
919+
"bomFormat": "CycloneDX",
920+
"specVersion": "1.5",
921+
"version": 1,
922+
"serialNumber": "urn:uuid:1a2b3c4d-1234-abcd-9876-a3b4c5d6e7f9",
923+
"components": [
924+
{
925+
"type": "library",
926+
"bom-ref": "pkg:npm/[email protected]",
927+
"purl": "pkg:npm/[email protected]",
928+
"name": "sample",
929+
"version": "2.0.0",
930+
"description": "Node.js Sampler package",
931+
"properties": [
932+
{
933+
"name": "foo",
934+
"value": "bar"
935+
}
936+
]
937+
},
938+
{
939+
"type": "library",
940+
"bom-ref": "pkg:npm/[email protected]",
941+
"purl": "pkg:npm/[email protected]",
942+
"name": "body-parser",
943+
"version": "1.19.0",
944+
"description": "Node.js body parsing middleware",
945+
"hashes": [
946+
{
947+
"alg": "SHA-1",
948+
"content": "96b2709e57c9c4e09a6fd66a8fd979844f69f08a"
949+
}
950+
]
951+
}
952+
],
953+
"properties": [
954+
{
955+
"name": "abc",
956+
"value": "123"
957+
}
958+
]
959+
}
960+
```
961+
962+
Output BOM results without `properties``:
963+
964+
```json
965+
{
966+
"bomFormat": "CycloneDX",
967+
"specVersion": "1.5",
968+
"serialNumber": "urn:uuid:1a2b3c4d-1234-abcd-9876-a3b4c5d6e7f9",
969+
"version": 1,
970+
"components": [
971+
{
972+
"type": "library",
973+
"bom-ref": "pkg:npm/[email protected]",
974+
"name": "sample",
975+
"version": "2.0.0",
976+
"description": "Node.js Sampler package",
977+
"purl": "pkg:npm/[email protected]"
978+
},
979+
{
980+
"type": "library",
981+
"bom-ref": "pkg:npm/[email protected]",
982+
"name": "body-parser",
983+
"version": "1.19.0",
984+
"description": "Node.js body parsing middleware",
985+
"hashes": [
986+
{
987+
"alg": "SHA-1",
988+
"content": "96b2709e57c9c4e09a6fd66a8fd979844f69f08a"
989+
}
990+
],
991+
"purl": "pkg:npm/[email protected]"
992+
}
993+
]
994+
}
995+
```
996+
997+
##### Example: Trim `name` and `description` from entire JSON BOM
998+
999+
```bash
1000+
./sbom-utility trim -i test/trim/trim-cdx-1-5-sample-small-components-only.sbom.json --keys=name,description --quiet
1001+
```
1002+
1003+
Output BOM results without `name` or `description`:
1004+
1005+
```json
1006+
{
1007+
"bomFormat": "CycloneDX",
1008+
"specVersion": "1.5",
1009+
"serialNumber": "urn:uuid:1a2b3c4d-1234-abcd-9876-a3b4c5d6e7f9",
1010+
"version": 1,
1011+
"components": [
1012+
{
1013+
"type": "library",
1014+
"bom-ref": "pkg:npm/[email protected]",
1015+
"version": "2.0.0",
1016+
"purl": "pkg:npm/[email protected]",
1017+
"properties": [
1018+
{
1019+
"value": "bar"
1020+
}
1021+
]
1022+
},
1023+
{
1024+
"type": "library",
1025+
"bom-ref": "pkg:npm/[email protected]",
1026+
"version": "1.19.0",
1027+
"hashes": [
1028+
{
1029+
"alg": "SHA-1",
1030+
"content": "96b2709e57c9c4e09a6fd66a8fd979844f69f08a"
1031+
}
1032+
],
1033+
"purl": "pkg:npm/[email protected]"
1034+
}
1035+
],
1036+
"properties": [
1037+
{
1038+
"value": "123"
1039+
}
1040+
]
1041+
}
1042+
```
1043+
1044+
##### Example: Trim `properties` from only `components` path
1045+
1046+
```bash
1047+
./sbom-utility trim -i test/trim/trim-cdx-1-5-sample-small-components-only.sbom.json --keys=properties --from components --quiet
1048+
```
1049+
1050+
Output BOM results with `properties` removed from all `components`:
1051+
1052+
```json
1053+
{
1054+
"bomFormat": "CycloneDX",
1055+
"specVersion": "1.5",
1056+
"serialNumber": "urn:uuid:1a2b3c4d-1234-abcd-9876-a3b4c5d6e7f9",
1057+
"version": 1,
1058+
"components": [
1059+
{
1060+
"type": "library",
1061+
"bom-ref": "pkg:npm/[email protected]",
1062+
"name": "sample",
1063+
"version": "2.0.0",
1064+
"description": "Node.js Sampler package",
1065+
"purl": "pkg:npm/[email protected]"
1066+
},
1067+
{
1068+
"type": "library",
1069+
"bom-ref": "pkg:npm/[email protected]",
1070+
"name": "body-parser",
1071+
"version": "1.19.0",
1072+
"description": "Node.js body parsing middleware",
1073+
"hashes": [
1074+
{
1075+
"alg": "SHA-1",
1076+
"content": "96b2709e57c9c4e09a6fd66a8fd979844f69f08a"
1077+
}
1078+
],
1079+
"purl": "pkg:npm/[email protected]"
1080+
}
1081+
],
1082+
"properties": [
1083+
{
1084+
"name": "abc",
1085+
"value": "123"
1086+
}
1087+
]
1088+
}
1089+
```
1090+
8101091
---
8111092

8121093
### Validate
@@ -1165,7 +1446,7 @@ CVE-2020-25649 611 CVSSv31: 7.5 (high), CVSSv31: 8.2 (high), CVSS
11651446
11661447
### Diff
11671448
1168-
This *experimental* command will compare two BOMs and return the delta (or "diff") in JSON (diff-patch format) or text.
1449+
This *experimental* command will compare two *similar* BOMs and return the delta (or "diff") in JSON (diff-patch format) or text. This functionality is based upon code ancestral to that used to report file diffs between `git commit`s.
11691450
11701451
##### Notes
11711452
@@ -1217,6 +1498,25 @@ Use the `--format` flag on the to choose one of the supported output formats:
12171498
12181499
---
12191500
1501+
#### Completion
1502+
1503+
This command will generate command-line completion scripts, for the this utility, customized for various supported shells.
1504+
1505+
The completion command can be invoked as follows:
1506+
1507+
```bash
1508+
./sbom_utility completion [shell]
1509+
```
1510+
1511+
where valid values for `shell` are:
1512+
1513+
- bash
1514+
- fish
1515+
- powershell
1516+
- zsh
1517+
1518+
---
1519+
12201520
### Help
12211521
12221522
The utility supports the `help` command for the root command as well as any supported commands

0 commit comments

Comments
 (0)