feat: add support for targetting specific workspaces

[MalickBurger]

fixes #1126

[jkowalleck]

some things require chen. see my remarks.

also, please add proper unit tests

[MalickBurger]

some things require chen. see my remarks.

also, please add proper unit tests

Happy to do the changes. I would think some integration tests would be required for this change as it is not really feasible to test it in isolation due to the logic effectively sitting on the npm ls side.

I agree with the requirement of testing. I spent some time initially before making the PR going through the existing unit and integration tests and it is not very clear to me how we should go about adding tests for new functionality. For example, I do not see tests for several of the CLI options (such as --short-PURLs).

Could you provide a bit of a guide regarding how we should lay out tests for new functionality going forward as well as a brief of how the existing integration tests work and how they are structured. I intend on making additional PRs in the future for other issues and features so this would be very beneficial for me (and I'm sure other community members).

[jkowalleck]

@MalickBurger, for adding additional integration tests,

1. you would add new setups here, if needed: https://github.com/CycloneDX/cyclonedx-node-npm/tree/main/demo
2. Then, add the appropriate npm ls ... run here: https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/.github/workflows/npm-ls_demo-results.yml
3. have the npm ls ... result data gathered via a CI (Github action/workflow).
   you can dispatch the action in your repo here: https://github.com/MalickBurger/cyclonedx-node-npm/actions by dispatching the workflow (you might need to enable workflows for your repo)
4. download the action artifacts and add them here: https://github.com/CycloneDX/cyclonedx-node-npm/tree/main/tests/_data/npm-ls_demo-results
5. remove unnecessary data via https://github.com/CycloneDX/cyclonedx-node-npm/tree/main/tests/_data/npm-ls_demo-results/_duplicates-cleaner (might require a back-merge or rebase to pull the scripts in)

since you are planning on adding conditional parameters in the npm ls call, you need to incorporate this in the existing testing processes somehow.

[MalickBurger]

@MalickBurger, for adding additional integration tests,

1. you would add new setups here, if needed: https://github.com/CycloneDX/cyclonedx-node-npm/tree/main/demo
2. Then, add the appropriate npm ls ... run here: https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/.github/workflows/npm-ls_demo-results.yml
3. have the npm ls ... result data gathered via a CI (Github action/workflow).
   you can dispatch the action in your repo here: https://github.com/MalickBurger/cyclonedx-node-npm/actions by dispatching the workflow (you might need to enable workflows for your repo)
4. download the action artifacts and add them here: https://github.com/CycloneDX/cyclonedx-node-npm/tree/main/tests/_data/npm-ls_demo-results
5. remove unnecessary data via https://github.com/CycloneDX/cyclonedx-node-npm/tree/main/tests/_data/npm-ls_demo-results/_duplicates-cleaner (might require a back-merge or rebase to pull the scripts in)

since you are planning on adding conditional parameters in the npm ls call, you need to incorporate this in the existing testing processes somehow.

Looking at this, could you provide some testing support for an existing argument that is passed to npm ls by the plugin which I could then base my additional tests on? It would be easier and cleaner for a code maintainer to set these up so there is a standard to follow going forward for community members.

[jkowalleck]

@MalickBurger, for adding additional integration tests,

1. you would add new setups here, if needed: https://github.com/CycloneDX/cyclonedx-node-npm/tree/main/demo
2. Then, add the appropriate npm ls ... run here: https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/.github/workflows/npm-ls_demo-results.yml
3. have the npm ls ... result data gathered via a CI (Github action/workflow).
   you can dispatch the action in your repo here: https://github.com/MalickBurger/cyclonedx-node-npm/actions by dispatching the workflow (you might need to enable workflows for your repo)
4. download the action artifacts and add them here: https://github.com/CycloneDX/cyclonedx-node-npm/tree/main/tests/_data/npm-ls_demo-results
5. remove unnecessary data via https://github.com/CycloneDX/cyclonedx-node-npm/tree/main/tests/_data/npm-ls_demo-results/_duplicates-cleaner (might require a back-merge or rebase to pull the scripts in)

since you are planning on adding conditional parameters in the npm ls call, you need to incorporate this in the existing testing processes somehow.

Looking at this, could you provide some testing support for an existing argument that is passed to npm ls by the plugin which I could then base my additional tests on? It would be easier and cleaner for a code maintainer to set these up so there is a standard to follow going forward for community members.

will do. just bare with me, it may take a while

[jkowalleck]

will do. just bare with me, it may take a while

I am currently working on a solution. Will ping as soon as I have something for you.

[jkowalleck]

@MalickBurger ,

I've overhauled the integration tests.
It would currently be enough to add your tests, showcasing that the intended parameters are sent to npm-ls. This could be done here: https://github.com/CycloneDX/cyclonedx-node-npm/blob/main/tests/integration/cli.args-pass-through.test.js

you might add something like

// region workspace
['workspace not supported npm 6', `6.${rMinor}.${rPatch}`, ['--workspace my-wsA', '--workspace my-wsB'], [...npm6ArgsGeneral]],
['workspace npm 7', `7.${rMinor}.${rPatch}`, ['--workspace my-wsA', '--workspace my-wsB'], [...npm7ArgsGeneral, '--workspace my-wsA', '--workspace my-wsB']],
['workspace npm 8', `8.${rMinor}.${rPatch}`, ['--workspace my-wsA', '--workspace my-wsB'], [...npm8ArgsGeneral, '--workspace my-wsA', '--workspace my-wsB']],
['workspace npm 9', `9.${rMinor}.${rPatch}`, ['--workspace my-wsA', '--workspace my-wsB'], [...npm9ArgsGeneral, '--workspace my-wsA', '--workspace my-wsB']],
['workspace npm 10', `10.${rMinor}.${rPatch}`, ['--workspace my-wsA', '--workspace my-wsB'], [...npm10ArgsGeneral, '--workspace my-wsA', '--workspace my-wsB']]
// endregion workspace

PS: i've also prepared demo data for additional integration tests. I might add them as soon as the basic pass-through tests are done

[jkowalleck]

@MalickBurger , i see you've rebased this feature branch recently.
Are planning on integrating the needed tests as described here ?

[MalickBurger]

@MalickBurger , i see you've rebased this feature branch recently. Are planning on integrating the needed tests as described here ?

Hi @jkowalleck, yes I have been busy recently but am getting back to this now. Will be posting updates soon.

[jkowalleck]

merged in the latest master, fixed some merge conflicts.
did another round of review. result: there are open questions and issues

[MalickBurger]

merged in the latest master, fixed some merge conflicts. did another round of review. result: there are open questions and issues

Will take a look and update over the next few days when I have a chance

[jkowalleck]

seams like some tests are failing.

[MalickBurger]

seams like some tests are failing.

Should be resolved now.

[jkowalleck]

i am considering this feature to be integrated and marked as "experimental".
this means all related CLI switches are disabled per default, and are remarked as experimental.

[MalickBurger]

Hi @jkowalleck just want to find out where we stand regarding this PR. From my side I believe it is ready to merge.

[jkowalleck]

something seams not right

[jkowalleck]

i think some parts need discussion

[MalickBurger]

I think we are conflicting slightly on how this should be implemented because I am trying to design this in a way that is the most intuitive and yet still flexible experience for users. The aim is for it to be clear what the default behaviour is and how they can modify this behaviour if they wish.

The way you would like this done is to mimic npm ls exactly which I still don't fully understand since most users of the plugin wouldn't really care how the internals work they just want a tool that can create sboms for npm projects as easily as possible.

In an ideal world we could both have our way but I don't think it is the case here because the behaviour of npm ls with regards to workspaces is non intuitive and a bit of a black box (it also changes between version 7 and newer versions).

Before I go back and rewrite the logic again let's please fully decide how this is going to look, how we are going to explain it to users, and what default logic will be for all combination of workspace related parameters.

[jkowalleck]

The way you would like this done is to mimic npm ls exactly which I still don't fully understand since most users of the plugin wouldn't really care how the internals work they just want a tool that can create sboms for npm projects as easily as possible.

This PR and a reading npm docs and changelogs showed ma, that even npm developer had no clear vision how things should be working. the idea of workspaces might look simple at first, but it has so many "use cases", more than they imagined at first.

I had the idea to mimic npm, because i thought users knew how their npm worked, but were just not sure how to generate SBOM from the data they had.
If we mimicked npm, then we would have not to take care of all the strange edge cases every other npm version introduced - we would lay this into the hand of the users.

My idea was to go the "easy" path - from a product owner's perspective - as all responsibility would be shifted to the users. Especially with workspaces, I would expect to have experienced users who knew what they do.
Our discussions showed me, that my idea was bad from a user's perspective. They now would need to learn about their own npm and its edge cases. Now they would need to understand aspects they maybe did not even care about. This could be considered as bad user experience.

I think I understand your point, and you convinced me.
If your idea was to produce consistent results over all supported npm versions, then the current tests might be insufficient. For this reason, I would say:
implement it the way you envisioned it. 👍, and if needed, get rid of current tests

After the implementation is done, we might see which kind of test data is required and how it can be provided.
Since the feature is experimental, i'd suggest adding test data in a separate pullrequest later.

[mapau]

@MalickBurger and @jkowalleck what is the status here? I would like to use the cyclonedx-node-npm with some projects that uses npm workspaces. Let me know if I can help here in any way. I will try, the branch build on my projects for now.

[MalickBurger]

@MalickBurger and @jkowalleck what is the status here? I would like to use the cyclonedx-node-npm with some projects that uses npm workspaces. Let me know if I can help here in any way. I will try, the branch build on my projects for now.

Hi @mapau, I will be getting back to this later in the week, we are close (hopefully) to having this PR finalized. Happy if you would consider doing some testing once complete?

[MalickBurger]

Hi @jkowalleck,

I have refactored my changes based on our last discussion. I have simplified the logic and have tried to make it as simple and intuitive as possible to users while still mimicking npm ls where possible. I have designed it in such a way that users do not need to have an indepth understanding of npm ls or knowledge on which version they are running.

I have updated the tests but I need to do a few manual sanity tests as well however I will wait for now incase I need to make any additional changes.

[jkowalleck]

i have only implementation-related remarks.

the UX looks good.
what do you think, @mapau?

[mapau]

@jkowalleck I used the branch build on my projects that used npm workspace. All works as expected. The CLI help and info is clear and I would use this version as it is.

Only small thing, on my zsh the help output from the --no-workspaces section is not aligned with the other, see screenshot.

[MalickBurger]

@jkowalleck I used the branch build on my projects that used npm workspace. All works as expected. The CLI help and info is clear and I would use this version as it is.

Only small thing, on my zsh the help output from the --no-workspaces section is not aligned with the other, see screenshot.

Thanks for testing it out and providing feedback. I have made a change for the formatting issue and it is fixed.

[jkowalleck]

love the result. will merge soon.

Thank you very much for your persistence and championing this feature, @MalickBurger.
feel free to add yourself to the list of contributors :

cyclonedx-node-npm/package.json

Line 39 in 15da89b

"contributors": [

[MalickBurger]

love the result. will merge soon.

Thank you very much for your persistence and championing this feature, @MalickBurger. feel free to add yourself to the list of contributors :

cyclonedx-node-npm/package.json

Line 39 in 15da89b

"contributors": [

#1269