|
| 1 | +# How to write a DefectDojo parser |
| 2 | + |
| 3 | +> All commands assume that you're located at the root of the django-DefectDojo cloned repo. |
| 4 | +
|
| 5 | +## Pre-requisites |
| 6 | +- You have forked https://github.com/DefectDojo/django-DefectDojo and cloned locally. |
| 7 | +- Checkout `dev` and make sure you're up to date with the latest changes. |
| 8 | +- It's advised that you create a dedicated branch for your development, such as `git checkout -b parser-name` yet that's up to you. |
| 9 | + |
| 10 | +It is probably easier to use the docker-compose stack (and benefit from the hot-reload capbility for uWSGI). |
| 11 | +Set up your environment to use the dev or ptvsd environment, such as: |
| 12 | + |
| 13 | +`$ docker/setEnv.sh dev` |
| 14 | +or |
| 15 | +`$ docker/setEnv.sh ptvsd` (allows to set breakpoints in uWSGI) |
| 16 | + |
| 17 | +Please have a look at [DOCKER.md](../DOCKER.md) for more details. |
| 18 | + |
| 19 | +### docker images |
| 20 | +You'd want to build your docker images locally, and eventually pass in your local user's `uid` to be able to write to the image (handy for database migration files). Assuming your user's `uid` is `1000`, then: |
| 21 | + |
| 22 | +`$ docker-compose build --build-arg uid=1000` |
| 23 | + |
| 24 | +## Which files do you need to modify? |
| 25 | + |
| 26 | +| File | Purpose |
| 27 | +|------- |-------- |
| 28 | +|`dojo/fixtures/test_type.json` | Django fixture for the type of scan. Take the next available integer if you intend to push upstream. If you're planning to use only in your own fork, you could jump ahead by 1000 to avoid any potential conflicts. |
| 29 | +|`dojo/templates/dojo/import_scan_results.html` | Add the scan to the array presented in the drop-down box |
| 30 | +|`dojo/tools/<parser_dir>/__init__.py` | Empty file for class initialization |
| 31 | +|`dojo/tools/<parser_dir>/parser.py` | The meat. This is where you write your actual parser |
| 32 | +|`dojo/unittests/scans/<parser_dir>/{many_vulns,no_vuln,one_vuln}.json` | Sample files containing meaningful data for unit tests. The minimal set. |
| 33 | +|`dojo/unittests/test_<parser_dir>_parser.py` | The unittest class, holding unit tests definitions |
| 34 | +|`dojo/tools/factory.py` | Import there your new parser class and add it to the long "if/else" statement |
| 35 | + |
| 36 | +## Things to pay attention to |
| 37 | + |
| 38 | +Parsers may have many fields, out of which many of them may be optional. |
| 39 | + |
| 40 | +Always make sure you include checks to avoid potential `KeyError` errors (e.g. field does not exist), for those fields you are not absolutely certain will always be in file that will get uploaded. These translate to 500 error, and do not look good. |
| 41 | + |
| 42 | +## Unit tests |
| 43 | + |
| 44 | +Each parser must have unit tests, at least to test for 0 vuln, 1 vuln and many vulns. You can take a look at how other parsers have them for starters. The more quality tests, the better. |
| 45 | + |
| 46 | +### Test database |
| 47 | +To test your unit tests locally, you first need to grant some rights. Get your MySQL root password from the docker-compose logs, login as root and issue the following commands: |
| 48 | + |
| 49 | +``` |
| 50 | +MYSQL> grant all privileges on test_defectdojo.* to defectdojo@'%'; |
| 51 | +MYSQL> flush privileges; |
| 52 | +``` |
| 53 | + |
| 54 | +### Run your tests |
| 55 | + |
| 56 | +This local command will launch the unit test for your new parser |
| 57 | + |
| 58 | +`$ docker-compose exec uwsgi bash -c 'python manage.py test dojo.unittests.<your_unittest_py_file>.<main_class_name> -v2'` |
| 59 | + |
| 60 | +Example for the blackduck hub parser: |
| 61 | + |
| 62 | +`$ docker-compose exec uwsgi bash -c 'python manage.py test dojo.unittests.test_blackduck_csv_parser.TestBlackduckHubParser -v2'` |
| 63 | + |
| 64 | +> If you want to run all unit tests, simply run `$ docker-compose exec uwsgi bash -c 'python manage.py test dojo.unittests -v2'` |
| 65 | +
|
| 66 | +## Other files that could be involved |
| 67 | + |
| 68 | +### Change to the model |
| 69 | +In the event where you'd have to change the model, e.g. to increase a database column size to accomodate a longer string of data to be saved |
| 70 | +* Change what you need in `dojo/models.py` |
| 71 | +* Create a new migration file in dojo/db_migrations by running and including as part of your PR |
| 72 | + |
| 73 | + `$ docker-compose exec uwsgi bash -c 'python manage.py makemigrations -v2'` |
| 74 | + |
| 75 | +### Accept a different type of file to upload |
| 76 | +If you want to be able to accept a new type of file for your parser, take a look at `dojo/forms.py` around line 436 (at the time of this writing) or locate the 2 places (for import and re-import) where you find the string `attrs={"accept":`. |
| 77 | + |
| 78 | +Formats currently accepted: .xml, .csv, .nessus, .json, .html, .js, .zip. |
| 79 | + |
| 80 | +### A need for more than just the parser.py |
| 81 | + |
| 82 | +Of course, nothing prevents you from having more files than the `parser.py` file. It's python :-) |
| 83 | + |
| 84 | +## Example PRs |
| 85 | + |
| 86 | +If you want to take a look at previous parsers that are now part of DefectDojo, take a look at https://github.com/DefectDojo/django-DefectDojo/pulls?q=is%3Apr+label%3A%22import+scans%22+ |
| 87 | + |
| 88 | +## Update the readthedocs documentation |
| 89 | + |
| 90 | +The DefectDojo official documentation lives in another repository, https://github.com/DefectDojo/documentation |
| 91 | + |
| 92 | +Please update the `docs/integration.rst` with the details of your new parser and create a PR in that repo. Reference the PR in the main DefectDojo repository to establish an automatic link between the two. |
0 commit comments