Deduplicator

Keep one finding when the same vulnerability is reported twice, by one scanner or by several.

Updated

The same vulnerability often arrives more than once: a scanner reports it again in every scan, and two scanners report the same leaked secret or vulnerable dependency. Portal removes such duplicates in two steps:

  • Basic deduplication runs on import and doesn’t create a finding that already exists.
  • Deduplicator rules run in the background and merge findings that basic deduplication can’t recognize as the same, typically from different scanners.

Basic deduplication

Open the settings, then Basic Deduplication under Maintenance. It applies to reports imported over the API and from Auditor.

  • Deduplication Scope: compare a new finding with the findings of the same product, of all products of the same product type, or of all of Portal.
  • Main Criteria: the fields that make a finding’s fingerprint, such as Same Title, Same File Path and Same Line. A new finding with the same main criteria as an existing one is not imported. Change them with care: too few, and different findings are dropped as duplicates.
  • Asset criteria and Additional Criteria: compare only with findings that also have the same repository, Docker image, branch, scanner and so on.
  • Reopen resolved duplicates: when a resolved finding is reported again, move it back to Unverified. Turn it on so that a fix that was rolled back shows again.

Run basic deduplication applies the settings to the findings already in Portal; Dry run shows how many duplicates it would delete. To run it for one product, use the Manual Deduplication tab of the product’s Options.

Deduplicator rules

Open Deduplicator in the sidebar. A rule has two sides and a set of criteria:

  • Original: the conditions for the finding to keep, for example findings of Gitleaks.
  • Duplicate: the conditions for the findings to remove, for example findings of Trufflehog.
  • Criteria: what must be equal for two findings to be duplicates, for example Same File path, Same Line number and Same Repository.

The conditions are the same as in Auto Validator rules. A finding on the duplicate side that matches a finding on the original side by all criteria is deleted. Only findings that aren’t resolved are compared.

Findings to be affected in the form shows which findings the rule would delete now. Check it before you save.

There are ready-made rules for the built-in scanners: click Import Rules and upload the rule set. When you add a scanner, add rules for it too.

Settings

The settings at the top of the Deduplicator page turn the rules on and off (Enable Deduplicator) and set how often they run (Launch frequency (seconds)).