back2source: Create framework to support exclusion and exception for each technology

[pombredanne]

The outcome would be code and config files to selectively exclude certain common code from the analysis, such as path used in the Standard C library, or the standard library in Go

[pombredanne]

A proposed design is to extend the scancode configuration "ignored_patterns" with these elements:

• we ignore path patterns, but should privilege paths
• some paths are globally ignored (but then if that's the case, use the standard ignores)
• these paths are specific for a technology we analyze
• the paths could be source or binary side, and are specific to a pipeline, not a step nor an option

For instance, when I collect DWARF paths from an ELF, I may want to selectively ignore some of these paths that are for includes from the standard C library.

Therefore the structure for these would be:

- pipeline: map_deploy_to_develop
  - ignored_to_path_patterns:
    # ignored because these are standard shared libraries
    - "adasdasdad"
      
  - ignored_from_path_patterns:
    # ignored because these are standard shared libraries
    - "path/*/foo"

And these would be used specifically in each of the d2d mapping steps. These are glob patterns, not regex.

[AyanSinhaMahapatra]

This was completed through the PR:

• Add ecosystem specific inclusions or exclusions #1550

We now have a d2d_config module: https://github.com/aboutcode-org/scancode.io/blob/main/scanpipe/pipes/d2d_config.py#L69 which has the ecosystem/technology specific inclusions and exclusions so that these are handled automatically once the optional steps are selected for these.

Also added some related bugfixes:

• Fix d2d pipeline bugs from ecosytem configurations #1716