Recommended mechanisms for loading builds

Background

Klocwork uses an auto-matching algorithm when loading builds to determine which files have changed or moved when compared to previous builds. This is the foundation for matching issues between builds and has a big impact on the accuracy of results, build over build.

Important: To mitigate issue matching issues and improve accuracy, we recommend that you use the following mechanisms from day one of complex projects or projects that require a high level of consistency.

Use replace path

If files are incorrectly matched, issue data can get into inconsistent states and citing data may be lost or orphaned. Replace path should be used on all integration and CI build hosts to ensure that the source roots are consistent across all builds.

Use the replace path feature for any path that could change between systems, builds, and/or machines. Any of these can cause paths to source, libraries, and external tools to change, which can impede the auto-matching algorithm.

For example, if you have source folders, external libraries, or system files that have varying paths based on the environment, you should use the --replace-path option during kwbuildproject to automatically adjust these paths. This helps ensure that your project can be seamlessly built and analyzed across different setups.

As of Klocwork 2023.3, the replace path feature has been optimized to work in all scenarios.

Replace path examples

Replace path is especially useful when a project has multiple platforms, as there can never be an exact match between a Windows and a Linux path. It should also be used to replace network paths whenever possible.

Consider the following systems, with no match in base paths:

System 1 (Linux stream 1)

/path/to/our/source/code
/home/james/.maven
/ubuntu/libs
//192.168.0.1/network/path

System 2 (Linux stream 2)

/custom/made/source/code
/users/jeffry/.maven
/arch/libs
//127.0.0.1/network/path

System 3 (a Windows stream)

C:\workspace\source\code
C:\users\jeffry\.maven
C:\system32\libs
\\10.151.0.23\network\path

Even if you have identical files inside the paths, there will need to be a best effort match in order to align files. The most common reason for this type of system disparity is using streams where multiple platforms are being loaded to the same project.

You can solve the system disparity by using the replace path option during analysis, giving a much more consistent result and should be done for both machines in a similar manner.

kwbuildproject -f -H <license_server_url> -o tables kwinject.out \

                                     --replace-path /path/to/our/source/code=/source \
                                     --replace-path /home/james/.maven=/maven \
                                     --replace-path /ubuntu/libs=/system \
                                     --replace-path //192.168.0.1/network/path=/buildserver

Enable exact file matching

For important or critical files, consider using enable_exact_file_match on a project. This will disable auto-matching and give you more control over your project.

We have added support for .emp files that can be loaded (and edited) onto a project. When loaded, an .emp file will apply exact match to files in the list of folders specified within it.

The workflow is as follows:

1. Enable enable_exact_file match using kwadmin or validate admin:

   kwadmin set-project-property <project_name> enable_exact_file_match TRUE

2. Import an .emp file via the project configuration page on the Portal, or by using the command:

   kwadmin import-config <project_name> file.emp

   You can also modify an existing .emp file via the project configuration page on the Portal.

If the .emp file is empty or does not exist and enable_exact_file_match is set to true, the exact match will apply to all files.

Important: Enable exact match should only be used in conjunction with file matching overrides for effective results. See next section for details and example.

Use file matching overrides file

From Klocwork 2023.3, you can use the file matching overrides file to assist in matching files correctly.

File matching overrides provide a way to manually specify how files should be matched and handled. This is particularly useful when dealing with version control systems like Git or Helix Core.

Overrides also allow handling of scenarios that are currently not possible using auto-matching, such as

• renaming files (as opposed to moving files)

• marking an old file as deleted so it will not match any file

• marking a file as new to avoid matching it to an unrelated deleted file

A file overrides file is a simple text file with three operators: A for Added, D for Deleted, and R for Renamed. We recommend applying an overrides file between each build to prevent mismatches.

Tip: You can use tools such as git diff to generate and populate your overrides file. Refer to this sample script as a template for creating your own extraction script.

File matching overrides examples

Example 1

• A file named “/root/src/data/test.cpp” exists in build_1.

• It is later deleted in build_2.

• A new file named “/external/libraries/json/test.cpp” is added in build_3.

If auto-match is enabled and no override file is given, “/root/src/data/test.cpp” will be incorrectly matched to “/external/libraries/json/test.cpp”

Solution 1

We can mark the file deleted in build 2 to ensure it is not matched to any new files in build 2. We can also mark the new test.cpp as new (added) in build 3 and does not match any previous files.

1. Prior to loading build_2, prepare an overrides file (overrides.txt) with the content:

   D /root/src/data/test.cpp

2. When loading build_2, include the --file-overrides option:

   kwadmin load project tables --file-overrides overrides.txt

3. Prior to loading build_3, prepare a new overrides file, or edit the existing one, with the content:

   A /external/libraries/json/test.cpp

4. When loading build_3, include the --file-overrides option:

   kwadmin load project tables --file-overrides overrides.txt

Solution 2

Since the root paths differ between the two test.cpp files, we can direct enable_exact_file_match to look for files only in the first root path (/root/src).

1. Enable exact_file_match:

   kwadmin set-project-property <project_name> enable_exact_file_match TRUE

2. Prepare an .emp file to look for exact matches in the first path (path_1.emp):

   /root/src

3. Add the exact match configuration file on the Portal under build Configuration > Configuration Files > +. Alternatively, use the command:

   kwadmin import-config <project_name> path_1.emp

This would allow for auto-matching to occur for the other path (/external/libraries), but not in /root/src.

Important: Enable exact match should only be used in conjunction with file matching overrides for effective results.

Example 2

• A file named “/root/src/data/test.cpp” exists in build_1.

• It is then renamed to “/root/src/data/testUtil.cpp” in build_2.

The auto-matching algorithm will not match these two files as they have different names.

Solution

1. Add a rename to the overrides file (overrides.txt):

   R /root/src/data/test.cpp /root/src/data/testUtil.cpp

2. Load the overrides file onto build_2:

   kwadmin load project tables --file-overrides overrides.txt

   This generates a build_3 with the file marked as renamed from "test.cpp" to "testUtil.cpp".

Use build tags

From Klocwork 2023.3, you can add metadata to builds in the form of tags. They can be used to associate things like platform, branch and/or commit ids with a build. They can then be used to make precise decisions when generating file matching overrides between builds.

For more information on build tags, see Using build tags.

Structure your streams logically

Avoid using a completely flat stream structure, unless it makes the most sense. Structuring your streams logically will:

• Enhance the clarity and maintainability of your project by grouping related files and directories together

• Simplify the management of your projects and streams, when it comes to creating, editing, and deleting streams

• Allow the file matching algorithm to make better decisions when matching files

Stream structuring examples

Compare the following bad and good stream structures.

Bad flat structure example

- demosthenes
  - linux_x64_main
  - linux_x86_main
  - linux_x86_feature_1
  ...
  - windows_x64_main
  ...

Good hierarchical structure example

- demosthenes
  - linux
    - x64
      - main
        - feature_1
        - feature_2
      - v23.2
        - hotfix_1
    - x86
    ...
  - windows
    ...