Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 64 additions & 48 deletions docs/how-to-guides/testing/index.rst
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
.. _testing:

:octicon:`beaker` Test Management and Traceability
===================================================
==================================================

.. toctree::
.. toctree:: requirements
:maxdepth: 1
:hidden:

requirements

This guide demonstrates how to integrate test reports into your documentation with full traceability between test specifications, test cases, source code, and test results.
This guide demonstrates how to integrate test reports into your
documentation with full traceability between test specifications, test
cases, source code, and test results.

Overview
--------
Expand All @@ -22,12 +22,14 @@ A comprehensive testing approach includes:
* **Traceability**: Links between all these elements
* **Test Reports**: Results from test execution

By using Sphinx-Needs for test specifications, sphinx-codelinks for code tracing, and sphinx-test-reports for test results, we create a complete traceable testing ecosystem.
By using Sphinx-Needs for test specifications, sphinx-codelinks for
code tracing, and sphinx-test-reports for test results, we create a
complete traceable testing ecosystem.

Workflow
--------

.. mermaid::
.. mermaid::

graph LR
REQ[Requirements] --> TS[Test Specifications]
Expand All @@ -36,7 +38,7 @@ Workflow
TC --> RUN[Test Execution]
RUN --> REPORT[Test Reports]
REPORT --> DOC[Documentation]

style REQ fill:#FFB300
style TS fill:#A6BDD7
style TC fill:#A6BDD7
Expand All @@ -45,40 +47,44 @@ Workflow
style DOC fill:#F6768E

Step 1: Define Test Specifications
-----------------------------------
----------------------------------

Create test specifications using the ``test-spec`` directive to define what needs to be tested:
Create test specifications using the ``test-spec`` directive to define
what needs to be tested:

.. test-spec:: Factorial Function Test Specification
:id: TS_FACTORIAL
:status: open
:tags: factorial, math

Test the factorial function for:

* Negative numbers (should return 1)
* Zero (should return 1)
* Positive numbers (should return correct factorial)

The factorial function is critical for mathematical operations and must handle edge cases correctly.

The factorial function is critical for mathematical operations and
must handle edge cases correctly.

.. test-spec:: Prime Number Check Test Specification
:id: TS_PRIME
:status: open
:tags: prime, math

Test the IsPrime function for:

* Negative numbers (should return false)
* Trivial cases (0, 1, 2, 3)
* Positive numbers (both prime and composite)

Prime number detection is used in cryptographic operations and must be accurate.

Prime number detection is used in cryptographic operations and must be
accurate.

Step 2: Annotate Test Cases in Code
------------------------------------
-----------------------------------

Add sphinx-codelinks annotations to your test cases. The annotation format is e.g.:
Add sphinx-codelinks annotations to your test cases. The annotation
format is e.g.:

.. code-block:: cpp

Expand All @@ -94,20 +100,25 @@ Example from ``sample1_unittest.cpp``:
:lines: 76-88
:caption: Annotated test case with GTest properties

The annotation ``@Test negative factorial values, T_FACT_001, test`` creates a traceable link between the test code and the documentation.
The annotation ``@Test negative factorial values, T_FACT_001, test``
creates a traceable link between the test code and the documentation.

**GTest Properties** (``RecordProperty``):

* ``need_id``: Links the test execution result to the test case need (T_FACT_001)
* ``need_id``: Links the test execution result to the test case need
(T_FACT_001)
* ``requirement``: Links to the requirement being tested (REQ_MATH_001)
* ``test_spec``: Links to the test specification (TS_FACTORIAL)

These properties are included in the XML test report and enable automatic linking between test results and documentation needs.
These properties are included in the XML test report and enable
automatic linking between test results and documentation needs.

Step 3: Link Test Cases to Specifications
------------------------------------------
-----------------------------------------

Test cases are automatically discovered from source code using sphinx-codelinks. The test files contain ``@`` annotations that define test needs:
Test cases are automatically discovered from source code using
sphinx-codelinks. The test files contain ``@`` annotations that define
test needs:

.. code-block:: cpp

Expand All @@ -119,35 +130,40 @@ Test cases are automatically discovered from source code using sphinx-codelinks.
// ... test implementation
}

These annotations are parsed by sphinx-codelinks and automatically create test needs. Below are all test cases discovered from the test files:
These annotations are parsed by sphinx-codelinks and automatically
create test needs. Below are all test cases discovered from the test
files:

.. src-trace::
:project: eac-cpp
.. src-trace::
:project: x-as-code-cpp
:file: sample1_unittest.cpp

The test cases link to:

* **Test specifications** (e.g., ``TS_FACTORIAL``) via the ``:spec:`` field
* **Implementation** (e.g., ``IMPL_2``) via the ``:implements:`` field
* **Test specifications** (e.g., ``TS_FACTORIAL``) via the ``:spec:``
field
* **Implementation** (e.g., ``IMPL_2``) via the ``:implements:`` field
* **Requirements** (e.g., ``REQ_MATH_001``) via GTest properties

Step 4: Show Code Traceability
-------------------------------
------------------------------

Use sphinx-codelinks to display where test cases are implemented:

.. code-block:: rst

.. src-trace:: Test Case Implementation
:project: eac-cpp
:directory: src
.. src-trace:: Test Case Implementation
:project: x-as-code-cpp
:directory: src

This shows all annotated code locations, creating bidirectional links between documentation and source code.
This shows all annotated code locations, creating bidirectional links
between documentation and source code.

Step 5: Integrate Test Reports
-------------------------------
------------------------------

After running tests, integrate the test results using sphinx-test-reports:
After running tests, integrate the test results using
sphinx-test-reports:

.. code-block:: bash

Expand All @@ -158,13 +174,14 @@ After running tests, integrate the test results using sphinx-test-reports:
cd build
./eac_test --gtest_output=xml:test-results.xml

The test results XML file contains properties that link back to the test case needs, enabling complete traceability.
The test results XML file contains properties that link back to the
test case needs, enabling complete traceability.

Traceability Matrix
-------------------

Test Specifications to Test Cases
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

View which test cases implement each test specification:

Expand All @@ -174,7 +191,7 @@ View which test cases implement each test specification:
:style: table

Test Cases to Implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

View the connection between test cases and the code they test:

Expand All @@ -191,7 +208,7 @@ To generate code coverage reports with line-level detail:

# Run tests with coverage
./scripts/test_with_coverage.sh

# View coverage report
open src/build/coverage_html/index.html

Expand All @@ -214,20 +231,20 @@ Integrate testing into your CI/CD pipeline:
cd src
cmake -S . -B build
cmake --build build

- name: Run Tests
run: |
cd src/build
./eac_test --gtest_output=xml:test-results.xml

- name: Upload Test Results
uses: actions/upload-artifact@v4
with:
name: test-results
path: src/build/test-results.xml

Complete Traceability Flow
---------------------------
--------------------------

The complete flow from requirements to test results:

Expand All @@ -254,16 +271,15 @@ By combining:

You create a fully traceable testing ecosystem where:

✅ Every test links to its specification
✅ Every test links to the code it verifies
✅ Test results are automatically integrated
✅ Coverage is measured and reported
✅ Traceability is bidirectional and complete
✅ Every test links to its specification ✅ Every test links to the
code it verifies ✅ Test results are automatically integrated ✅
Coverage is measured and reported ✅ Traceability is bidirectional and
complete

Additional Resources
--------------------

* `GoogleTest Documentation <https://google.github.io/googletest/>`_
* `Sphinx-Test-Reports <https://sphinx-test-reports.readthedocs.io/>`_
* `Sphinx-Codelinks <https://codelinks.useblocks.com/>`_
* `Code Coverage with lcov <https://github.com/linux-test-project/lcov>`_
* `Code Coverage with lcov <https://github.com/linux-test-project/lcov>`_
68 changes: 39 additions & 29 deletions docs/how-to-guides/trace-code/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,76 +3,86 @@
:octicon:`link` Tracing Source Code
===================================

This guide explains how to use CodeLinks to trace source code files and lines to needs in your documentation.
This guide explains how to use CodeLinks to trace source code files
and lines to needs in your documentation.

**Step 1: Install CodeLinks**

Follow the installation instructions in the `CodeLinks documentation <https://codelinks.useblocks.com/basics/installation.html>`__.

**Step 2: Configure CodeLinks**

Create a file named ``src_trace.toml`` in your ``docs`` folder (next to ``conf.py``) with the following content:
Create a file named ``src_trace.toml`` in your ``docs`` folder (next
to ``conf.py``) with the following content:

.. literalinclude:: ../../src_trace.toml
:language: toml
:caption: src_trace.toml
:language: toml
:caption: src_trace.toml

**Step 3: Add the src-trace Directive**

In your documentation files, add the following directive to display traced source code links:
In your documentation files, add the following directive to display
traced source code links:

.. code-block:: rst

.. src-trace::
:project: eac-cpp
:directory: .
.. src-trace::
:project: x-as-code-cpp
:directory: .

**Step 4: Annotate Your Source Code**

Add comments to your source code files to link code lines to needs. For example, in ``src/main.cpp``:
Add comments to your source code files to link code lines to needs.
For example, in ``src/main.cpp``:

.. literalinclude:: ../../../src/main.cpp
:language: cpp
:caption: src/main.cpp
:language: cpp
:caption: src/main.cpp

**Step 5: View Traced Files and Lines**

After following the steps above, the documentation will show the traced files and lines for each need in the specified project (``eac-cpp``). Here we show only the implementation files (not test files):
After following the steps above, the documentation will show the
traced files and lines for each need in the specified project (``x-as-code-cpp``).
Here we show only the implementation files (not test files):

.. src-trace::
:project: eac-cpp
:file: main.cpp
.. src-trace::
:project: x-as-code-cpp
:file: main.cpp

.. src-trace::
:project: eac-cpp
:file: sample1.cpp
.. src-trace::
:project: x-as-code-cpp
:file: sample1.cpp

.. src-trace::
:project: eac-cpp
:file: sample2.cpp
.. src-trace::
:project: x-as-code-cpp
:file: sample2.cpp

**Note**: Test files are traced separately in the :ref:`testing guide <testing>` to avoid duplicate need IDs.
**Note**: Test files are traced separately in the :ref:`testing guide <testing>`
to avoid duplicate need IDs.

**Summary**

By following these steps, you can easily trace source code to requirements, improving traceability and documentation quality.
By following these steps, you can easily trace source code to
requirements, improving traceability and documentation quality.

Advanced configuration options and features are available in the `CodeLinks documentation <https://codelinks.useblocks.com/components/configuration.html>`__.

Linking Requirements to Code
-----------------------------
----------------------------

Finally you can link from/to the traced source code lines like this:

.. req:: Requirement linking to source code
:id: REQ_0815
:status: open

This is a requirement that links to a need that has traced source code lines.
This is a requirement that links to a need that has traced source code
lines.

.. note::

Current limitation: ``ubCode`` is not aware of this need id yet. This means that
the ``ubCode`` navigation inside Visual Studio Code will not work and jumping from
this ``rst`` file to the source code line will not work. This will be implemented
and supported in a future release.
Current limitation: ``ubCode`` is not aware of this need id yet. This
means that the ``ubCode`` navigation inside Visual Studio Code will
not work and jumping from this ``rst`` file to the source code line
will not work. This will be implemented and supported in a future
release.
Loading