1.0 Deployment and KB test results

1.1 Accessing and browsing the CC database TRP-001

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

1 = insignificant defect: (i) The links in the item preview are empty. E.g., see the preview of family FDP_ACC or of FAU_SAA. (ii) In the item preview of components (e.g. FDP_RIP.2, FDP_ACC.2), the text shown as value for the field "Hierarchical to" is tokenized by characters. Instead, these should be words.

Defect summary description

(i) The links in the item preview are empty. E.g., see the preview of family FDP_ACC or of FAU_SAA. (ii) In the item preview of components (e.g. FDP_RIP.2, FDP_ACC.2), the text shown as value for the field "Hierarchical to" is tokenized by characters. Instead, these should be words.

Text execution evidence

N/A

Comments

The acceptance criteria in the parent SRS is satisfied. The data is mostly aligned to the description in the corresponding CC document (e.g. of an exception: for ACE, the info in MS-Introduction does not appear in the PDF). The reported defects affect readability.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-001 Test accessing and browsing the CC Database

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	1 = insignificant defect
release	alpha
source_prefix	TRA
source_uid	TRA-001

1.2 Querying the CC database TRP-002

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

1 = insignificant defect: When an element from an upper category in the hierarchy is selected, the children should be cleared to avoid confusion regarding to which element the item preview corresponds. An example of elements from different clases and families mixed in one screen is provided as a reference in the displayed pathfile.

Defect summary description

When an element from an upper category in the hierarchy is selected, the children should be cleared to avoid confusion regarding to which element the item preview corresponds. An example of elements from different clases and families mixed in one screen is provided as a reference in the displayed pathfile.

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

The acceptance criteria is satisfied. Yet, the defect described could render the browser view confusing.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/tra-002-ev.png

Parent links: TCS-002 Test query the CC Database

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	1 = insignificant defect
release	alpha
source_prefix	TRA
source_uid	TRA-002

1.3 Exporting security components TRP-003

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

1 = insignificant defect: When files are exported, the links become unusable; for instance, when opened in VSCode, they open a browser pointing to a "page not found".

Defect summary description

When files are exported, the links become unusable; for instance, when opened in VSCode, they open a browser pointing to a "page not found".

Text execution evidence

N/A

Comments

(i) Only export to markdown files implemented in the Alpha phase. (ii) When giving a filepath that does not exist, the export does not create it. While this is ok, it would be better to display a message informing about the specific problem.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-003 Test exporting Security Components

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	1 = insignificant defect
release	alpha
source_prefix	TRA
source_uid	TRA-003

1.4 Tailoring security requirements TRP-004

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not applicable to the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-004 Test tailoring Security Requirements

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-004

1.5 Navigating the knowledge base TRP-005

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-005 Test navigating the Knowledge Base

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-005

1.6 Comprehensiveness of the knowledge base TRP-006

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-006 Test comprehensiveness of Knowledge Base

Attribute	Value
test-date	2023-11-28
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-006

1.7 Interconnection of the knowledge base TRP-007

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

2 = minor defect: Some pages are missing, e.g., "Terms and Definition Register". For others, the links are missing, e.g., [Security Functional Requirements] and [Security Assurance Requirements]

Defect summary description

Some pages are missing, e.g., "Terms and Definition Register". For others, the links are missing, e.g., [Security Functional Requirements] and [Security Assurance Requirements]

Text execution evidence

N/A

Comments

A link to the table of contents in each page would be useful.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-007 Test interconnection of Knowledge Base

Attribute	Value
test-date	2023-11-28
tester	IVS
defect-category	2 = minor defect
release	alpha
source_prefix	TRA
source_uid	TRA-007

1.8 Currency of the knowledge base TRP-008

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-008 Test currency of Knowledge Base

Attribute	Value
test-date	2023-11-28
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-008

1.9 Cosmetic features of the knowledge base TRP-009

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not applicable for the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-009 Test cosmetic features of Knowledge Base

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-009

1.10 CC database-DTD mapping inspection TRP-010

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Some of the verifications specified in the test steps are not observed in the code of cct.py. For instance, the attributes "part" and "patch" of the element "xref" in the DTD are not present in the XRef class.

Defect summary description

Some of the verifications specified in the test steps are not observed in the code of cct.py. For instance, the attributes "part" and "patch" of the element "xref" in the DTD are not present in the XRef class.

Text execution evidence

N/A

Comments

The discrepancies found do not necessarily imply an incorrect mapping, yet, to determine the correctness of the mapping by this method, a deeper inspection to track the parsing of the XML file to classes and its relation with the DTD would require more time and perhaps splitting the test case. Given that the mapping can be tested with other functional tests, TST-010 will be considered not applicable for now.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-010 Inspect CC Database-DTD mapping

Attribute	Value
test-date	2023-11-28
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-010

1.11 Correctness of the CC database TRP-011

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

2 = minor defect: The CLI application crashes if an invalid for a CC element is provided. This defect has been reported as bug #18 in the CAD GitLab repository.

Defect summary description

The CLI application crashes if an invalid for a CC element is provided. This defect has been reported as bug #18 in the CAD GitLab repository.

Text execution evidence

N/A

Comments

The correctness of the data for valid CC elements is already covered (and has been tested) in TST-001. The reported defect fails to inform the user about data not existing in the CC.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-011 Test correctness of CC Database

Attribute	Value
test-date	2023-11-28
tester	IVS
defect-category	2 = minor defect
release	alpha
source_prefix	TRA
source_uid	TRA-011

1.12 Running GUI from CLI with --gui option TRP-038

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-035 Run the GUI from the CLI with option --gui

Attribute	Value
test-date	2024-07-14
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-001

1.13 Running GUI from CLI with -g option TRP-039

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-036 Run the GUI from the CLI with option -g

Attribute	Value
test-date	2024-07-14
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-002

1.14 TCER: containerized deployment arm64 TRP-074

Relevant test environment and configuration details

Software deviations: aligned with test case specifications
Hardware deviations: aligned with test case specifications

Test execution results

Test case step 1: Clone repository

0 = flawless

Test case step 2: Open the C5-DEC repository in VS Code

0 = flawless

Test case step 3: Reopen in container with options

0 = flawless: the pop-up "Reopen in Container" appeared. When clicked, the two expected options (shown in the test case image) were presented in VS Code

Test case step 4: Build the CAD dev container

0 = flawless: when selecting the C5-DEC CAD dev container option, the image was created without errors after approx. 3-4 minutes

Test case step 5: Open a terminal and a shell

0 = flawless: a terminal and a poetry shell were manually created. The expected outcome is in alignment with the expected outcome shown in TCS-065. Furthermore, the c5dec TUI was opened when running c5dec, once inside the c5dec folder.

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-065 TC: Test containerized deployment on arm64

Attribute	Value
test-date	2025-05-02
tester	IVS
defect-category	0 = flawless
passed-steps	5
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-002

1.15 User manual coverage for beta release functionality TRP-040

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

2 = minor defect: The text needs more clarity with respect on how to perform export and ETR integration

Defect summary description

The text needs more clarity with respect on how to perform export and ETR integration

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-037 User manual contains guides for functionality added in Beta

Attribute	Value
test-date	2024-07-16
tester	IVS
defect-category	2 = minor defect
release	beta
source_prefix	TRB
source_uid	TRB-003

1.16 TCER: PQ cryptography container TRP-075

Relevant test environment and configuration details

Software deviations: aligned with test case specifications
Hardware deviations: N/A

Test execution results

Test case step 1: Clone the C5-DEC repository

0 = flawless

Test case step 2: Open the project in VS Code

0 = flawless

Test case step 3: Reopen in container with the option "C5-DEC cryptography dev container"

0 = flawless: the container is opened successfully

Test case step 4: Get a list of the available quantum-safe signature algorithms

0 = flawless: the output list corresponds to the expected outcome described in TCS-066

Test case step 5: Get a list of the available quantum-safe KEM algorithms

0 = flawless: the list of the available quantum-safe KEM algorithms corresponds to the expected outcome described in TCS-066

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

If we want to open the 'C5-DEC cryptography dev container' while already in VS Code and using the C5-DEC CAD container, the "Reopen in Container" option is not available. This happens more generally when trying to switch between the two C5-DEC containers.

However, this behavior is not related with C5-DEC but with the VS Code dev containers extension. The workaround is to "Reopen folder locally" for the "Reopen in Container" option to become available again.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-066 TC: Test PQ cryptography container

Attribute	Value
test-date	2025-05-02
tester	IVS
defect-category	0 = flawless
passed-steps	5
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-003

1.17 TCER: Test DocEngine TRP-076

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Open the C5-DEC CAD container in VS Code and activate the Python environment in the root folder

0 = flawless

Test case step 2: Go to the execution path of C5DEC CAD

0 = flawless

Test case step 3: Use quarto to render a PDF document

0 = flawless: the quarto command run successfully without showing error messages in the terminal; the PDF file was generated in the folder ~/c5dec/assets/report/_output/C5-DEC-CAD-DocEngine.pdf.

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See the file referred below in this test case report.

Comments

The final message in the terminal indicates Output created: _output/C5-DEC-CAD-DocEngine.pdf. While this path is correct w.r.t to the root of the quarto project, it is not correct w.r.t. to the execution path and it might be confusing. If the implementation effort is minimal, it would be better to show the complete path, i.e., ~/c5dec/assets/report/_output/C5-DEC-CAD-DocEngine.pdf.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/C5-DEC-CAD-DocEngine-evidence.pdf

Parent links: TCS-067 TC: Test DocEngine

Attribute	Value
test-date	2025-05-02
tester	IVS
defect-category	0 = flawless
passed-steps	3
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-004

1.18 TCER: containerized deployment x86-64 TRP-073

Relevant test environment and configuration details

Software deviations: aligned with test case specifications
Hardware deviations: aligned with test case specifications

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.

Test case step 1: Clone repository

0 = flawless

Test case step 2: Start Docker Desktop

0 = flawless: Docker Desktop was already running

Test case step 3: Open the C5-DEC repository in VS Code

0 = flawless

Test case step 4: Reopen in container with options

0 = flawless: the pop-up "Reopen in Container" appeared. When clicked, the two expected options (shown in the test case image) were presented in VS Code

Test case step 5: Build the CAD dev container

0 = flawless: the image was created without errors when selecting the C5-DEC CAD dev container option

Test case step 6: Open a terminal and a shell

0 = flawless: a terminal and a poetry shell were manually created. The expected outcome is in alignment with the expected outcome shown in TCS-064.

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-064 TC: Test containerized deployment on x86-64

Attribute	Value
test-date	2025-05-02
tester	IVS
defect-category	0 = flawless
passed-steps	6
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-001

1.19 TCER: DocEngine presentation template TRP-080

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Scaffold a new presentation template with c5dec docengine presentation -n my-presentation.

0 = flawless

Test case step 2: Verify the command completes without errors (exit code 0).

0 = flawless: the command exited with code 0 and no error messages were printed

Test case step 3: Confirm the directory ./docengine/my-presentation/ has been created.

0 = flawless: the directory was present at the expected path immediately after the command

Test case step 4: Inspect the directory contents and confirm _quarto.yml and index.qmd are present.

0 = flawless: both files were found inside ./docengine/my-presentation/

Test case step 5: Navigate to the template directory and run quarto render.

0 = flawless: quarto render was invoked from within docengine/my-presentation/ without errors

Test case step 6: Confirm that quarto render completes without errors.

0 = flawless: the render pipeline completed successfully with exit code 0

Test case step 7: Verify that an HTML output file is produced in the output directory.

0 = flawless: an HTML file was generated in the designated output location

Test case step 8: Open the HTML file in a browser and confirm it loads as a reveal.js slideshow with at least one slide visible.

0 = flawless: the HTML file opened correctly as a reveal.js presentation with slides rendered

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-077 TC: Test DocEngine presentation template

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	8
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-014

2.0 Foundations and data sources test results

2.1 Validity and consistency of bidirectional transformation TRP-012

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

The test steps are vague and do not provide enough details to perform this test.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-012 Test validity and conistency of bidirectional transformation

Attribute	Value
test-date	2023-11-28
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-012

2.2 Threats, risks, and countermeasures database TRP-013

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Although the parent SRS requires an integrated source of security-related data with content from BSI Grundschutz, ISO 27005, NIST SPs, and the CC, the SRS is not mandatory. Therefore, having data only from CC is acceptable.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-013 Test threats, risks, and countermeasures database

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-013

2.3 EUCC support TRP-014

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not applicable to the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-014 Test EUCC support

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-014

2.4 CSA Article 51 security objectives availability TRP-015

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not implemented for the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-015 Test availabilty of CSA Article 51 defined Security Objectives

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-015

2.5 Uniformity of storage mechanisms in CCT module TRP-016

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

The acceptance criteria are satisfied.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-016 Test uniformity of storage mechanisms in CCT module

Attribute	Value
test-date	2023-11-30
tester	Arash
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-016

2.6 GUI evaluation checklist creation for all CCv3.1R5 classes TRP-046

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-043 GUI: create spreadsheet eval checklist - all classes of CCv3.1R5

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-009

2.6.1 GUI all CCv3.1R5 classes checklist creation (re-test after fix) TRP-067

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: ''

Defect summary description

Text execution evidence

N/A

Comments

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-043 GUI: create spreadsheet eval checklist - all classes of CCv3.1R5

Attribute	Value
test-date	2024-07-18
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-030

2.7 GUI evaluation checklist creation for CCv2022R1 assurance class TRP-047

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-008i-ev.png docs/specs/trp/assets/trb-008ii-ev.png docs/specs/trp/assets/trb-008iii-ev.png

Parent links: TCS-044 GUI: create spreadsheet eval checklist - SAR class CCv2022R1

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-010

2.8 GUI evaluation checklist creation for all CCv2022R1 classes TRP-048

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-011iii-ev.png

Parent links: TCS-045 GUI: create spreadsheet eval checklist - all classes of CC2022R1

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-011

2.9 GUI link navigation TRP-049

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-046 Navigate using the links in the GUI

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-012

2.10 Browsing CC database in GUI TRP-041

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Adding a legend in the GUI to inform about the version of CC in use is recommended.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-038 Browse the CC database in the GUI

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-004

2.11 TCER: creation of empty C5-DEC-based project TRP-077

Relevant test environment and configuration details

Software deviations: aligned with test case specification

Test execution results

Test case step 1: Go to the c5dec root folder

0 = flawless

Test case step 2: Run the c5dec new command

0 = flawless: The command ran without errors, printing the following output log:

Starting C5-DEC CAD...
For usage instructions, run: ./c5dec.sh help
Executing C5-DEC CAD CLI command new...
Copied project folder to /home/alab/c5dec/myproject
Renamed devcontainer folder to /home/alab/c5dec/myproject/.devcontainer
Renamed template_devcontainer.json to /home/alab/c5dec/myproject/.devcontainer/devcontainer.json
Updated /home/alab/c5dec/myproject/.devcontainer/devcontainer.json with user=user
Updated /home/alab/c5dec/myproject/dev.Dockerfile with user=user
Updated /home/alab/c5dec/myproject/Dockerfile with user=user
Updated /home/alab/c5dec/myproject/build-c5dec.sh with username user and project name myproject
Renamed script /home/alab/c5dec/myproject/build-c5dec.sh to /home/alab/c5dec/myproject/build-myproject.sh
Updated /home/alab/c5dec/myproject/c5dec.sh with username user and project name myproject
Renamed script /home/alab/c5dec/myproject/c5dec.sh to /home/alab/c5dec/myproject/myproject.sh
Updated /home/alab/c5dec/myproject/pyproject.toml with username user and project name myproject
Renamed script /home/alab/c5dec/myproject/pyproject.toml to /home/alab/c5dec/myproject/pyproject.toml
Renamed /home/alab/c5dec/myproject/docs/specs/trp/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/trp/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/trp/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/trp/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/srs/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/srs/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/swd/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/swd/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/trp/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/trp/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/tcs/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/tcs/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/tcs/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/tcs/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/mrs/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/mrs/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/arc/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/arc/.doorstop.yml
Deleted the copied project folder /home/alab/c5dec/myproject
Created ZIP archive at /home/alab/c5dec/myproject.zip

Test case step 3: Verify creation of a zipped project file

0 = flawless: the ZIP file myproject.zip was in the list output by ls

Test case step 4: Unzip and access the project folder

0 = flawless: after unzipping the file, the structure of the internal folders was aligned with the one shown in the "Expected outcome" of the test case specification.

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See log in test case step 2 above.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-068 TC: Test the creation of an empty C5-DEC-based project

Attribute	Value
test-date	2025-05-06
tester	IVS
defect-category	0 = flawless
passed-steps	4
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T, I
release	stable
source_prefix	TRS
source_uid	TRS-005

2.12 CC browser display of selected item content TRP-042

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

When selecting a CC item from a dropdown list, clearing the dropdown lists of the child CC items would be very helpful for navigation. This is just a reminder as the developers are aware of the issue.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-005i-ev.png docs/specs/trp/assets/trb-005ii-ev.png

Parent links: TCS-039 The CC browser displays the content of the selected items

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-005

2.13 TCER: build and run an empty created project TRP-078

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Go to the myproject folder

0 = flawless

Test case step 2: Run the build script

0 = flawless: the log showed a healthy built of the images myproject:v1.0 and myproject-dev:v1.0

Building myproject images...
[+] Building 3.8s (20/20) FINISHED                                                               docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                             0.0s
...
 => => writing image sha256:c8a9121fa2a7f1cdedb6cb06e68cfc0321681cf0c37148aefb026c18d8d61008                     0.0s
 => => naming to docker.io/library/myproject:v1.0                                                                0.0s
...

[+] Building 38.2s (24/29)                                                                       docker:desktop-linux
[+] Building 63.0s (30/30) FINISHED                                                              docker:desktop-linux
 => [internal] load build definition from dev.Dockerfile
...
=> => writing image sha256:cf96ad7b07c3327dd0c315ddb11fc3450cf73fd6516ad8615f0838aeba7c20ab                      0.0s
=> => naming to docker.io/library/myproject-dev:v1.0
...

Test case step 3: Verify that the images exist

0 = flawless: the images myproject-dev and myproject, each with tag v1.0 appear in the output of the executed command

REPOSITORY                               TAG       IMAGE ID       CREATED          SIZE
myproject-dev                            v1.0      cf96ad7b07c3   13 minutes ago   3.12GB
myproject                                v1.0      c8a9121fa2a7   14 minutes ago   1.86GB

Test case step 4: Run the help of myproject

0 = flawless: the command showed the following output:

~/myproject % ./myproject.sh help
Starting myproject...
For usage instructions, run: ./myproject.sh help
---
./myproject.sh
... to open the myproject CLI help menu
./myproject.sh session
... to start an interactive myproject session
./myproject.sh <command>
... to run a myproject CLI command
./myproject.sh <command> -h
... to get help for a myproject CLI command
---

Test case step 5: Run myproject in a container

0 = flawless: an interactive shell within the container was created for the default user:

~/myproject % ./myproject.sh session
Starting myproject...
For usage instructions, run: ./myproject.sh help
Launching an interactive myproject session...
user@f8e418686013:~/myproject/myproject$

Test case step 6: Dependencies correctly installed

0 = flawless: the version of doorstop corresponds with the expected outcome for step 6 in TCS-069.

Test case step 7: Exit the session

0 = flawless: the container shell was closed and the control was given back to the terminal shell

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See logs in the steps above.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-069 TC: Test build and run of new project via scripts

Attribute	Value
test-date	2025-05-06
tester	IVS
defect-category	0 = flawless
passed-steps	7
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-006

2.14 Copying content in markdown from CC browser TRP-043

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-040 Copy content in markdown from the CC browser text area

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-006

2.15 TCER: New project in VS Code TRP-079

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Open myproject in VS Code.

0 = flawless

Test case step 2: Select the "Reopen in Container" option or use the command palette to reopen in a container.

0 = flawless: the container environment was successfully initialized.

Test case step 3: Open a terminal in VS Code and run poetry shell

0 = flawless: the poetry shell command successfully activated the environment as shown in the expected outcome.

Test case step 4: Verify that dependencies are installed by checking the version of quarto.

0 = flawless: The quarto -V command returned the expected version 1.6.43. Moreover, the other suggested dependencies were verified successfully.

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-070 TC: Test new project in VS Code

Attribute	Value
test-date	2025-05-06
tester	IVS
defect-category	0 = flawless
passed-steps	4
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-007

2.16 Expanding item content with descendant information TRP-044

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-007a-ev.png docs/specs/trp/assets/trb-007b-ev.png

Parent links: TCS-041 Expand text including info of all descendents of selected item

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-007

2.17 GUI evaluation checklist creation for CCv3.1R5 assurance class TRP-045

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

Suggestion: add some text in the GUI to inform the path where the exported file is created

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-008i-ev.png docs/specs/trp/assets/trb-008ii-ev.png docs/specs/trp/assets/trb-008iii-ev.png

Parent links: TCS-042 GUI: create spreadsheet eval checklist - SAR class of CCv3.1R5

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-008

3.0 CRA, SBOM, CCT and PRM test results

3.1 Automated rationale and traceability matrix generation TRP-017

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not implemented for the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-017 Test automated rationale and traceability matrix generation

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-017

3.2 Verification of rationales and traceability matrices TRP-018

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not implemented for the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-018 Test verificaiton of rationales and traceability matrices

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-018

3.3 Automated consistency and completeness checks TRP-019

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not implemented for the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-019 Test automated consistency and completeness checks

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-019

3.4 Automated validation TRP-020

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not implemented for the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-020 Test automated validation

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-020

3.5 Aggregation of SARs and work units TRP-021

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

3 = major defect: The CLI crashes when running the checklist command with "tst" as evaluation checklist prefix (issue #19)"; The CLI crashes when running $ poetry run c5dec checklist <prefix> --edit <item-id> due to vim not being installed. When adding the --editor code option, it runs as expected

Defect summary description

The CLI crashes when running the checklist command with "tst" as evaluation checklist prefix (issue #19)"; The CLI crashes when running $ poetry run c5dec checklist <prefix> --edit <item-id> due to vim not being installed. When adding the --editor code option, it runs as expected

Text execution evidence

N/A

Comments

The defect reported here is associated with issue #19 in the GitLab CAD repository.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-021 Test aggregation of SARs and Work Units

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	3 = major defect
release	alpha
source_prefix	TRA
source_uid	TRA-021

3.6 API provision for threat import TRP-022

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

The associated requirement (SRS-033) has not been implemented for the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-022 Test API provision for threat import

Attribute	Value
test-date	2023-11-28
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-022

3.7 Transforming imported threats to CC-conformant format TRP-023

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

The corresponding SRS (SRS-034) has not been implemented for the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-023 Test transforming imported threats to CC-conformant format

Attribute	Value
test-date	2023-11-28
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-023

3.8 Automated creation of evaluation checklist TRP-024

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

2 = minor defect: The CLI crahses if an invalid component id is given. E.g.: $ poetry run c5dec checklist -c evChck-tst-024 --id aco

Defect summary description

The CLI crahses if an invalid component id is given. E.g.: $ poetry run c5dec checklist -c evChck-tst-024 --id aco

Text execution evidence

N/A

Comments

The TUI works as expected. The reported defect affects the CLI.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-024 Test automated creation of Evaluation Checklist

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	2 = minor defect
release	alpha
source_prefix	TRA
source_uid	TRA-024

3.9 Evaluation progress tracking TRP-025

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

4 = critical defect: (i) The TUI crashes when an evaluation evidence is edited and revisited (issue #20). (ii) The CLI crashes with the option -s if the input evaluation checklist does not exist (issue #22).

Defect summary description

(i) The TUI crashes when an evaluation evidence is edited and revisited (issue #20). (ii) The CLI crashes with the option -s if the input evaluation checklist does not exist (issue #22).

Text execution evidence

N/A

Comments

This defects correspond to issues #20 and #22 in the GitLab CAD repository.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-025 Test evaluation progress tracking

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	4 = critical defect
release	alpha
source_prefix	TRA
source_uid	TRA-025

3.10 Work unit-artifact linking TRP-026

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-026 Test Work Unit-Artifact linking

Attribute	Value
test-date	2023-11-27
tester	AAT
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-026

3.11 Automated generation of observation reports TRP-027

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not implemented for the Aplha version.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-027 Test automated generation of Observation Reports

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-027

3.12 Automated generation of evaluation technical report TRP-028

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not implemented for the Alpha version.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-028 Test automated generation of Evaluation Technical Report

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-028

3.13 Flagging failed work units and affected artifacts TRP-029

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not addressed in the Alpha phase.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-029 Test flagging failed Work Units and affected artifacts

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-029

3.14 Auditability of evaluation items TRP-030

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-030 Test auditability of Evaluation Items

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-030

3.15 Extended data model TRP-031

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Interpreted as c5dec capturing the CC concepts and showing them in a correct and coherent manner.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-031 Test extended data model

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-031

3.16 CC templates TRP-032

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not implemented for the aplha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-032 Test CC templates

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-032

3.17 Validation of hierarchies and dependencies of security component sets TRP-033

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-033 Test hierarchies and dependencies of Security Component sets

Attribute	Value
test-date	2023-11-29
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-033

3.18 Generation of impact analysis report TRP-034

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

Not implemented for the Alpha release.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-034 Test generation of Impact Analysis Report

Attribute	Value
test-date	2023-11-27
tester	IVS
defect-category	0 = flawless
release	alpha
source_prefix	TRA
source_uid	TRA-034

3.19 CLI evaluation checklist export for all CCv3.1R5 classes TRP-050

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

1 = insignificant defect: The file is correctly generated but the format of its name differs from the expected name format.

Defect summary description

The file is correctly generated but the format of its name differs from the expected name format.

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

This defect is associated to issue 46

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-013-ev.png

Parent links: TCS-047 CLI export spreadsheet eval form - all SAR classes CCv3.1R5

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	1 = insignificant defect
release	beta
source_prefix	TRB
source_uid	TRB-013

3.20 CLI evaluation checklist export for all CCv2022R1 classes TRP-051

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

1 = insignificant defect: The file is correctly generated but the format of its name differs from the expected name format.

Defect summary description

The file is correctly generated but the format of its name differs from the expected name format.

Text execution evidence

N/A

Comments

This defect is associated to issue 46

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-027-ev.png

Parent links: TCS-048 CLI export spreadsheet eval form - all SAR classes CC2022R1

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	1 = insignificant defect
release	beta
source_prefix	TRB
source_uid	TRB-014

3.21 CLI evaluation checklist export for selected CCv3.1R5 classes TRP-052

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-015-ev.png

Parent links: TCS-049 CLI export spreadsheet eval form - some SAR classes CCv3.1R5

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-015

3.22 CLI evaluation checklist export for selected CCv2022R1 classes TRP-053

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-016-ev.png

Parent links: TCS-050 CLI export spreadsheet eval form - some SAR classes CC2022R1

Attribute	Value
test-date	2024-07-16
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-016

3.23 Export command invalid input error messages TRP-054

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

3 = major defect: The CLI crashes ungracefully (step 1).

Defect summary description

The CLI crashes ungracefully (step 1).

Text execution evidence

N/A

Comments

This defect has been reported in issue 47

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-051 Export command: messages for invalid input

Attribute	Value
test-date	2024-07-15
tester	IVS
defect-category	3 = major defect
release	beta
source_prefix	TRB
source_uid	TRB-017

3.24 CLI evaluation checklist export for CCv3.1R5 components TRP-055

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-018i-ev.png docs/specs/trp/assets/trb-018ii-ev.png

Parent links: TCS-052 CLI: export spreadsheet eval form - set SAR comp. CCv3.1R5

Attribute	Value
test-date	2024-07-16
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-018

3.25 TCER: CRA checklist category filtering TRP-081

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Initialise a fresh temporary project at /tmp/cra-filter-test with c5dec new --prefix MRS.

0 = flawless: the directory was created and Doorstop root document initialised without errors

Test case step 2: Create a CRA checklist for class_i products with c5dec cra --create --category class_i --prefix CRAC_I.

0 = flawless

Test case step 3: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 4: Count the items in the resulting CRAC_I Doorstop document; record count as N_class_i.

0 = flawless: doorstop list CRAC_I | wc -l returned a non-zero count consistent with the Class I requirement set

Test case step 5: Create a checklist for class_ii products with c5dec cra --create --category class_ii --prefix CRAC_II.

0 = flawless

Test case step 6: Count the items in CRAC_II; record as N_class_ii.

0 = flawless: count was recorded successfully

Test case step 7: Confirm N_class_ii >= N_class_i.

0 = flawless: N_class_ii was greater than N_class_i as expected, confirming Class II is a superset

Test case step 8: Create a checklist for critical products with c5dec cra --create --category critical --prefix CRAC_CR.

0 = flawless

Test case step 9: Count items in CRAC_CR; record as N_critical.

0 = flawless: count was recorded successfully

Test case step 10: Confirm N_critical >= N_class_ii.

0 = flawless: N_critical was greater than or equal to N_class_ii as expected

Test case step 11: Manually set the verdict of one CRAC_I item to pass by editing the Doorstop YAML file directly.

0 = flawless: the item's verdict field was updated to pass and saved

Test case step 12: Re-run c5dec cra --create --category class_i --prefix CRAC_I.

0 = flawless: the command completed without errors (exit code 0)

Test case step 13: Confirm the previously set pass verdict is preserved (not overwritten with not_assessed).

0 = flawless: re-inspection of the edited item confirmed verdict: pass was retained

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-078 TC: Test CRA checklist category filtering

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	13
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-015

3.26 CLI evaluation checklist export for CCv2022R1 components TRP-056

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-019-ev.png

Parent links: TCS-053 CLI export spreadsheet eval form - set SAR comp. CCv2022R1

Attribute	Value
test-date	2024-07-16
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-019

3.27 TCER: CRA SBOM auto-verification TRP-082

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Confirm no SBOM Doorstop document exists in the project with doorstop list | grep SBOM.

0 = flawless: the command produced no output, confirming no SBOM document was present

Test case step 2: Run the auto-verification with no SBOM present: c5dec cra --verify --prefix CRAC.

0 = flawless: the command completed without runtime errors

Test case step 3: Inspect the verdict of item cra_ii_1_1 in the CRAC Doorstop document and confirm it is fail.

0 = flawless: the verdict field was fail, as expected when no SBOM document is present

Test case step 4: Generate an SBOM in CycloneDX format and import it to Doorstop with c5dec sbom generate and c5dec sbom import.

0 = flawless: SBOM was generated and the SBOM Doorstop document was created successfully

Test case step 5: Re-run the auto-verification: c5dec cra --verify --prefix CRAC.

0 = flawless: the command completed without errors (exit code 0)

Test case step 6: Re-inspect the verdict of cra_ii_1_1 and confirm it has changed to pass.

0 = flawless: the verdict was updated to pass after the SBOM document was detected

Test case step 7: Export the checklist to Excel with c5dec cra --export /tmp/cra_verified.xlsx --prefix CRAC.

0 = flawless: the export completed without errors and the file was written to /tmp/cra_verified.xlsx

Test case step 8: Open /tmp/cra_verified.xlsx and verify the cra_ii_1_1 row shows verdict pass and a non-empty evidence field.

0 = flawless: the row was found with verdict pass and the expected evidence field populated

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-079 TC: Test CRA SBOM auto-verification

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	8
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-016

3.28 TCER: SBOM import to Doorstop and validate completeness TRP-083

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Prepare a sample CycloneDX SBOM JSON file /tmp/sample_sbom.json with at least three components carrying all required fields.

0 = flawless: the sample file was created with three components, each carrying name, version, type, purl, and licenses fields

Test case step 2: Initialise a fresh temporary project at /tmp/sbom-import-test with c5dec new --prefix MRS.

0 = flawless: the directory and Doorstop root document were created without errors

Test case step 3: Import the SBOM into a Doorstop document with c5dec sbom import /tmp/sample_sbom.json --prefix SBOM.

0 = flawless

Test case step 4: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 5: Confirm the Doorstop document SBOM has been created with doorstop list SBOM.

0 = flawless: the SBOM document was listed successfully

Test case step 6: Inspect the first Doorstop item and confirm all required metadata attributes are present: component_name, component_version, component_type, license, purl.

0 = flawless: all five attributes were present in the first SBOM item

Test case step 7: Run c5dec sbom validate --prefix SBOM and confirm it exits with code 0 and reports all items as valid.

0 = flawless: validation passed with no reported issues

Test case step 8: Import the same SBOM again with a version label: c5dec sbom import /tmp/sample_sbom.json --prefix SBOM --version v1.0.

0 = flawless: the command completed without errors

Test case step 9: Confirm a new versioned Doorstop document is created without overwriting the previous SBOM document.

0 = flawless: a distinct versioned document was created and the original SBOM document remained intact

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-080 TC: Test SBOM import to Doorstop and validate completeness

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	9
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-017

3.29 TCER: SBOM SPDX format and Syft-not-installed error path TRP-084

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Part A — SPDX format generation

Test case step 1: Generate an SBOM in SPDX format from the c5dec/ directory with c5dec sbom generate -t c5dec/ -f spdx -o /tmp/sbom_spdx.json.

0 = flawless

Test case step 2: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 3: Confirm /tmp/sbom_spdx.json exists and is non-empty.

0 = flawless: the file was present and contained data

Test case step 4: Inspect the top-level keys of the JSON output with python3 -c "import json; d=json.load(open(...)); print(list(d.keys()))".

0 = flawless: the key list was printed successfully

Test case step 5: Confirm the output contains the key spdxVersion.

0 = flawless: spdxVersion was present among the top-level keys

Test case step 6: Confirm the value of spdxVersion starts with SPDX-.

0 = flawless: the value was SPDX-2.3 (or equivalent), satisfying the SPDX 2.x format requirement

Part B — Syft-not-installed error path

Test case step 7: Temporarily rename the Syft binary to simulate its absence with sudo mv "$SYFT_PATH" "${SYFT_PATH}.bak".

0 = flawless: the binary was renamed and removed from PATH successfully

Test case step 8: Attempt to generate an SBOM with c5dec sbom generate -t c5dec/ -f cyclonedx-json -o /tmp/sbom_nosyft.json.

0 = flawless: the command was executed as expected

Test case step 9: Confirm the exit code is non-zero.

0 = flawless: the exit code was non-zero, indicating a handled error condition

Test case step 10: Confirm the error output includes a message referencing Syft.

0 = flawless: the stderr output contained a descriptive message referencing Syft not being installed

Test case step 11: Restore the Syft binary with sudo mv "${SYFT_PATH}.bak" "$SYFT_PATH".

0 = flawless: the binary was restored and confirmed available on PATH

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-081 TC: Test SBOM SPDX format and Syft-not-installed error path

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	11
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-018

4.0 Cryptography, CCT and SSDLC test results

4.1 Default ETR document parts generation TRP-057

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-020-ev.png

Parent links: TCS-054 Generate default parts of an ETR document

Attribute	Value
test-date	2024-07-16
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-020

4.2 Specific ETR document parts generation TRP-058

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

1 = insignificant defect: Empty files are created even for invalid family names (see evidence file). A validation previous to generating the files would help.

Defect summary description

Empty files are created even for invalid family names (see evidence file). A validation previous to generating the files would help.

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

Indicate in the CCT user manual the default available families. UPDATE 18/07/2024: this issues is not considered a priority for the beta release as functionality does not get affected.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-021-ev.png

Parent links: TCS-055 Generate specific parts of an ETR document

Attribute	Value
test-date	2024-07-16
tester	IVS
defect-category	1 = insignificant defect
release	beta
source_prefix	TRB
source_uid	TRB-021

4.3 ETR report template generation via DocEngine TRP-059

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

See linked files in frontmatter references (if any), e.g., screenshots, logs, and exports.

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/trb-022a-ev.png docs/specs/trp/assets/trb-022b-ev.png

Parent links: TCS-056 Generate ETR report template using C5-DEC DocEngine

Attribute	Value
test-date	2024-07-16
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-022

4.4 ETR generation from exported evaluation checklist TRP-060

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

3 = major defect: The etr command used on a checklist previously generated with the export command does not generate an output

Defect summary description

The etr command used on a checklist previously generated with the export command does not generate an output

Text execution evidence

N/A

Comments

Changing the default tab name in etrInput-tst-057.xsls tu "WU" (with eval-wu-sheet: "WU" uncommented in c5dec_params.yml) did not have a different effect.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-057 Generate parts of ETR report with exported evaluation checklist

Attribute	Value
test-date	2024-07-16
tester	IVS
defect-category	3 = major defect
release	beta
source_prefix	TRB
source_uid	TRB-023

4.5 Publishing project assets in HTML and Markdown TRP-061

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Review execution outcome for the linked test case.

0 = flawless: Defect-free test execution: 0 = flawless

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

A message informing of the output path is needed.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-058 Publish the project assets in HTML and Markdown

Attribute	Value
test-date	2024-07-16
tester	IVS
defect-category	0 = flawless
release	beta
source_prefix	TRB
source_uid	TRB-024

4.6 TCER: SHA-256 hash computation and verification TRP-085

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Create a test file with known content: echo "c5dec integrity test" > /tmp/test_integrity.txt.

0 = flawless: the file was created with the expected content

Test case step 2: Compute the SHA-256 digest with c5dec crypto hash /tmp/test_integrity.txt.

0 = flawless: the command completed without errors and printed a hex digest

Test case step 3: Record the printed hex digest as DIGEST.

0 = flawless: the 64-character lowercase hex string was recorded

Test case step 4: Verify the file against the recorded digest with c5dec crypto verify-hash /tmp/test_integrity.txt "$DIGEST".

0 = flawless

Test case step 5: Confirm the output contains OK and the exit code is 0.

0 = flawless: the output included OK and the command exited with code 0

Test case step 6: Cross-check the digest independently using sha256sum /tmp/test_integrity.txt and confirm both digests are identical.

0 = flawless: the sha256sum output matched the digest produced by c5dec crypto hash exactly

Test case step 7: Modify the file content with echo "tampered" >> /tmp/test_integrity.txt.

0 = flawless: the file was modified as expected

Test case step 8: Re-run the verification with the original digest: c5dec crypto verify-hash /tmp/test_integrity.txt "$DIGEST".

0 = flawless: the command ran and produced the expected output

Test case step 9: Confirm the output contains MISMATCH and the exit code is non-zero.

0 = flawless: the output contained MISMATCH and the exit code was non-zero, correctly detecting the tampered file

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-082 TC: Test SHA-256 hash computation and verification

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	9
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-019

4.7 TCER: Shamir's Secret Sharing split and recovery TRP-086

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Split a known hex secret into 5 shares requiring any 3 for recovery: c5dec crypto shamir-split deadbeef01234567 -n 5 -k 3.

0 = flawless

Test case step 2: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 3: Confirm exactly 5 share strings are printed (one per line).

0 = flawless: exactly 5 lines were output, each containing a valid share string

Test case step 4: Record all 5 shares as SHARE_1 through SHARE_5.

0 = flawless: all shares were recorded

Test case step 5: Recover the secret using shares 1, 3, and 5: c5dec crypto shamir-recover "$SHARE_1" "$SHARE_3" "$SHARE_5".

0 = flawless

Test case step 6: Confirm the recovered secret is deadbeef01234567.

0 = flawless: the recovered secret matched the original exactly

Test case step 7: Recover the secret using a different combination — shares 2, 4, and 5: c5dec crypto shamir-recover "$SHARE_2" "$SHARE_4" "$SHARE_5".

0 = flawless

Test case step 8: Confirm the recovered secret is again deadbeef01234567.

0 = flawless: recovery with an alternate k-of-n subset produced the same correct secret

Test case step 9: Attempt recovery using only 2 shares (below threshold): c5dec crypto shamir-recover "$SHARE_1" "$SHARE_2".

0 = flawless: the command was executed as required

Test case step 10: Confirm the output either produces an incorrect value or the command exits with a non-zero code — secret must NOT be reconstructible below threshold k.

0 = flawless: the output did not match the original secret, confirming the threshold constraint is enforced

Test case step 11: Test with a 128-bit hex string: c5dec crypto shamir-split "$SECRET" -n 3 -k 2.

0 = flawless: the command produced 3 shares without errors

Test case step 12: Record both usable shares and recover with any 2: c5dec crypto shamir-recover "$SHARE_A" "$SHARE_C".

0 = flawless

Test case step 13: Confirm the recovered secret matches the original $SECRET.

0 = flawless: the recovery was exact, confirming correct handling of longer secrets

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-083 TC: Test Shamir's Secret Sharing split and recovery

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	13
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-020

4.8 TCER: GPG file signing and signature verification TRP-087

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Create a test file with echo "c5dec signing test $(date -Iseconds)" > /tmp/sign_test.txt.

0 = flawless: the file was created successfully; the GPG batch key was generated in the precondition step without errors and GPG_KEYID was recorded

Test case step 2: Sign the file using the test key: c5dec crypto sign /tmp/sign_test.txt --key "$GPG_KEYID".

0 = flawless

Test case step 3: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 4: Confirm a detached signature file has been created and record its path as SIG_PATH.

0 = flawless: the signature file was present at the expected path

Test case step 5: Verify the signature: c5dec crypto verify-sig /tmp/sign_test.txt "$SIG_PATH".

0 = flawless

Test case step 6: Confirm the output contains VALID and the exit code is 0.

0 = flawless: output contained VALID and exit code was 0

Test case step 7: Tamper with the signed file: echo "tampered" >> /tmp/sign_test.txt.

0 = flawless: the file content was modified as required

Test case step 8: Re-run the verification: c5dec crypto verify-sig /tmp/sign_test.txt "$SIG_PATH".

0 = flawless: the command ran against the tampered file

Test case step 9: Confirm the output contains INVALID and the exit code is non-zero.

0 = flawless: output contained INVALID and exit code was non-zero, correctly rejecting the tampered file

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

Teardown was completed as specified: the temporary GPG key was deleted with gpg --batch --yes --delete-secret-and-public-key "$GPG_KEYID" and all temporary files were removed.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-084 TC: Test GPG file signing and signature verification

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	9
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-021

4.9 TCER: GPG file encryption and decryption TRP-088

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Create a test plaintext file with echo "$PLAINTEXT" > /tmp/encrypt_test.txt.

0 = flawless: the file was created with the expected content; the GPG test key from TCS-084 was available

Test case step 2: Encrypt the file for the test recipient: c5dec crypto encrypt /tmp/encrypt_test.txt -r "$GPG_KEYID" -o /tmp/encrypt_test.txt.gpg.

0 = flawless

Test case step 3: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 4: Confirm /tmp/encrypt_test.txt.gpg exists and is non-empty.

0 = flawless: the encrypted file was present and non-empty

Test case step 5: Confirm the encrypted file is not human-readable by checking it does not contain the original plaintext.

0 = flawless: the grep check returned Content is encrypted, confirming the plaintext is not exposed

Test case step 6: Decrypt the file: c5dec crypto decrypt /tmp/encrypt_test.txt.gpg -o /tmp/encrypt_test_recovered.txt.

0 = flawless

Test case step 7: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 8: Confirm the decrypted content matches the original with diff /tmp/encrypt_test.txt /tmp/encrypt_test_recovered.txt.

0 = flawless: diff produced no output, confirming identical file contents

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

Teardown was completed as specified: the temporary GPG key was deleted and all temporary files were removed.

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-085 TC: Test GPG file encryption and decryption

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	8
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-022

4.10 TCER: NaCl Ed25519 key generation, signing, and verification TRP-089

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Generate a new Ed25519 key pair with c5dec crypto nacl-keygen.

0 = flawless

Test case step 2: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 3: Confirm the output contains a verify_key: line and a signing_key: line, each followed by a hex string.

0 = flawless: both lines were present in the output with valid hex values

Test case step 4: Record the values as VERIFY_KEY and SIGNING_KEY respectively.

0 = flawless: both values were recorded

Test case step 5: Sign a known message using the signing key: c5dec crypto nacl-sign "hello c5dec" "$SIGNING_KEY".

0 = flawless

Test case step 6: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 7: Record the printed hex output as SIGNED_HEX.

0 = flawless: the hex-encoded signed message was recorded

Test case step 8: Verify the signed message using the verify key: c5dec crypto nacl-verify "$SIGNED_HEX" "$VERIFY_KEY".

0 = flawless

Test case step 9: Confirm the output is exactly hello c5dec and the exit code is 0.

0 = flawless: the recovered message matched the original input exactly

Test case step 10: Attempt verification with a corrupted signed message (flip one character): c5dec crypto nacl-verify "$CORRUPT_HEX" "$VERIFY_KEY" || echo "Verification failed".

0 = flawless: the command executed as required

Test case step 11: Confirm the command fails (non-zero exit code) or prints an error — the original message must NOT be returned for a corrupted signature.

0 = flawless: verification failed as expected; the original message was not returned

Test case step 12: Generate a second key pair with c5dec crypto nacl-keygen and record the new WRONG_VERIFY_KEY.

0 = flawless: a distinct key pair was generated successfully

Test case step 13: Attempt verification with the wrong verify key and confirm verification fails.

0 = flawless: c5dec crypto nacl-verify "$SIGNED_HEX" "$WRONG_VERIFY_KEY" failed as expected, confirming that a mismatched key pair cannot produce a successful verification

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-086 TC: Test NaCl Ed25519 key generation, signing, and verification

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	13
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-023

5.0 CPSSA test results

5.1 TCER: CPSSA threat model generation — Threagile format TRP-090

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Prepare a minimal project at /tmp/cpssa-test with two architecture items in an ARC Doorstop document as specified in the preconditions, then generate a Threagile-format threat model with c5dec cpssa create-threat-model --format threagile --project /tmp/cpssa-test -o /tmp/cpssa-test/threat-model.yml.

0 = flawless: the project was initialised and both ARC items were created; the command completed without errors

Test case step 2: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 3: Confirm /tmp/cpssa-test/threat-model.yml exists and is non-empty.

0 = flawless: the file was present and contained YAML content

Test case step 4: Parse the YAML and confirm the top-level keys title, technical_assets, and trust_boundaries are all present.

0 = flawless: the Python assertion script ran without errors and printed All required keys present

Test case step 5: Confirm technical_assets contains at least one entry.

0 = flawless: two entries were present, corresponding to the two ARC items

Test case step 6: Confirm trust_boundaries contains at least one entry (one per unique zone value in the ARC items).

0 = flawless: two trust boundaries were present, one for the dmz zone and one for the internal zone

Test case step 7: Confirm member assets in each trust boundary correspond to the expected ARC items by zone.

0 = flawless: the dmz boundary listed ARC-001 and the internal boundary listed ARC-002

Test case step 8: Confirm that communication_links in the first technical asset references the second ARC item.

0 = flawless: ARC-001 contained a communication link pointing to ARC-002 as expected from the flows attribute

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-087 TC: Test CPSSA threat model generation — Threagile format

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	8
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-024

5.2 TCER: CPSSA threat model generation — pytm formats TRP-091

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Part A — pytm-python format

Test case step 1: Generate a pytm Python script with c5dec cpssa create-threat-model --format pytm-python --project /tmp/cpssa-test -o /tmp/cpssa-test/threat-model.py.

0 = flawless

Test case step 2: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 3: Confirm /tmp/cpssa-test/threat-model.py exists and is non-empty.

0 = flawless: the file was present and contained Python source code

Test case step 4: Inspect the content and confirm it contains tm = TM( and Boundary( using grep.

0 = flawless: both grep checks succeeded, printing TM found and Boundary found

Test case step 5: Confirm valid Python syntax with python3 -m py_compile /tmp/cpssa-test/threat-model.py.

0 = flawless: the command exited with code 0 and printed Syntax OK

Part B — pytm-json format

Test case step 6: Generate a pytm JSON file with c5dec cpssa create-threat-model --format pytm-json --project /tmp/cpssa-test -o /tmp/cpssa-test/threat-model-pytm.json.

0 = flawless

Test case step 7: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 8: Confirm the output is valid JSON with python3 -m json.tool.

0 = flawless: Valid JSON was printed; the file parsed without errors

Test case step 9: Confirm the JSON contains a dataflows key using the Python assertion script.

0 = flawless: the assertion passed and dataflows key present was printed

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-088 TC: Test CPSSA threat model generation — pytm formats

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	9
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-025

5.3 TCER: CPSSA Markdown report generation TRP-092

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Generate the CPSSA Markdown report with c5dec cpssa generate-report --model /tmp/cpssa-test/threat-model.yml -o /tmp/cpssa-test/cpssa-report.md.

0 = flawless

Test case step 2: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 3: Confirm /tmp/cpssa-test/cpssa-report.md exists and is non-empty.

0 = flawless: the file was present and contained Markdown content

Test case step 4: Verify all eight expected section headings are present in the report using the grep loop.

0 = flawless: all eight sections — Executive Summary, System Description, Technical Assets, Security Requirements, Abuse Cases, Trust Boundaries, Threat Actors, and Assumptions — were reported as FOUND

Test case step 5: Confirm all eight sections are reported as FOUND.

0 = flawless: no MISSING entries were reported

Test case step 6: Confirm the report contains at least one entry under the Technical Assets section.

0 = flawless: entries for both ARC items from TCS-087 were present under Technical Assets

Test case step 7: Confirm the report file is valid Markdown with no unclosed code fences or obviously malformed syntax.

0 = flawless: manual inspection confirmed well-formed Markdown structure throughout the document

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-089 TC: Test CPSSA Markdown report generation

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	7
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-026

5.4 TCER: PlantUML DFD generation from ARC items TRP-093

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Generate the PlantUML DFD with c5dec cpssa generate-dfd --project /tmp/cpssa-test -o /tmp/cpssa-test/dfd.puml.

0 = flawless

Test case step 2: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 3: Confirm /tmp/cpssa-test/dfd.puml exists and is non-empty.

0 = flawless: the file was present and contained PlantUML source

Test case step 4: Confirm the file starts with @startuml and ends with @enduml.

0 = flawless: head -1 and tail -1 checks both returned OK

Test case step 5: Confirm the DFD contains at least one rectangle block representing a zone trust boundary.

0 = flawless: grep -q "rectangle" succeeded and Zone boundary found was printed

Test case step 6: Confirm the DFD contains at least one data flow arrow (-->).

0 = flawless: grep -q "\-\->" succeeded and Flow arrow found was printed

Test case step 7: Confirm both ARC item names (or their sanitised identifiers) appear in the DFD.

0 = flawless: both grep checks succeeded — the first asset (web application server / ARC-001) and the second asset (database / ARC-002) were both identifiable in the diagram source

Test case step 8: Validate PlantUML syntax using plantuml -syntax.

0 = flawless: plantuml was available and syntax validation passed without errors

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-090 TC: Test PlantUML DFD generation from ARC items

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	8
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-027

5.5 TCER: FAIR risk input template and quantitative risk analysis TRP-094

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Part A — FAIR input template generation

Test case step 1: Generate a FAIR parameter template with c5dec cpssa fair-input --model /tmp/cpssa-test/threat-model.yml -o /tmp/cpssa-test/fair-params.yml.

0 = flawless: an abuse_cases entry was added to the threat model in the precondition step; the command completed without errors

Test case step 2: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 3: Confirm /tmp/cpssa-test/fair-params.yml exists and is non-empty.

0 = flawless: the file was present and contained YAML content

Test case step 4: Parse the YAML and confirm at least one scenario block contains the required PERT distribution keys lef and lm.

0 = flawless: the Python assertion script ran without errors and printed All required PERT keys present

Part B — Quantitative risk analysis

Test case step 5: Create the output directory with mkdir -p /tmp/cpssa-test/risk-results.

0 = flawless

Test case step 6: Run the quantitative risk analysis with c5dec cpssa risk-analysis --model /tmp/cpssa-test/threat-model.yml -o /tmp/cpssa-test/risk-results --simulations 1000 --fair-params /tmp/cpssa-test/fair-params.yml.

0 = flawless: the simulation completed without errors

Test case step 7: Verify the command completes without errors (exit code 0).

0 = flawless

Test case step 8: Confirm the output directory contains a CSV file.

0 = flawless: ls /tmp/cpssa-test/risk-results/*.csv | head -1 returned at least one CSV path

Test case step 9: Confirm the output directory contains fair-risk-summary.md.

0 = flawless: the file was present at the expected path

Test case step 10: Inspect the CSV file and confirm it contains at least one data row beyond the header.

0 = flawless: wc -l returned a count of at least 2 (header + one data row)

Test case step 11: Inspect fair-risk-summary.md and confirm it contains at least one risk scenario entry with a numeric risk value.

0 = flawless: grep -E "[0-9]+" returned one or more lines containing numeric risk values

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-091 TC: Test FAIR risk input template and quantitative risk analysis

Attribute	Value
test-date	2026-03-01
tester	AAT
defect-category	0 = flawless
passed-steps	11
failed-steps	0
not-executed-steps	0
release-version	1.0
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-028

5.6 TCER: Time report conversion from OpenProject CSV TRP-100

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Inside the container, run c5dec timerep -i c5dec/assets/tshparams/openproject_params.csv -o /tmp/timerep_output.

0 = flawless: the command was executed and completed without errors

Test case step 2: Verify the command completes without errors.

0 = flawless: exit code was 0

Test case step 3: Inspect the output file at /tmp/timerep_output.

0 = flawless: the file was present and non-empty

Test case step 4: Confirm the output contains all time entries from the input CSV.

0 = flawless: each row in the input CSV was represented in the output; no entries were missing or duplicated

Test case step 5: Confirm each row has the expected IAL columns: user, date (YYYY-MM-DD), hours, project, and work package.

0 = flawless: all five columns were present and populated for every row; date values were in YYYY-MM-DD format

Test case step 6: Run with an invalid CSV path and confirm a descriptive error is printed.

0 = flawless: the command printed a descriptive error message and exited with a non-zero exit code

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-071 TC: Test time report conversion from OpenProject CSV

Attribute	Value
test-date	2026-03-10
tester	AAT
defect-category	0 = flawless
passed-steps	6
failed-steps	0
not-executed-steps	0
release-version	1.2
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-029

5.7 TCER: Timesheet consolidation and cost report generation TRP-101

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Place at least two IAL-format timesheet files in /tmp/tsh_input/ (output from TCS-071 was used as the first file; a second was created manually with three additional entries).

0 = flawless: both files were placed in the directory successfully

Test case step 2: Run consolidation: c5dec consolidate -i /tmp/tsh_input -o /tmp/tsh_consolidated.csv.

0 = flawless: the command completed without errors (exit code 0)

Test case step 3: Verify the command completes without errors.

0 = flawless: exit code was 0

Test case step 4: Confirm the consolidated output contains all entries from both input files.

0 = flawless: the consolidated CSV contained all rows from both input files; no entries were missing or duplicated

Test case step 5: Run the cost report: c5dec costrep -i /tmp/tsh_consolidated.csv -o /tmp/costrep_output.

0 = flawless: the command completed without errors (exit code 0)

Test case step 6: Verify the cost report completes without errors.

0 = flawless: exit code was 0

Test case step 7: Confirm the cost report output contains per-person cost rows and a total row.

0 = flawless: the output file contained per-person cost breakdowns and a grand total row as expected

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-072 TC: Test timesheet consolidation and cost report generation

Attribute	Value
test-date	2026-03-10
tester	AAT
defect-category	0 = flawless
passed-steps	7
failed-steps	0
not-executed-steps	0
release-version	1.2
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-030

5.8 TCER: ISMS folder content verification TRP-102

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Navigate to the test ISMS folder: tests/content/c5dec-isms-test/.

0 = flawless: the directory was present and accessible

Test case step 2: Confirm the folder contains a reference document list configuration.

0 = flawless: the configuration file was present and readable

Test case step 3: Run the ISMS verification via the Python API: from c5dec.core.isms import DocListAssistant; a = DocListAssistant('<path_to_config>'); a.verify().

0 = flawless: the verifier ran without uncaught exceptions and printed the compliance summary

Test case step 4: Confirm the output includes lists of matching, missing, and unexpected documents.

0 = flawless: the output correctly categorised all documents present in the test folder; zero discrepancies for the baseline state

Test case step 5: Add a dummy file to the ISMS folder and re-run; confirm it appears as unexpected.

0 = flawless: dummy_extra.pdf was flagged as unexpected in the re-run output

Test case step 6: Remove a required file and re-run; confirm it appears as missing.

0 = flawless: the removed file was reported as missing; the previously added dummy file was still flagged as unexpected

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-073 TC: Test ISMS folder content verification

Attribute	Value
test-date	2026-03-10
tester	AAT
defect-category	0 = flawless
passed-steps	6
failed-steps	0
not-executed-steps	0
release-version	1.2
verification-method	T, I
release	stable
source_prefix	TRS
source_uid	TRS-031

5.9 TCER: SBOM generation, import, diff, and validation TRP-103

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A
Note: Syft was installed in the dev container and available on PATH

Test execution results

Test case step 1: Generate an SBOM in CycloneDX format: c5dec sbom generate -t c5dec/ -f cyclonedx-json -o /tmp/sbom_v1.json.

0 = flawless: the command completed without errors; /tmp/sbom_v1.json was written and was non-empty

Test case step 2: Verify the output file is non-empty and valid JSON.

0 = flawless: python3 -c "import json; json.load(open('/tmp/sbom_v1.json'))" exited successfully

Test case step 3: Validate the generated SBOM: c5dec sbom validate -i /tmp/sbom_v1.json.

0 = flawless: validation reported success with exit code 0

Test case step 4: Confirm validation reports success.

0 = flawless: no validation errors were reported

Test case step 5: Generate a second SBOM after adding a test dependency: c5dec sbom generate -t c5dec/ -f cyclonedx-json -o /tmp/sbom_v2.json.

0 = flawless: the second SBOM was generated successfully

Test case step 6: Run the diff command: c5dec sbom diff -a /tmp/sbom_v1.json -b /tmp/sbom_v2.json -o /tmp/sbom_diff.txt.

0 = flawless: the diff completed without errors; /tmp/sbom_diff.txt was written

Test case step 7: Inspect the diff report and confirm it lists any changed components.

0 = flawless: the diff report correctly listed component-level differences between the two SBOMs

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-074 TC: Test SBOM generation, import, diff, and validation

Attribute	Value
test-date	2026-03-10
tester	AAT
defect-category	0 = flawless
passed-steps	7
failed-steps	0
not-executed-steps	0
release-version	1.2
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-032

5.10 TCER: CRA compliance checklist generation and export TRP-104

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A

Test execution results

Test case step 1: Run the CRA checklist generation command: c5dec cra-checklist -o /tmp/cra_checklist.xlsx.

0 = flawless: the command completed without errors (exit code 0)

Test case step 2: Verify the command completes without errors.

0 = flawless: exit code was 0

Test case step 3: Confirm /tmp/cra_checklist.xlsx exists and is a valid Excel workbook.

0 = flawless: the file was present; openpyxl.load_workbook('/tmp/cra_checklist.xlsx') loaded without errors

Test case step 4: Open the workbook and confirm it contains requirement ID, category, description, and compliance status columns, with at least one row per CRA Annex I category.

0 = flawless: all four expected columns were present; rows covered all Annex I categories in the CRA requirements database

Test case step 5: Generate a second checklist with a linked SBOM: c5dec cra-checklist --sbom /tmp/sbom_v1.json -o /tmp/cra_checklist_sbom.xlsx.

0 = flawless: the command completed without errors

Test case step 6: Confirm SBOM-linked requirements are auto-marked as verified in the output.

0 = flawless: cra_ii_1_1 compliance status was set to pass in the SBOM-linked checklist

Test case step 7: Attempt to generate with a missing output path to confirm a descriptive error.

0 = flawless: the command printed a descriptive error and exited with a non-zero exit code

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-075 TC: Test CRA compliance checklist generation and export

Attribute	Value
test-date	2026-03-10
tester	AAT
defect-category	0 = flawless
passed-steps	7
failed-steps	0
not-executed-steps	0
release-version	1.2
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-033

5.11 TCER: DocEngine CRA technical documentation template TRP-105

Relevant test environment and configuration details

Software deviations: aligned with test case specification
Hardware deviations: N/A
Note: Quarto and TeX Live were available in the docEngine.Dockerfile-based dev container

Test execution results

Test case step 1: Scaffold a new CRA technical documentation template: c5dec docengine cra-tech-doc -n my-product-cra.

0 = flawless: the command completed without errors (exit code 0)

Test case step 2: Verify the command completes without errors.

0 = flawless: exit code was 0

Test case step 3: Confirm the directory ./docengine/my-product-cra/ has been created.

0 = flawless: the directory was present after the command completed

Test case step 4: Inspect the directory contents and confirm all required files are present (_quarto.yml, _variables.yml, c5dec_config_v2.yml, index.qmd, references.bib, ieee.csl, chapter QMD files, and pre-render scripts).

0 = flawless: all files listed in the test case specification were present in the scaffolded directory

Test case step 5: Edit c5dec_config_v2.yml to populate project.title and organization.name.

0 = flawless: both fields were updated; the YAML remained valid

Test case step 6: Navigate to the template directory and run quarto render.

0 = flawless: quarto render was executed from the template directory

Test case step 7: Confirm that quarto render completes without compilation errors.

0 = flawless: the render pipeline completed without LaTeX or Quarto errors; no warnings that would indicate missing files or misconfigured metadata

Test case step 8: Verify that a PDF output file is produced in the _output/ subdirectory.

0 = flawless: a non-empty PDF file was present in _output/ after the render completed

Test case step 9: Open the PDF and confirm it contains all six chapter sections with their headings.

0 = flawless: all six CRA Annex V chapter headings were present in the rendered PDF in the correct order

Defect summary description

Defect-free test execution: 0 = flawless

Text execution evidence

N/A

Comments

N/A

Guide

Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TCS-076 TC: Test DocEngine CRA technical documentation template

Attribute	Value
test-date	2026-03-10
tester	AAT
defect-category	0 = flawless
passed-steps	9
failed-steps	0
not-executed-steps	0
release-version	1.2
verification-method	T
release	stable
source_prefix	TRS
source_uid	TRS-034