1.0 TRA-001

Test execution results for the TST in the parent link.

Parent links: TST-001 Test accessing and browsing the CC Database

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	1
defect_description	(i) The links in the item preview are empty. E.g., see the preview of family FDP_ACC or of FAU_SAA. (ii) In the item preview of components (e.g. FDP_RIP.2, FDP_ACC.2), the text shown as value for the field "Hierarchical to" is tokenized by characters. Instead, these should be words.
comments	The acceptance criteria in the parent SRS is satisfied. The data is mostly aligned to the description in the corresponding CC document (e.g. of an exception: for ACE, the info in MS-Introduction does not appear in the PDF). The reported defects affect readability.

2.0 TRA-002

Test execution results for the TST in the parent link.

docs/specs/tra/assets/tra-002-ev.png

Parent links: TST-002 Test query the CC Database

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	1
defect_description	When an element from an upper category in the hierarchy is selected, the children should be cleared to avoid confusion regarding to which element the item preview corresponds. An example of elements from different clases and families mixed in one screen is provided as a reference in the displayed pathfile.
comments	The acceptance criteria is satisfied. Yet, the defect described could render the browser view confusing.

3.0 TRA-003

Test execution results for the TST in the parent link.

Parent links: TST-003 Test exporting Security Components

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	1
defect_description	When files are exported, the links become unusable; for instance, when opened in VSCode, they open a browser pointing to a "page not found".
comments	(i) Only export to markdown files implemented in the Alpha phase. (ii) When giving a filepath that does not exist, the export does not create it. While this is ok, it would be better to display a message informing about the specific problem.

4.0 TRA-004

Test execution results for the TST in the parent link.

Parent links: TST-004 Test tailoring Security Requirements

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	n/a
comments	Not applicable to the Alpha release.

5.0 TRA-005

Test execution results for the TST in the parent link.

Parent links: TST-005 Test navigating the Knowledge Base

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	0

6.0 TRA-006

Test execution results for the TST in the parent link.

Parent links: TST-006 Test comprehensiveness of Knowledge Base

Attribute	Value
test_date	28-11-2023
tester	IVS
defect_category	0

7.0 TRA-007

Test execution results for the TST in the parent link.

Parent links: TST-007 Test interconnection of Knowledge Base

Attribute	Value
test_date	28-11-2023
tester	IVS
defect_category	2
defect_description	Some pages are missing, e.g., "Terms and Definition Register". For others, the links are missing, e.g., [Security Functional Requirements] and [Security Assurance Requirements]
comments	A link to the table of contents in each page would be useful.

8.0 TRA-008

Test execution results for the TST in the parent link.

Parent links: TST-008 Test currency of Knowledge Base

Attribute	Value
test_date	28-11-2023
tester	IVS
defect_category	0

9.0 TRA-009

Test execution results for the TST in the parent link.

Parent links: TST-009 Test cosmetic features of Knowledge Base

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	n/a
comments	Not applicable for the Alpha release.

10 TRA-010

Test execution results for the TST in the parent link.

Parent links: TST-010 Inspect CC Database-DTD mapping

Attribute	Value
test_date	28-11-2023
tester	IVS
defect_category	n/a
defect_description	Some of the verifications specified in the test steps are not observed in the code of cct.py. For instance, the attributes "part" and "patch" of the element "xref" in the DTD are not present in the XRef class.
comments	The discrepancies found do not necessarily imply an incorrect mapping, yet, to determine the correctness of the mapping by this method, a deeper inspection to track the parsing of the XML file to classes and its relation with the DTD would require more time and perhaps splitting the test case. Given that the mapping can be tested with other functional tests, TST-010 will be considered not applicable for now.

11 TRA-011

Test execution results for the TST in the parent link.

Parent links: TST-011 Test correctness of CC Database

Attribute	Value
test_date	28-11-2023
tester	IVS
defect_category	2
defect_description	The CLI application crashes if an invalid for a CC element is provided. This defect has been reported as bug #18 in the CAD GitLab repository.
comments	The correctness of the data for valid CC elements is already covered (and has been tested) in TST-001. The reported defect fails to inform the user about data not existing in the CC.

12 TRA-012

Test execution results for the TST in the parent link.

Parent links: TST-012 Test validity and conistency of bidirectional transformation

Attribute	Value
test_date	28-11-2023
tester	IVS
defect_category	ignored
comments	The test steps are vague and do not provide enough details to perform this test.

13 TRA-013

Test execution results for the TST in the parent link.

Parent links: TST-013 Test threats, risks, and countermeasures database

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	0
comments	Although the parent SRS requires an integrated source of security-related data with content from BSI Grundschutz, ISO 27005, NIST SPs, and the CC, the SRS is not mandatory. Therefore, having data only from CC is acceptable.

14 TRA-014

Test execution results for the TST in the parent link.

Parent links: TST-014 Test EUCC support

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	n/a
comments	Not applicable to the Alpha release.

15 TRA-015

Test execution results for the TST in the parent link.

Parent links: TST-015 Test availabilty of CSA Article 51 defined Security Objectives

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	n/a
comments	Not implemented for the Alpha release.

16 TRA-016

TC passed.

Parent links: TST-016 Test uniformity of storage mechanisms in CCT module

Attribute	Value
test_date	30-11-2023
tester	Arash
defect_category	0
comments	The acceptance criteria are satisfied.

17 TRA-017

Test execution results for the TST in the parent link.

Parent links: TST-017 Test automated rationale and traceability matrix generation

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	n/a
comments	Not implemented for the Alpha release.

18 TRA-018

Test execution results for the TST in the parent link.

Parent links: TST-018 Test verificaiton of rationales and traceability matrices

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	n/a
comments	Not implemented for the Alpha release.

19 TRA-019

Test execution results for the TST in the parent link.

Parent links: TST-019 Test automated consistency and completeness checks

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	n/a
comments	Not implemented for the Alpha release.

20 TRA-020

Test execution results for the TST in the parent link.

Parent links: TST-020 Test automated validation

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	n/a
comments	Not implemented for the Alpha release.

21 TRA-021

Test execution results for the TST in the parent link.

Parent links: TST-021 Test aggregation of SARs and Work Units

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	3
defect_description	The CLI crashes when running the checklist command with "tst" as evaluation checklist prefix (issue #19)"; The CLI crashes when running $ poetry run c5dec checklist <prefix> --edit <item-id> due to vim not being installed. When adding the --editor code option, it runs as expected
comments	The defect reported here is associated with issue #19 in the GitLab CAD repository.

22 TRA-022

Test execution results for the TST in the parent link.

Parent links: TST-022 Test API provision for threat import

Attribute	Value
test_date	28-11-2023
tester	IVS
defect_category	n/a
comments	The associated requirement (SRS-033) has not been implemented for the Alpha release.

23 TRA-023

Test execution results for the TST in the parent link.

Parent links: TST-023 Test transforming imported threats to CC-conformant format

Attribute	Value
test_date	28-11-2023
tester	IVS
defect_category	n/a
comments	The corresponding SRS (SRS-034) has not been implemented for the Alpha release.

24 TRA-024

Test execution results for the TST in the parent link.

Parent links: TST-024 Test automated creation of Evaluation Checklist

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	2
defect_description	The CLI crahses if an invalid component id is given. E.g.: $ poetry run c5dec checklist -c evChck-tst-024 --id aco
comments	The TUI works as expected. The reported defect affects the CLI.

25 TRA-025

Test execution results for the TST in the parent link.

Parent links: TST-025 Test evaluation progress tracking

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	4
defect_description	(i) The TUI crashes when an evaluation evidence is edited and revisited (issue #20). (ii) The CLI crashes with the option -s if the input evaluation checklist does not exist (issue #22).
comments	This defects correspond to issues #20 and #22 in the GitLab CAD repository.

26 TRA-026

Test execution results for the TST in the parent link.

Parent links: TST-026 Test Work Unit-Artifact linking

Attribute	Value
test_date	27-11-2023
tester	AAT
defect_category	[0-4]

27 TRA-027

Test execution results for the TST in the parent link.

Parent links: TST-027 Test automated generation of Observation Reports

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	n/a
comments	Not implemented for the Aplha version.

28 TRA-028

Test execution results for the TST in the parent link.

Parent links: TST-028 Test automated generation of Evaluation Technical Report

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	n/a
comments	Not implemented for the Alpha version.

29 TRA-029

Test execution results for the TST in the parent link.

Parent links: TST-029 Test flagging failed Work Units and affected artifacts

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	n/a
comments	Not addressed in the Alpha phase.

30 TRA-030

Test execution results for the TST in the parent link.

Parent links: TST-030 Test auditability of Evaluation Items

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	0

31 TRA-031

Test execution results for the TST in the parent link.

Parent links: TST-031 Test extended data model

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	0
comments	Interpreted as c5dec capturing the CC concepts and showing them in a correct and coherent manner.

32 TRA-032

Test execution results for the TST in the parent link.

Parent links: TST-032 Test CC templates

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	n/a
comments	Not implemented for the aplha release.

33 TRA-033

Test execution results for the TST in the parent link.

Parent links: TST-033 Test validation of hierarchies and dependencies of Security Component sets

Attribute	Value
test_date	29-11-2023
tester	IVS
defect_category	0

34 TRA-034

Test execution results for the TST in the parent link.

Parent links: TST-034 Test generation of Impact Analysis Report

Attribute	Value
test_date	27-11-2023
tester	IVS
defect_category	n/a
comments	Not implemented for the Alpha release.