1 TCER: containerized deployment x86-64 TRS-001
Relevant test environment and configuration details
- Software deviations: aligned with test case specifications
- Hardware deviations: aligned with test case specifications
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Clone repository
- 0 = flawless
Test case step 2: Start Docker Desktop
- 0 = flawless: Docker Desktop was already running
Test case step 3: Open the C5-DEC repository in VS Code
- 0 = flawless
Test case step 4: Reopen in container with options
- 0 = flawless: the pop-up "Reopen in Container" appeared. When clicked, the two expected options (shown in the test case image) were presented in VS Code
Test case step 5: Build the CAD dev container
- 0 = flawless: the image was created without errors when selecting the
C5-DEC CAD dev containeroption
Test case step 6: Open a terminal and a shell
- 0 = flawless: a terminal and a poetry shell were manually created. The expected outcome is in alignment with the expected outcome shown in TSS-001.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawless
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TSS-001 TC: Test containerized deployment on x86-64
| Attribute | Value |
|---|---|
| test-date | 2025-05-02 |
| tester | IVS |
| defect-category | 0 = flawless |
| passed-steps | 6 |
| failed-steps | 0 |
| not-executed-steps | 0 |
| release-version | 1.0 |
| verification-method | T |
2 TCER: containerized deployment arm64 TRS-002
Relevant test environment and configuration details
- Software deviations: aligned with test case specifications
- Hardware deviations: aligned with test case specifications
Test execution results
Test case step 1: Clone repository
- 0 = flawless
Test case step 2: Open the C5-DEC repository in VS Code
- 0 = flawless
Test case step 3: Reopen in container with options
- 0 = flawless: the pop-up "Reopen in Container" appeared. When clicked, the two expected options (shown in the test case image) were presented in VS Code
Test case step 4: Build the CAD dev container
- 0 = flawless: when selecting the
C5-DEC CAD dev containeroption, the image was created without errors after approx. 3-4 minutes
Test case step 5: Open a terminal and a shell
- 0 = flawless: a terminal and a poetry shell were manually created. The expected outcome is in alignment with the expected outcome shown in TSS-002. Furthermore, the c5dec TUI was opened when running
c5dec, once inside the c5dec folder.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawless
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TSS-002 TC: Test containerized deployment on arm64
| Attribute | Value |
|---|---|
| test-date | 2025-05-02 |
| tester | IVS |
| defect-category | 0 = flawless |
| passed-steps | 5 |
| failed-steps | 0 |
| not-executed-steps | 0 |
| release-version | 1.0 |
| verification-method | T |
3 TCER: PQ cryptography container TRS-003
Relevant test environment and configuration details
- Software deviations: aligned with test case specifications
- Hardware deviations: N/A
Test execution results
Test case step 1: Clone the C5-DEC repository
- 0 = flawless
Test case step 2: Open the project in VS Code
- 0 = flawless
Test case step 3: Reopen in container with the option "C5-DEC cryptography dev container"
- 0 = flawless: the container is opened successfully
Test case step 4: Get a list of the available quantum-safe signature algorithms
- 0 = flawless: the output list corresponds to the expected outcome described in TSS-003
Test case step 5: Get a list of the available quantum-safe KEM algorithms
- 0 = flawless: the list of the available quantum-safe KEM algorithms corresponds to the expected outcome described in TSS-003
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawless
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
If we want to open the 'C5-DEC cryptography dev container' while already in VS Code and using the C5-DEC CAD container, the "Reopen in Container" option is not available. This happens more generally when trying to switch between the two C5-DEC containers.
However, this behavior is not related with C5-DEC but with the VS Code dev containers extension. The workaround is to "Reopen folder locally" for the "Reopen in Container" option to become available again.
Guide
- Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TSS-003 TC: Test PQ cryptography container
| Attribute | Value |
|---|---|
| test-date | 2025-05-02 |
| tester | IVS |
| defect-category | 0 = flawless |
| passed-steps | 5 |
| failed-steps | 0 |
| not-executed-steps | 0 |
| release-version | 1.0 |
| verification-method | T |
4 TCER: Test DocEngine TRS-004
Relevant test environment and configuration details
- Software deviations: aligned with test case specification
- Hardware deviations: N/A
Test execution results
Test case step 1: Open the C5-DEC CAD container in VS Code and activate the Python environment in the root folder
- 0 = flawless
Test case step 2: Go to the execution path of C5DEC CAD
- 0 = flawless
Test case step 3: Use quarto to render a PDF document
- 0 = flawless: the quarto command run successfully without showing error messages in the terminal; the PDF file was generated in the folder
~/c5dec/assets/report/_output/C5-DEC-CAD-DocEngine.pdf.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawless
Text execution evidence
See the file referred below in this test case report.
Comments
The final message in the terminal indicates Output created: _output/C5-DEC-CAD-DocEngine.pdf.
While this path is correct w.r.t to the root of the quarto project, it is not correct w.r.t. to the execution path and it might be confusing. If the implementation effort is minimal, it would be better to show the complete path, i.e., ~/c5dec/assets/report/_output/C5-DEC-CAD-DocEngine.pdf.
Guide
- Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
docs/specs/trs/assets/C5-DEC-CAD-DocEngine-evidence.pdf
Parent links: TSS-004 TC: Test DocEngine
| Attribute | Value |
|---|---|
| test-date | 2025-05-02 |
| tester | IVS |
| defect-category | 0 = flawless |
| passed-steps | 3 |
| failed-steps | 0 |
| not-executed-steps | 0 |
| release-version | 1.0 |
| verification-method | T |
5 TCER: creation of empty C5-DEC-based project TRS-005
Relevant test environment and configuration details
- Software deviations: aligned with test case specification
Test execution results
Test case step 1: Go to the c5dec root folder
- 0 = flawless
Test case step 2: Run the c5dec new command
- 0 = flawless: The command ran without errors, printing the following output log:
Starting C5-DEC CAD...
For usage instructions, run: ./c5dec.sh help
Executing C5-DEC CAD CLI command new...
Copied project folder to /home/alab/c5dec/myproject
Renamed devcontainer folder to /home/alab/c5dec/myproject/.devcontainer
Renamed template_devcontainer.json to /home/alab/c5dec/myproject/.devcontainer/devcontainer.json
Updated /home/alab/c5dec/myproject/.devcontainer/devcontainer.json with user=user
Updated /home/alab/c5dec/myproject/dev.Dockerfile with user=user
Updated /home/alab/c5dec/myproject/Dockerfile with user=user
Updated /home/alab/c5dec/myproject/build-c5dec.sh with username user and project name myproject
Renamed script /home/alab/c5dec/myproject/build-c5dec.sh to /home/alab/c5dec/myproject/build-myproject.sh
Updated /home/alab/c5dec/myproject/c5dec.sh with username user and project name myproject
Renamed script /home/alab/c5dec/myproject/c5dec.sh to /home/alab/c5dec/myproject/myproject.sh
Updated /home/alab/c5dec/myproject/pyproject.toml with username user and project name myproject
Renamed script /home/alab/c5dec/myproject/pyproject.toml to /home/alab/c5dec/myproject/pyproject.toml
Renamed /home/alab/c5dec/myproject/docs/specs/tra/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/tra/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/trs/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/trs/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/srs/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/srs/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/swd/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/swd/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/trb/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/trb/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/tss/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/tss/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/tst/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/tst/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/mrs/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/mrs/.doorstop.yml
Renamed /home/alab/c5dec/myproject/docs/specs/arc/template_doorstop.yml to /home/alab/c5dec/myproject/docs/specs/arc/.doorstop.yml
Deleted the copied project folder /home/alab/c5dec/myproject
Created ZIP archive at /home/alab/c5dec/myproject.zip
Test case step 3: Verify creation of a zipped project file
- 0 = flawless: the ZIP file
myproject.zipwas in the list output byls
Test case step 4: Unzip and access the project folder
- 0 = flawless: after unzipping the file, the structure of the internal folders was aligned with the one shown in the "Expected outcome" of the test case specification.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawless
Text execution evidence
See log in test case step 2 above.
Comments
N/A
Guide
- Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TSS-005 TC: Test the creation of an empty C5-DEC-based project
| Attribute | Value |
|---|---|
| test-date | 2025-05-06 |
| tester | IVS |
| defect-category | 0 = flawless |
| passed-steps | 4 |
| failed-steps | 0 |
| not-executed-steps | 0 |
| release-version | 1.0 |
| verification-method | T, I |
6 TCER: build and run an empty created project TRS-006
Relevant test environment and configuration details
- Software deviations: aligned with test case specification
- Hardware deviations: N/A
Test execution results
Test case step 1: Go to the myproject folder
- 0 = flawless
Test case step 2: Run the build script
- 0 = flawless: the log showed a healthy built of the images
myproject:v1.0andmyproject-dev:v1.0
Building myproject images...
[+] Building 3.8s (20/20) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
...
=> => writing image sha256:c8a9121fa2a7f1cdedb6cb06e68cfc0321681cf0c37148aefb026c18d8d61008 0.0s
=> => naming to docker.io/library/myproject:v1.0 0.0s
...
[+] Building 38.2s (24/29) docker:desktop-linux
[+] Building 63.0s (30/30) FINISHED docker:desktop-linux
=> [internal] load build definition from dev.Dockerfile
...
=> => writing image sha256:cf96ad7b07c3327dd0c315ddb11fc3450cf73fd6516ad8615f0838aeba7c20ab 0.0s
=> => naming to docker.io/library/myproject-dev:v1.0
...
Test case step 3: Verify that the images exist
- 0 = flawless: the images
myproject-devandmyproject, each with tagv1.0appear in the output of the executed command
REPOSITORY TAG IMAGE ID CREATED SIZE
myproject-dev v1.0 cf96ad7b07c3 13 minutes ago 3.12GB
myproject v1.0 c8a9121fa2a7 14 minutes ago 1.86GB
Test case step 4: Run the help of myproject
- 0 = flawless: the command showed the following output:
~/myproject % ./myproject.sh help
Starting myproject...
For usage instructions, run: ./myproject.sh help
---
./myproject.sh
... to open the myproject CLI help menu
./myproject.sh session
... to start an interactive myproject session
./myproject.sh <command>
... to run a myproject CLI command
./myproject.sh <command> -h
... to get help for a myproject CLI command
---
Test case step 5: Run myproject in a container
- 0 = flawless: an interactive shell within the container was created for the default user:
~/myproject % ./myproject.sh session
Starting myproject...
For usage instructions, run: ./myproject.sh help
Launching an interactive myproject session...
user@f8e418686013:~/myproject/myproject$
Test case step 6: Dependencies correctly installed
- 0 = flawless: the version of doorstop corresponds with the expected outcome for step 6 in TSS-006.
Test case step 7: Exit the session
- 0 = flawless: the container shell was closed and the control was given back to the terminal shell
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawless
Text execution evidence
See logs in the steps above.
Comments
N/A
Guide
- Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TSS-006 TC: Test build and run of new project via scripts
| Attribute | Value |
|---|---|
| test-date | 2025-05-06 |
| tester | IVS |
| defect-category | 0 = flawless |
| passed-steps | 7 |
| failed-steps | 0 |
| not-executed-steps | 0 |
| release-version | 1.0 |
| verification-method | T |
7 TCER: New project in VS Code TRS-007
Relevant test environment and configuration details
- Software deviations: aligned with test case specification
- Hardware deviations: N/A
Test execution results
Test case step 1: Open myproject in VS Code.
- 0 = flawless
Test case step 2: Select the "Reopen in Container" option or use the command palette to reopen in a container.
- 0 = flawless: the container environment was successfully initialized.
Test case step 3: Open a terminal in VS Code and run poetry shell
- 0 = flawless: the
poetry shellcommand successfully activated the environment as shown in the expected outcome.
Test case step 4: Verify that dependencies are installed by checking the version of quarto.
- 0 = flawless: The
quarto -Vcommand returned the expected version1.6.43. Moreover, the other suggested dependencies were verified successfully.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawless
Text execution evidence
N/A
Comments
N/A
Guide
- Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TSS-007 TC: Test new project in VS Code
| Attribute | Value |
|---|---|
| test-date | 2025-05-06 |
| tester | IVS |
| defect-category | 0 = flawless |
| passed-steps | 4 |
| failed-steps | 0 |
| not-executed-steps | 0 |
| release-version | 1.0 |
| verification-method | T |