1 Verify persisted output locations for VECTOR-Code artifacts TCS-001

Preconditions and setup actions

  • Prepare a readable sample project directory.
  • Ensure the current workspace is writable.

Test steps

  1. Run python3 main.py <sample-project> --name sample-app from the VECTOR-Code directory.
  2. Inspect the output/databases, output/results, and output/cbom directories after execution.
  3. Confirm that generated artifacts remain accessible after the process exits.

Expected outcome

  1. VECTOR-Code creates the expected output directory structure.
  2. Generated analysis artifacts are written to local files.
  3. Artifact paths remain inspectable after completion.

Parent links: SRS-001 Persist output artifacts in predictable locations

Child links: TRP-001 Starter report for persisted output verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data A readable sample project path with writable local output directories
version 0.1

2 Verify supported language detection TCS-002

Preconditions and setup actions

  • Prepare sample repositories containing known language mixes.
  • Ensure cloc is available in the runtime environment.

Test steps

  1. Run the VECTOR-Code workflow against a sample project containing supported languages.
  2. Observe the reported detected languages and percentages.
  3. Repeat with a project that contains no supported language above threshold.

Expected outcome

  1. Supported languages above threshold are reported.
  2. Python, C, and C++ map to the implemented source-analysis path.
  3. A project with no supported language above threshold terminates with an explicit warning or failure.

Parent links: SRS-002 Detect supported source languages from the target project

Child links: TRP-002 Starter report for language detection verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Sample projects containing Python, C, and C++ code above and below threshold
version 0.1

3 Verify CodeQL database creation TCS-003

Preconditions and setup actions

  • Prepare a readable sample project.
  • Ensure the CodeQL CLI is installed and reachable from PATH.

Test steps

  1. Run VECTOR-Code against the sample project.
  2. Inspect the created database directories under output/databases.
  3. Verify that languages mapping to the same CodeQL language do not produce duplicate database names.

Expected outcome

  1. CodeQL databases are created for the detected CodeQL languages.
  2. Database names follow the db-<codeql-language> convention.
  3. Duplicate databases are not created for multiple source languages that share the same CodeQL target.

Parent links: SRS-003 Create CodeQL databases through the external CLI

Child links: TRP-003 Starter report for CodeQL database verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 3
test_data Sample project with detectable supported language content and CodeQL CLI installed
version 0.1

4 Verify SARIF generation from CodeQL queries TCS-004

Preconditions and setup actions

  • Prepare created CodeQL databases.
  • Ensure the configured query paths exist.

Test steps

  1. Run the VECTOR-Code workflow until the query stage completes.
  2. Inspect the output/results directory.
  3. Confirm that missing query paths or failed queries are reported explicitly.

Expected outcome

  1. Successful query execution creates SARIF output files.
  2. SARIF files follow the crypto-<language>.sarif naming pattern.
  3. Query-path or execution failures do not appear as successful findings.

Parent links: SRS-004 Run cryptographic inventory queries on created databases

Child links: TRP-004 Starter report for SARIF generation verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 3
test_data Created CodeQL databases and available inventory query packs
version 0.1

5 Verify SARIF-to-CBOM conversion TCS-005

Preconditions and setup actions

  • Prepare one or more valid SARIF result files.
  • Ensure the cryptobom command is available.

Test steps

  1. Run the VECTOR-Code workflow through the CBOM generation stage.
  2. Inspect the output/cbom directory.
  3. Confirm that failed conversions are reported and omitted from the successful output set.

Expected outcome

  1. CBOM JSON files are created from valid SARIF files.
  2. Generated file names derive from the source SARIF stem.
  3. Failed conversions do not appear as successful outputs.

Parent links: SRS-005 Convert source-analysis findings into CBOM artifacts

Child links: TRP-005 Starter report for SARIF-to-CBOM verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Valid SARIF output files and cryptobom CLI installed
version 0.1

6 Verify raw TLS scan output generation TCS-006

Preconditions and setup actions

  • Prepare a reachable TLS-enabled endpoint.
  • Ensure testssl.sh is available at the configured path.

Test steps

  1. Run python3 network-scanning.py --protocol tls --target <target> --port <port>.
  2. Inspect the created raw TLS scan file.
  3. Confirm that the file is non-empty and associated with the requested target.

Expected outcome

  1. The TLS workflow accepts valid target and port input.
  2. A raw JSON scan file is created.
  3. Empty or missing output files are treated as failure conditions.

Parent links: SRS-006 Scan TLS-enabled services by target and port

Child links: TRP-006 Starter report for raw TLS scan verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Controlled TLS endpoint and reachable port
version 0.1

7 Verify TLS CBOM decomposition TCS-007

Preconditions and setup actions

  • Prepare a valid raw TLS scan JSON file.
  • Ensure the TLS converter script and mapping files are present.

Test steps

  1. Run the TLS workflow through CBOM generation.
  2. Inspect the generated TLS CBOM file.
  3. Confirm that the output contains decomposed algorithm elements rather than only opaque suite names.

Expected outcome

  1. The converter generates a CBOM JSON file for the TLS scan.
  2. Mapped cipher-suite components appear as explicit algorithm records.
  3. Hybrid or PQ-related entries are represented when present in the mapped input data.

Parent links: SRS-007 Model TLS findings with classical, hybrid, and PQ-aware decomposition

Child links: TRP-007 Starter report for TLS CBOM verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 3
test_data TLS scan JSON containing cipher suites, groups, and supported mapped entries
version 0.1

8 Verify raw SSH scan output generation TCS-008

Preconditions and setup actions

  • Prepare a reachable SSH-enabled endpoint.
  • Ensure zgrab2 is available in PATH.

Test steps

  1. Run python3 network-scanning.py --protocol ssh --target <target> --port <port>.
  2. Inspect the created raw SSH scan file.
  3. Confirm that the file is non-empty and associated with the requested target.

Expected outcome

  1. The SSH workflow accepts valid target and port input.
  2. A raw JSON scan file is created.
  3. Empty or missing output files are treated as failure conditions.

Parent links: SRS-008 Scan SSH-enabled services by target and port

Child links: TRP-008 Starter report for raw SSH scan verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Controlled SSH endpoint and reachable port
version 0.1

9 Verify Linux runtime assumptions and tool-path checks TCS-009

Preconditions and setup actions

  • Prepare one environment that satisfies the expected runtime layout.
  • Prepare one environment with a missing required script or unwritable output location.

Test steps

  1. Run VECTOR-Code and VECTOR-Network in the valid environment.
  2. Observe creation of required directories and use of configured tool paths.
  3. Repeat in the invalid environment and record the resulting failure behavior.

Expected outcome

  1. The valid environment supports normal startup and artifact generation.
  2. Required tool paths and writable directories are part of the runtime contract.
  3. Missing required scripts or paths are surfaced explicitly.

Parent links: SRS-009 Provide a ready-to-use Linux execution environment

Child links: TRP-009 Starter report for runtime assumption verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Runtime environment with and without required tool paths and writable output locations
version 0.1

10 Verify safe failure behavior TCS-010

Preconditions and setup actions

  • Prepare invalid target input, an out-of-range port, and one environment with a missing required tool.

Test steps

  1. Run the network workflow with an invalid port.
  2. Run the network workflow with an empty target.
  3. Run a workflow with a missing required tool or missing expected result file.

Expected outcome

  1. Invalid input is rejected before the scanner runs.
  2. Missing tools or files produce explicit failures.
  3. Conversion does not continue after a validation failure.

Parent links: SRS-010 Fail safely for invalid inputs, missing tools, and incomplete outputs

Child links: TRP-010 Starter report for safe failure verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Invalid ports, empty targets, missing tools, and empty result files
version 0.1

11 Verify quantum risk classification TCS-011

Preconditions:

  1. Docker engine running
  2. VS Code with the "Dev Containers" extension.

Setup:

  1. Open the project in Dev Containers.
  2. Activate virtual environment:
poetry shell

Test steps

  1. All steps below are automated in tests/test_algorithm_classifier.py. To run:
poetry run pytest tests/test_algorithm_classifier.py -v

Expected outcome

  1. All 33 test methods pass. No failures, no errors. Each algorithm is assigned the correct classification label and risk score as defined in tor/vector_score/data/algorithm-risk-catalog.yaml:

Category 1 — quantum-vulnerable (expected: classification = "quantum-vulnerable", risk_score = "high")

1.1. Running classify("RSA", "pke", None) — asserts classification == "quantum-vulnerable" and risk_score == "high". 1.2. Running classify("ECDHE", "key-agree", None) — asserts classification == "quantum-vulnerable" and risk_score == "high". 1.3. Running classify("DHE", "key-agree", None) — asserts classification == "quantum-vulnerable" and risk_score == "high". 1.4. Running classify("ECDSA", "signature", None) — asserts classification == "quantum-vulnerable" and risk_score == "high". 1.5. Running classify("DSA", "signature", None) — asserts classification == "quantum-vulnerable" and risk_score == "high".

Category 2 — classically-deprecated (expected: classification = "classically-deprecated", risk_score = "high")

1.6. Running classify("RC4", "stream-cipher", None) — asserts classification == "classically-deprecated" and risk_score == "high". 1.7. Running classify("DES", "block-cipher", None) — asserts classification == "classically-deprecated". 1.8. Running classify("MD5", "hash", None) — asserts classification == "classically-deprecated".

Category 3 — non-hybrid (expected: classification = "non-hybrid", risk_score = "medium")

1.9. Running classify("ML-KEM-768", "kem", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.10. Running classify("ML-KEM-1024", "kem", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.11. Running classify("ML-DSA-65", "signature", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.12. Running classify("SLH-DSA", "signature", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.13. Running classify("ML-KEM-512", "kem", None) — asserts classification == "non-hybrid". 1.14. Running classify("AES-128-GCM", "block-cipher", "128") — asserts classification == "non-hybrid" and risk_score == "medium". 1.15. Running classify("SHA-1", "hash", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.16. Running classify("3DES-EDE", "block-cipher", None) — asserts classification == "non-hybrid" and risk_score == "medium".

Category 4 — hybrid (expected: classification = "hybrid", risk_score = "low")

1.17. Running classify("X25519MLKEM768", None, None) — asserts classification == "hybrid" and risk_score == "low". 1.18. Running classify("SecP256r1MLKEM768", None, None) — asserts classification == "hybrid" and risk_score == "low".

Category 5 — quantum-safe (expected: classification = "quantum-safe", risk_score = "none")

1.19. Running classify("AES-256-GCM", "block-cipher", "256") — asserts classification == "quantum-safe" and risk_score == "none". 1.20. Running classify("SHA-256", "hash", "256") — asserts classification == "quantum-safe" and risk_score == "none". 1.21. Running classify("SHA-384", "hash", "384") — asserts classification == "quantum-safe". 1.22. Running classify("SHA-512", "hash", "512") — asserts classification == "quantum-safe". 1.23. Running classify("ChaCha20-Poly1305", "ae", None) — asserts classification == "quantum-safe".

Category 6 — unknown (expected: classification = "unknown", risk_score = "high")

1.24. Running classify("SuperQuantumAlgo9000", None, None) — asserts classification == "unknown".

Parent links: SRS-011 Classify cryptographic algorithms in a CBOM by quantum risk

Child links: TRP-011 Test report for algorithm classification

Attribute Value
platform Visual Studio Code
execution_type Manual
verification_method T
release alpha
complexity 2
version 0.2

12 Verify that annotated CBOM contains 'pqcmat' risk properties TCS-012

Preconditions:

  1. Docker engine running
  2. VS Code with the "Dev Containers" extension.

Setup:

  1. Open the project with Dev Containers.
  2. Activate virtual environment:
poetry shell

Test steps

  1. Run the unit tests at tests/test_cbom_scorer.py:
pytest tests/test_cbom_scorer.py
  1. Open the unit tests file and ensure that the tests cover the following aspects:

  • For each asset of type algorithm (components.cryptoProperties.assetType == "algorithm") in the annotated CBOM output, a properties array is present with the entries:

    • "name": "pqcmat:risk-classification" with a non-empty value.
    • "name": "pqcmat:risk-score" with one of the values: high, medium, low or none
    • "name": "pqcmat:rationale"
    • "name": "pqcmat:recommended-migration".

  • The annotated CBOM metadata.properties contains an entry with "name": "pqcmat:scored-at".

  • Non-algorithm components (protocol, certificate and any where components.cryptoProperties.assetType != "algorithm") do not have pqcmat: properties.
  • CBOM files are annotated regardless of whether they use the format "type": "cryptographic-asset" or "type": "crypto-asset" to identify the components.

Expected outcome

  1. All the tests pass with no errors.
  2. All the described validations are covered by the tests.

Parent links: SRS-012 Produce a CBOM annotated with quantum risk properties

Child links: TRP-012 Test report for CBOM annotated with pqcmat properties

Attribute Value
platform Visual Studio Code
execution_type Manual
verification_method Test, Inspection
release alpha
complexity 4
test_data tests/data/sample4risk-scorer-cbom.json, tests/data/sample-short-frmt-cbom.json
version 0.3

13 Verify creation of unified CBOM TCS-013

Preconditions:

  1. Docker engine running
  2. VS Code with the "Dev Containers" extension.

Setup:

  1. Open the project with Dev Containers.
  2. Activate the virtual environment:
poetry shell

Test steps

  1. Run vector code on the test repository with Python and C++ files
vector code tests/data/mixed_repo/
  1. Go to the tor/vector_code/output/ folder and verify that the following folders exist:
  • cbom: containing the crypto-combined-cbom.json file
  • databases: with folders db-cpp and db-python
  • results: containing two SARIF files, one for C++ and one for Python
  1. From the output folder, verify that algorithms from both source files were detected by counting the number of occurrences of "location": "crypto_func.py" and "location": "crypto_utils.cpp" in the CBOM file:
  • grep -c '"location": "crypto_func.py"' cbom/crypto-combined-cbom.json
  • grep -c '"location": "crypto_utils.cpp"' cbom/crypto-combined-cbom.json

Expected outcome

  1. The command finishes with the message "Completed successfully" and shows the path of the generated CBOM.
  2. The folder exists and contains the referred components.
  3. The CBOM file contains 2 occurrences of 'crypto_func.py' and 3 of 'crypto_utils.cpp'.

Parent links: SRS-014 Produce a single unified CBOM for multi-language projects

Child links: TRP-013 Test report for unified CBOM creation verification

Attribute Value
platform Visual Studio Code
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data tests/data/mixed_repo
version 0.3

14 Verify Markdown risk report structure and source locations TCS-014

Preconditions and setup actions

  1. TCS-013 has been executed successfully; the combined CBOM exists at tor/vector_code/output/cbom/crypto-combined-cbom.json.
  2. Virtual environment is active (poetry shell). cd

Test steps

  1. From the project home (cd ~/vector-project/), run vector score on the CBOM produced by TCS-013:
vector score tor/vector_code/output/cbom/crypto-combined-cbom.json
  1. Verify that the output files crypto-combined-cbom_scored.json and crypto-combined-cbom_risk_report.md were created in the cbom directory:
ls tor/vector_code/output/cbom

  1. Open the report and verify that it has a header that includes the name of the target application and a timestamp.

  2. Verify that a summary table is present with columns "Risk classification", "Risk score" and "Count", and that the sum of the algorithms in the last column corresponds to the number of scored algorithms shown in the output of step 1.

  3. Verify that the report contains two risk classification sections titled "Quantum-safe" and "Unknown/Unclassified", each with a table including the following columns: Algorithm, Primitive, Key size, Rationale, and Recommended migration.

  4. Verify source locations for the files crypto_func.py and crypto_utils.cpp are present in the tables under each section.

  5. Verify that a "Normative references" section listing all the cited references is present.

Expected outcome

  1. The command displays the path where the risk report is written and the number of algorithms scored, which must be 5.
  2. Both _scored.json and _risk_report.md files are present.
  3. The report includes the name of the scanned target in the header and includes a timestamp in the top section.
  4. A Summary section contains a table with a "Risk classification" column and an "Algorithm count" column; the total number of algorithms reported on the summary is 5.
  5. The sub-sections "Quantum-safe" and "Unknown/unclassified" are present and each contains a table; the table has columns for Algorithm, Primitive, Key size, Rationale, and Recommended migration.
  6. Both source file names appear in the report at least once each, with an associated line number.
  7. A normative references section (## References or similar) is present and lists at least one standard.

Parent links: SRS-013 Produce a Markdown risk report grouped by classification, SRS-015 Show source locations of VECTOR-Code output in risk report

Child links: TRP-014 Test report for Markdown risk report

Attribute Value
platform Docker container or Visual Studio Code
execution_type Manual
verification_method T
release alpha
complexity 2
test_data tor/vector_code/output/cbom/crypto-combined-cbom.json (produced in TCS-013)
version 0.3