1 Verify persisted output locations for VECTOR-Code artifacts TCS-001

Preconditions and setup actions

  • Prepare a readable sample project directory.
  • Ensure the current workspace is writable.

Test steps

  1. Run python3 main.py <sample-project> --name sample-app from the VECTOR-Code directory.
  2. Inspect the output/databases, output/results, and output/cbom directories after execution.
  3. Confirm that generated artifacts remain accessible after the process exits.

Expected outcome

  1. VECTOR-Code creates the expected output directory structure.
  2. Generated analysis artifacts are written to local files.
  3. Artifact paths remain inspectable after completion.

Parent links: SRS-001 Persist output artifacts in predictable locations

Child links: TRP-001 Starter report for persisted output verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data A readable sample project path with writable local output directories
version 0.1

2 Verify supported language detection TCS-002

Preconditions and setup actions

  • Prepare sample repositories containing known language mixes.
  • Ensure cloc is available in the runtime environment.

Test steps

  1. Run the VECTOR-Code workflow against a sample project containing supported languages.
  2. Observe the reported detected languages and percentages.
  3. Repeat with a project that contains no supported language above threshold.

Expected outcome

  1. Supported languages above threshold are reported.
  2. Python, C, and C++ map to the implemented source-analysis path.
  3. A project with no supported language above threshold terminates with an explicit warning or failure.

Parent links: SRS-002 Detect supported source languages from the target project

Child links: TRP-002 Starter report for language detection verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Sample projects containing Python, C, and C++ code above and below threshold
version 0.1

3 Verify CodeQL database creation TCS-003

Preconditions and setup actions

  • Prepare a readable sample project.
  • Ensure the CodeQL CLI is installed and reachable from PATH.

Test steps

  1. Run VECTOR-Code against the sample project.
  2. Inspect the created database directories under output/databases.
  3. Verify that languages mapping to the same CodeQL language do not produce duplicate database names.

Expected outcome

  1. CodeQL databases are created for the detected CodeQL languages.
  2. Database names follow the db-<codeql-language> convention.
  3. Duplicate databases are not created for multiple source languages that share the same CodeQL target.

Parent links: SRS-003 Create CodeQL databases through the external CLI

Child links: TRP-003 Starter report for CodeQL database verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 3
test_data Sample project with detectable supported language content and CodeQL CLI installed
version 0.1

4 Verify SARIF generation from CodeQL queries TCS-004

Preconditions and setup actions

  • Prepare created CodeQL databases.
  • Ensure the configured query paths exist.

Test steps

  1. Run the VECTOR-Code workflow until the query stage completes.
  2. Inspect the output/results directory.
  3. Confirm that missing query paths or failed queries are reported explicitly.

Expected outcome

  1. Successful query execution creates SARIF output files.
  2. SARIF files follow the crypto-<language>.sarif naming pattern.
  3. Query-path or execution failures do not appear as successful findings.

Parent links: SRS-004 Run cryptographic inventory queries on created databases

Child links: TRP-004 Starter report for SARIF generation verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 3
test_data Created CodeQL databases and available inventory query packs
version 0.1

5 Verify SARIF-to-CBOM conversion TCS-005

Preconditions and setup actions

  • Prepare one or more valid SARIF result files.
  • Ensure the cryptobom command is available.

Test steps

  1. Run the VECTOR-Code workflow through the CBOM generation stage.
  2. Inspect the output/cbom directory.
  3. Confirm that failed conversions are reported and omitted from the successful output set.

Expected outcome

  1. CBOM JSON files are created from valid SARIF files.
  2. Generated file names derive from the source SARIF stem.
  3. Failed conversions do not appear as successful outputs.

Parent links: SRS-005 Convert source-analysis findings into CBOM artifacts

Child links: TRP-005 Starter report for SARIF-to-CBOM verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Valid SARIF output files and cryptobom CLI installed
version 0.1

6 Verify raw TLS scan output generation TCS-006

Preconditions and setup actions

  • Prepare a reachable TLS-enabled endpoint.
  • Ensure testssl.sh is available at the configured path.

Test steps

  1. Run python3 network-scanning.py --protocol tls --target <target> --port <port>.
  2. Inspect the created raw TLS scan file.
  3. Confirm that the file is non-empty and associated with the requested target.

Expected outcome

  1. The TLS workflow accepts valid target and port input.
  2. A raw JSON scan file is created.
  3. Empty or missing output files are treated as failure conditions.

Parent links: SRS-006 Scan TLS-enabled services by target and port

Child links: TRP-006 Starter report for raw TLS scan verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Controlled TLS endpoint and reachable port
version 0.1

7 Verify TLS CBOM decomposition TCS-007

Preconditions and setup actions

  • Prepare a valid raw TLS scan JSON file.
  • Ensure the TLS converter script and mapping files are present.

Test steps

  1. Run the TLS workflow through CBOM generation.
  2. Inspect the generated TLS CBOM file.
  3. Confirm that the output contains decomposed algorithm elements rather than only opaque suite names.

Expected outcome

  1. The converter generates a CBOM JSON file for the TLS scan.
  2. Mapped cipher-suite components appear as explicit algorithm records.
  3. Hybrid or PQ-related entries are represented when present in the mapped input data.

Parent links: SRS-007 Model TLS findings with classical, hybrid, and PQ-aware decomposition

Child links: TRP-007 Starter report for TLS CBOM verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 3
test_data TLS scan JSON containing cipher suites, groups, and supported mapped entries
version 0.1

8 Verify raw SSH scan output generation TCS-008

Preconditions and setup actions

  • Prepare a reachable SSH-enabled endpoint.
  • Ensure zgrab2 is available in PATH.

Test steps

  1. Run python3 network-scanning.py --protocol ssh --target <target> --port <port>.
  2. Inspect the created raw SSH scan file.
  3. Confirm that the file is non-empty and associated with the requested target.

Expected outcome

  1. The SSH workflow accepts valid target and port input.
  2. A raw JSON scan file is created.
  3. Empty or missing output files are treated as failure conditions.

Parent links: SRS-008 Scan SSH-enabled services by target and port

Child links: TRP-008 Starter report for raw SSH scan verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Controlled SSH endpoint and reachable port
version 0.1

9 Verify Linux runtime assumptions and tool-path checks TCS-009

Preconditions and setup actions

  • Prepare one environment that satisfies the expected runtime layout.
  • Prepare one environment with a missing required script or unwritable output location.

Test steps

  1. Run VECTOR-Code and VECTOR-Network in the valid environment.
  2. Observe creation of required directories and use of configured tool paths.
  3. Repeat in the invalid environment and record the resulting failure behavior.

Expected outcome

  1. The valid environment supports normal startup and artifact generation.
  2. Required tool paths and writable directories are part of the runtime contract.
  3. Missing required scripts or paths are surfaced explicitly.

Parent links: SRS-009 Provide a ready-to-use Linux execution environment

Child links: TRP-009 Starter report for runtime assumption verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Runtime environment with and without required tool paths and writable output locations
version 0.1

10 Verify safe failure behavior TCS-010

Preconditions and setup actions

  • Prepare invalid target input, an out-of-range port, and one environment with a missing required tool.

Test steps

  1. Run the network workflow with an invalid port.
  2. Run the network workflow with an empty target.
  3. Run a workflow with a missing required tool or missing expected result file.

Expected outcome

  1. Invalid input is rejected before the scanner runs.
  2. Missing tools or files produce explicit failures.
  3. Conversion does not continue after a validation failure.

Parent links: SRS-010 Fail safely for invalid inputs, missing tools, and incomplete outputs

Child links: TRP-010 Starter report for safe failure verification

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data Invalid ports, empty targets, missing tools, and empty result files
version 0.1

11 Verify quantum risk classification TCS-011

Preconditions:

  1. Docker engine running
  2. VS Code with the "Dev Containers" extension.

Setup:

  1. Open the project in Dev Containers.
  2. Activate virtual environment:
poetry shell

Test steps

  1. All steps below are automated in tests/test_algorithm_classifier.py. To run:
poetry run pytest tests/test_algorithm_classifier.py -v

Expected outcome

  1. All 33 test methods pass. No failures, no errors. Each algorithm is assigned the correct classification label and risk score as defined in tor/vector_score/data/algorithm-risk-catalog.yaml:

Category 1 — quantum-vulnerable (expected: classification = "quantum-vulnerable", risk_score = "high")

1.1. Running classify("RSA", "pke", None) — asserts classification == "quantum-vulnerable" and risk_score == "high". 1.2. Running classify("ECDHE", "key-agree", None) — asserts classification == "quantum-vulnerable" and risk_score == "high". 1.3. Running classify("DHE", "key-agree", None) — asserts classification == "quantum-vulnerable" and risk_score == "high". 1.4. Running classify("ECDSA", "signature", None) — asserts classification == "quantum-vulnerable" and risk_score == "high". 1.5. Running classify("DSA", "signature", None) — asserts classification == "quantum-vulnerable" and risk_score == "high".

Category 2 — classically-deprecated (expected: classification = "classically-deprecated", risk_score = "high")

1.6. Running classify("RC4", "stream-cipher", None) — asserts classification == "classically-deprecated" and risk_score == "high". 1.7. Running classify("DES", "block-cipher", None) — asserts classification == "classically-deprecated". 1.8. Running classify("MD5", "hash", None) — asserts classification == "classically-deprecated".

Category 3 — non-hybrid (expected: classification = "non-hybrid", risk_score = "medium")

1.9. Running classify("ML-KEM-768", "kem", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.10. Running classify("ML-KEM-1024", "kem", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.11. Running classify("ML-DSA-65", "signature", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.12. Running classify("SLH-DSA", "signature", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.13. Running classify("ML-KEM-512", "kem", None) — asserts classification == "non-hybrid". 1.14. Running classify("AES-128-GCM", "block-cipher", "128") — asserts classification == "non-hybrid" and risk_score == "medium". 1.15. Running classify("SHA-1", "hash", None) — asserts classification == "non-hybrid" and risk_score == "medium". 1.16. Running classify("3DES-EDE", "block-cipher", None) — asserts classification == "non-hybrid" and risk_score == "medium".

Category 4 — hybrid (expected: classification = "hybrid", risk_score = "low")

1.17. Running classify("X25519MLKEM768", None, None) — asserts classification == "hybrid" and risk_score == "low". 1.18. Running classify("SecP256r1MLKEM768", None, None) — asserts classification == "hybrid" and risk_score == "low".

Category 5 — quantum-safe (expected: classification = "quantum-safe", risk_score = "none")

1.19. Running classify("AES-256-GCM", "block-cipher", "256") — asserts classification == "quantum-safe" and risk_score == "none". 1.20. Running classify("SHA-256", "hash", "256") — asserts classification == "quantum-safe" and risk_score == "none". 1.21. Running classify("SHA-384", "hash", "384") — asserts classification == "quantum-safe". 1.22. Running classify("SHA-512", "hash", "512") — asserts classification == "quantum-safe". 1.23. Running classify("ChaCha20-Poly1305", "ae", None) — asserts classification == "quantum-safe".

Category 6 — unknown (expected: classification = "unknown", risk_score = "high")

1.24. Running classify("SuperQuantumAlgo9000", None, None) — asserts classification == "unknown".

Parent links: SRS-011 Classify cryptographic algorithms in a CBOM by quantum risk

Child links: TRP-011 Test report for algorithm classification

Attribute Value
platform Visual Studio Code
execution_type Manual
verification_method T
release alpha
complexity 2
version 0.2

12 Verify that annotated CBOM contains 'pqcmat' risk properties TCS-012

Preconditions:

  1. Docker engine running
  2. VS Code with the "Dev Containers" extension.

Setup:

  1. Open the project with Dev Containers.
  2. Activate virtual environment:
poetry shell

Test steps

  1. Run the unit tests at tests/test_cbom_scorer.py:
pytest tests/test_cbom_scorer.py
  1. Open the unit tests file and ensure that the tests cover the following aspects:

  • For each asset of type algorithm (components.cryptoProperties.assetType == "algorithm") in the annotated CBOM output, a properties array is present with the entries:

    • "name": "pqcmat:risk-classification" with a non-empty value.
    • "name": "pqcmat:risk-score" with one of the values: high, medium, low or none
    • "name": "pqcmat:rationale"
    • "name": "pqcmat:recommended-migration".

  • The annotated CBOM metadata.properties contains an entry with "name": "pqcmat:scored-at".

  • Non-algorithm components (protocol, certificate and any where components.cryptoProperties.assetType != "algorithm") do not have pqcmat: properties.
  • CBOM files are annotated regardless of whether they use the format "type": "cryptographic-asset" or "type": "crypto-asset" to identify the components.

Expected outcome

  1. All the tests pass with no errors.
  2. All the described validations are covered by the tests.

Parent links: SRS-012 Produce a CBOM annotated with quantum risk properties

Child links: TRP-012 Test report for CBOM annotated with pqcmat properties

Attribute Value
platform Visual Studio Code
execution_type Manual
verification_method Test, Inspection
release alpha
complexity 4
test_data tests/data/sample4risk-scorer-cbom.json, tests/data/sample-short-frmt-cbom.json
version 0.3

13 Verify creation of unified CBOM TCS-013

Preconditions:

  1. Docker engine running
  2. VS Code with the "Dev Containers" extension.

Setup:

  1. Open the project with Dev Containers.
  2. Activate the virtual environment:
poetry shell

Test steps

  1. Run vector code on the test repository with Python and C++ files
vector code tests/data/mixed_repo/
  1. Go to the tor/vector_code/output/ folder and verify that the following folders exist:
  • cbom: containing the crypto-combined-cbom.json file
  • databases: with folders db-cpp and db-python
  • results: containing two SARIF files, one for C++ and one for Python
  1. From the output folder, verify that algorithms from both source files were detected by counting the number of occurrences of "location": "crypto_func.py" and "location": "crypto_utils.cpp" in the CBOM file:
  • grep -c '"location": "crypto_func.py"' cbom/crypto-combined-cbom.json
  • grep -c '"location": "crypto_utils.cpp"' cbom/crypto-combined-cbom.json

Expected outcome

  1. The command finishes with the message "Completed successfully" and shows the path of the generated CBOM.
  2. The folder exists and contains the referred components.
  3. The CBOM file contains 2 occurrences of 'crypto_func.py' and 3 of 'crypto_utils.cpp'.

Parent links: SRS-014 Produce a single unified CBOM for multi-language projects

Child links: TRP-013 Test report for unified CBOM creation verification

Attribute Value
platform Visual Studio Code
execution_type Manual
verification_method Test
release alpha
complexity 2
test_data tests/data/mixed_repo
version 0.3

14 Verify Markdown risk report structure and source locations TCS-014

Preconditions and setup actions

  1. TCS-013 has been executed successfully; the combined CBOM exists at tor/vector_code/output/cbom/crypto-combined-cbom.json.
  2. Virtual environment is active (poetry shell). cd

Test steps

  1. From the project home (cd ~/vector-project/), run vector score on the CBOM produced by TCS-013:
vector score tor/vector_code/output/cbom/crypto-combined-cbom.json
  1. Verify that the output files crypto-combined-cbom_scored.json and crypto-combined-cbom_risk_report.md were created in the cbom directory:
ls tor/vector_code/output/cbom

  1. Open the report and verify that it has a header that includes the name of the target application and a timestamp.

  2. Verify that a summary table is present with columns "Risk classification", "Risk score" and "Count", and that the sum of the algorithms in the last column corresponds to the number of scored algorithms shown in the output of step 1.

  3. Verify that the report contains two risk classification sections titled "Quantum-safe" and "Unknown/Unclassified", each with a table including the following columns: Algorithm, Primitive, Key size, Rationale, and Recommended migration.

  4. Verify source locations for the files crypto_func.py and crypto_utils.cpp are present in the tables under each section.

  5. Verify that a "Normative references" section listing all the cited references is present.

Expected outcome

  1. The command displays the path where the risk report is written and the number of algorithms scored, which must be 5.
  2. Both _scored.json and _risk_report.md files are present.
  3. The report includes the name of the scanned target in the header and includes a timestamp in the top section.
  4. A Summary section contains a table with a "Risk classification" column and an "Algorithm count" column; the total number of algorithms reported on the summary is 5.
  5. The sub-sections "Quantum-safe" and "Unknown/unclassified" are present and each contains a table; the table has columns for Algorithm, Primitive, Key size, Rationale, and Recommended migration.
  6. Both source file names appear in the report at least once each, with an associated line number.
  7. A normative references section (## References or similar) is present and lists at least one standard.

Parent links: SRS-013 Produce a Markdown risk report grouped by classification, SRS-015 Show source locations of VECTOR-Code output in risk report

Child links: TRP-014 Test report for Markdown risk report

Attribute Value
platform Docker container or Visual Studio Code
execution_type Manual
verification_method T
release alpha
complexity 2
test_data tor/vector_code/output/cbom/crypto-combined-cbom.json (produced in TCS-013)
version 0.3

15 Validate a web-based interface for running scans and visualizing results TCS-015

Preconditions:

  1. Docker engine running
  2. VS Code with the "Dev Containers" extension.

Setup:

  1. Open the project with Dev Containers.
  2. Activate the virtual environment:
poetry shell
  1. Start the server:
VECTOR_ROOT=/home/vector/vector-project VECTOR_PORT=5992 python3 tor/gui/app.py

Test steps

  1. Observe the persistent navigation sidebar.

  2. Validate VECTOR-Code scan submission. On "New scan", with the VECTOR-Code panel selected (default):

    2.1. Click "Start scan" with Source empty.

    2.2. Insert the test source code project path (/home/vector/vector-project/tests/data/mixed_repo) as Source, leave Application name empty, click "Start scan".

    2.3. Enter https://github.com/pyca/cryptography as Source, Application name crypto-app, click "Start scan".

  3. Validate VECTOR-Network scan submission. Switch to the VECTOR-Network panel:

    3.1. Observe the default Protocol and Port values.

    3.2. Switch Protocol to SSH, then switch back to TLS.

    3.3. Click "Start scan" with Target empty.

    3.4. Enter a Target, set Port to 0, click "Start scan".

    3.5. Enter Target github.com, Port 443, Protocol TLS, click "Start scan".

  4. Validate scan progress monitoring:

    4.1. Observe the detail page of the scan from step 3 while it is running.

    4.2. While the scan above is running, submit a new VECTOR-Network scan against Target 10.255.255.1, Port 443, Protocol TLS (this target is expected to fail).

  5. Validate failure display. Using the unreachable-target scan from step 4:

    5.1. Wait for the scan to reach a finished state and examine the detail page.

    5.2. For SSH protocol, insert githb.com as Source, click "Start scan".

    5.3. In VECTOR-Code, insert /tmp/test as Source, click "Start scan".

  6. Validate scan history:

    6.1. Open "Scan history".

  7. Validate results:

    7.1. Open Results for the completed VECTOR-Network scan from step 3, on the Risk report tab.

    7.2. Click "Download report".

    7.3. Switch to the "CBOM explorer" tab.

    7.4. Use the classification filter to isolate a single value (e.g. "Quantum vulnerable"). Reset the filter to "All".

    7.5. Click "Download CBOM".

    7.6. Switch to the "Raw output" tab for the VECTOR-Network scan from step 3.

    7.7. Open the same tab for the completed VECTOR-Code scan from step 2.

  8. Review steps 2-7 for any unexpected server errors.

Expected outcomes

  1. The navigation sidebar shows exactly two items: New scan, Scan history.

  2. VECTOR-Code scan submission:

    2.1. The inline error "Source path or GitHub URL is required." is shown and no scan is created.

    2.2. The scan with the path is created and the browser navigates to its scan detail page automatically.

    2.3. The scan with the Github project is created and the browser navigates to its scan detail page automatically. The application name crypto-app is shown in the scan detail page (scan monitoring page) as entered.

  3. VECTOR-Network scan submission:

    3.1. Protocol defaults to TLS and Port defaults to 443.

    3.2. Port updates to 22 automatically when SSH is selected, and returns to 443 when switching back to TLS.

    3.3. The inline error "Target is required" is shown.

    3.4. The inline error "Port must be between 1 and 65535" is shown.

    3.5. A new scan is created and the browser navigates automatically to a scan details page which shows the real-time CLI output.

  4. Scan progress monitoring:

    4.1. The scan's status and output update on the page without a full page reload while the scan is running.

    4.2. The new submission is accepted and both scans proceed independently, and the interface stays responsive throughout. This target is expected to fail, see step 5.

  5. Failure display:

    5.1. The detail page displays a specific error message describing the failure: Error: testssl.sh exited with code 246 - target likely unreachable, refused the connection, or DNS resolution failed

    5.2. The detail page displays a specific error message describing the failure: Error: CBOM generation failed. Please check the endpoint target information is correct and the network is available

    5.3. The detail page displays a specific error message describing the failure: Error: Path '/tmp/test' does not exist

  6. Scan history:

    6.1.1. The table includes: scan type, target, application name, submission timestamp, duration and status.

    6.1.2. Every row, including the failed scan from step 5, links to its scan detail page.

    6.1.3. Only completed (status "done") rows passes to a Results page; the failed scan from step 5 does not.

  7. Results:

    7.1. The Risk report tab is shown by default and a visual summary of findings is displayed at the top of the tab. The rendered risk report is shown as formatted HTML, not as raw Markdown text.

    7.2. A .md file downloads and its content matches what is shown in the tab.

    7.3. The CBOM is presented as distinct, structured components, one per algorithm found. Each component shows: variant name, primitive type, and a colour-coded risk classification badge.

    7.4. Only matching components remain visible when filtered by classification; resetting to "All" shows all components again.

    7.5. A .json file downloads containing the full annotated CBOM.

    7.6. The raw scan output for the VECTOR-Network scan is shown as collapsible JSON.

    7.7. Check that 7.1-7.6 are validated for VECTOR-Code scan results.

  8. No step at any point displays a raw Python stack trace in the browser.

Parent links: SRS-017 Provide a web-based interface for running scans and visualizing results

Child links: TRP-015 Test report for the VECTOR web interface

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method T
release alpha
complexity 2
test_data see referenced files or controlled scan targets
version 0.4

16 Verify standalone container deployment TCS-016

Preconditions

  1. Docker engine running.

Setup

  1. In the project root directory, build the standalone image from the CLI:
docker build -t vector -f Dockerfile .
  1. Create a results folder in the project root directory:
mkdir results

Test steps

  1. Run the VECTOR-Code scanner via a standard docker run invocation, mounting the test repository as source and the local results directory over the CLI's output path:
docker run --rm \
  --user "$(id -u):$(id -g)" \
  -v $(pwd)/tests/data/mixed_repo:/src \
  -v $(pwd)/results:/output \
  vector code /src --name my-app --output /output
  1. Run the VECTOR-Network scanner via a standard docker run invocation, mounting the local results directory over the CLI's output path:
docker run --rm \
  --user "$(id -u):$(id -g)" \
  -v $(pwd)/results:/output \
  vector network --target github.com --protocol ssh --port 22 --output /output

docker run --rm \
  --user "$(id -u):$(id -g)" \
  -v $(pwd)/results:/output \
  vector network --target github.com --protocol tls --port 443 --output /output
  1. Run the VECTOR-Score to generate a risk report:
docker run --rm \
  --user "$(id -u):$(id -g)" \
  -v $(pwd)/results:/output \
  vector score /output/github.com_ssh_cbom.json

docker run --rm \
  --user "$(id -u):$(id -g)" \
  -v $(pwd)/results:/output \
  vector score /output/github.com_tls_cbom.json

docker run --rm \
  --user "$(id -u):$(id -g)" \
  -v $(pwd)/results:/output \
  vector score /output/cbom/crypto-combined-cbom.json

Expected outcome

  1. The VECTOR-Code scan finishes successfully:
ls results/
  • The results/ folder on the host contains the following, confirming output was written via the mount:
    • cbom/: containing the crypto-combined-cbom.json file
    • databases/: with folders db-cpp and db-python
    • results/: containing two SARIF files, one for C++ and one for Python
  1. The VECTOR-Network scan finishes successfully and the results/ folder on the host contains the resulted CBOM files for both TLS and SSH protocols:
ls results/ | grep tls
ls results/ | grep ssh
  1. The VECTOR-Score successfully generates the risk report for VECTOR-Code and VECTOR-Network CBOM outputs.
ls results/ | grep risk_report
ls results/cbom | grep risk_report

Parent links: SRS-018 Support standalone container deployment

Child links: TRP-016 Test report standalone container deployment

Attribute Value
platform GNU/Linux workstation
execution_type Manual
verification_method T
release alpha
complexity 2
version 0.4

17 Verify the runner script VECTOR in CLI and GUI mode TCS-017

Preconditions

  1. Docker Engine running
  2. vector.sh is executable:
chmod +x vector.sh
  1. mkdir -p results in the repository root, for CLI-mode output.
  2. If there is a env: ‘bash\r’: No such file or directory error when running ./vector.sh, run this command:
sed -i 's/\r$//' vector.sh

This error is due to CRLF endings, which is not supported in Unix.

Test steps

  1. Run a script help:
./vector.sh help
./vector.sh -h
./vector.sh --help
  1. Verify build behavior:

2.1. Run VECTOR CLI for building the image:

./vector.sh cli

2.2. Re-run the CLI to validate that the image is not re-built:

./vector.sh cli

2.3. Run image re-build:

./vector.sh build

2.4. Verify build failure:

mv Dockerfile Dockerfile.bak
./vector.sh build
mv Dockerfile.bak Dockerfile
  1. Verify GUI mode:

3.1. Run default GUI mode and test VECTOR-Code and VECTOR-Network as in TCS-015 test steps section.

./vector.sh

3.2. Stop the Web GUI with Ctrl+C, and follow these steps:

3.2.1. On the host, clone a test Python project into /tmp and start the runner:

git clone --depth 1 https://github.com/Legrandin/pycryptodome.git /tmp/pycryptodome
./vector.sh gui --port 8080 --source-dir /tmp

3.2.2. In the browser, open http://localhost:8080/new-scan, and insert /mnt/host-home/pycryptodome as Source in the VECTOR Code tab. If it is tested in VM, replace localhost with IP address of VM.

  1. Verify CLI mode:
./vector.sh cli code tests/data/mixed_repo --name mixed-test --output ./results
./vector.sh cli network --protocol tls --target github.com --port 443 --output ./results
./vector.sh cli score ./results/cbom/crypto-combined-cbom.json
./vector.sh cli code https://github.com/kokke/tiny-AES-c --name tiny-aes-url --output ./results
  1. Verify failures:

5.1. Verify non-existent path for VECTOR Code:

./vector.sh cli code ./does-not-exist --name x

5.2. Verify non-existent results folder:

./vector.sh cli code tests/data/mixed_repo --output ./results-missing

5.3. Verify non-existent CBOM file for VECTOR Score:

./vector.sh cli score ./no-such-cbom.json
  1. Confirm that no step above 1–5 required installing or invoking Python, Poetry, or any project dependency on the host.

Expected outcome

  1. All help invocations print usage and exit.

2.1. Prints that no local image was found, then builds it, then shows the vector CLI help (code, network, score, gui listed).

2.2. Prints straight to the CLI help with no "Building..." line.

2.3. Always rebuilds the image, regardless of 2.2.

2.4. Fails with a clear message and does not attempt to start a container:

ERROR: failed to build: failed to solve: failed to read dockerfile: open Dockerfile: no such file or directory vector.sh: error: Image build failed; not starting a container.

3.1. The tests over VECTOR-Code and VECTOR-Network in GUI passes according to TCS-015 expected outcomes.

3.2.1. The project is cloned as /tmp/pycryptodome and GUI opens in the browser using the link http://localhost:8080/.

3.2.2. The scan is created and the browser navigates automatically to its scan detail page. When the scan finishes successfully, the Results page shows the Risk report tab.

  1. All four commands complete successfully:

4.1. results/cbom/ contains a CBOM from the mixed_repo scan

4.2. results/ contains network scan output

4.3. The score command produces a *_risk_report.md next to the CBOM it was given

4.4. results/cbom/ contains a CBOM from the GitHub URL scan

  1. All three commands exit non-zero with an error naming the specific missing path (source directory, output directory, or CBOM file) and suggesting how to fix it. In each case, no container is started. The error messages should be shown as this:
./vector.sh cli code ./does-not-exist --name x
vector.sh: error: Source directory does not exist: ./does-not-exist
vector.sh: error: Create it first, e.g.: mkdir -p "./does-not-exist"

./vector.sh cli code tests/data/mixed_repo --output ./results-missing
vector.sh: error: Output directory does not exist: ./results-missing
vector.sh: error: Create it first, e.g.: mkdir -p "./results-missing"

./vector.sh cli score ./no-such-cbom.json
vector.sh: error: CBOM file does not exist: ./no-such-cbom.json
  1. Confirmed that every step above used only docker (via the script) and the script itself.

Parent links: SRS-019 Provide a runner script to build and run VECTOR in CLI or GUI mode

Child links: TRP-017 Test report for the runner script VECTOR in CLI and GUI mode

Attribute Value
platform GNU/Linux workstation or dev container
execution_type Manual
verification_method Test (T)
release alpha
complexity 3
test_data see referenced files or controlled scan targets
version 0.4

18 Verify GitHub repository URL support for VECTOR-Code TCS-018

Preconditions

  1. Docker Engine running.
  2. vector.sh is executable:
chmod +x vector.sh
  1. mkdir -p results in the repository root, for CLI-mode output.
  2. If there is a env: 'bash\r': No such file or directory error when running ./vector.sh, run this command:
sed -i 's/\r$//' vector.sh

This error is due to CRLF endings, which is not supported in Unix. 5. Outbound network access to github.com.

Test steps

  1. Run VECTOR-Code against a public GitHub repository URL:

1.1. With capturing the console output:

./vector.sh cli code https://github.com/pycrypto/pycrypto --output ./results | tee scan.log

1.2. Confirm the cloned source was removed once the CodeQL databases were built:

grep "Removing cloned source" scan.log
  1. Verify a clone failure is handled explicitly:
rm -rf ./results && mkdir -p ./results
./vector.sh cli code https://github.com/vector-tor-integration-test/non_existent_repository_that_should_not_exist --output ./results
echo "exit code: $?"
ls ./results
  1. Verify a non-GitHub-looking path argument is rejected before any container is started:
./vector.sh cli code https://gitlab.com/owner/repo --output ./results
echo "exit code: $?"
  1. Verify the same GitHub URL scan via the Web GUI:

4.1. Start the web interface:

./vector.sh gui

4.2. In the browser, open New Scan, select VECTOR-Code, enter https://github.com/pycrypto/pycrypto as the Source, leave Application name blank, and submit.

Expected outcome

1.1. Console output (scan.log) resembles:

GitHub URL detected, cloning repository
  Cloned:  https://github.com/pycrypto/pycrypto
  Path:    /tmp/vector_code_lyx3l51t
Language detection
  Detected: Python (36.5%)
  Detected: C (26.9%)

Creating CodeQL databases
  Created: db-python
  Created: db-cpp

  Removing cloned source (databases built)

Running crypto queries
  Results generated: crypto-python.sarif
  Results generated: crypto-cpp.sarif

Generating CBOM
  CBOM file generated at: /output/cbom/crypto-combined-cbom.json

Completed successfully

Exit code is 0.

1.2. scan.log contains the line Removing cloned source (databases built).

  1. The command exits with code 1 and prints an explicit error, e.g.:
GitHub URL detected, cloning repository
  Error: git clone failed for 'https://github.com/vector-tor-integration-test/non_existent_repository_that_should_not_exist.git': the repository was not found, or it may be private and require authentication.

./results contains no databases, results, or cbom subdirectories.

  1. vector.sh does not recognize the argument as a GitHub URL (it only matches https://github.com/... or git@github.com:...), so it is validated as a local directory and rejected before Docker is invoked. The command exits with code 1 and prints:
vector.sh: error: Source directory does not exist: https://gitlab.com/owner/repo
vector.sh: error: Create it first, e.g.: mkdir -p "https://gitlab.com/owner/repo"
  1. The scan is created and the browser navigates to its scan detail page; once the scan finishes successfully, the Results page shows CBOM.

Parent links: SRS-016 Accept a GitHub repository URL as a source analysis target

Child links: TRP-018 Test report for GitHub repository URL support for VECTOR-Code

Attribute Value
platform Visual Studio Code
execution_type Manual
verification_method T
release alpha
complexity 3
version 0.4