1.0 ADBox test execution results

1.1 TCER: Deploy ADBox via Docker and shell scripts TRP-001

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape

Test execution results

Test case step 1: Clone the ADBox repository — 0 = flawless

Test case step 2: Change the working directory — 0 = flawless

Test case step 3: Build the image — 0 = flawless

Test case step 4: Run the container via shell script — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-001 Deploy ADBox via Docker and shell scripts

Attribute Value
test-date 2024-07-23
tester AGI
defect-category 0 = flawless
passed-steps 4
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.3 TCER: Install ADBox as dev container TRP-003

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape

Test execution results

The docker build command fails. See reference. Test repeated.

Test case step 4: Open project in dev container in VS Code — 4 = critical defect: dev container build failure — not enough memory

All other steps: 0 = flawless

Defect summary description

Defect category: 4 = critical defect

Dev container build failure — not enough memory (step 4). See referenced screenshot.

Text execution evidence

See tra-003_Dev_container_error.png in assets.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

docs/specs/trp/assets/tra-003_Dev_container_error.png

Parent links: TST-003 Install ADBox as dev container

Attribute Value
test-date 2024-07-24
tester AGI
defect-category 4 = critical defect
passed-steps 5
failed-steps 1
not-executed-steps 0
release-version 0.1
verification-method T

1.4 TCER: Install ADBox as dev container TRP-004

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape

Test execution results

Container size ~26 GB.

Test case step 6: Run ADBox via its entrypoint — 1 = insignificant defect: entrypoint specification unclear (poetry run adbox required)

All other steps: 0 = flawless

Defect summary description

Defect category: 1 = insignificant defect

Entrypoint specification unclear (change last step: entrypoint poetry run adbox).

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-003 Install ADBox as dev container

Attribute Value
test-date 2024-07-24
tester AGI
defect-category 1 = insignificant defect
passed-steps 5
failed-steps 1
not-executed-steps 0
release-version 0.1
verification-method T

1.5 TCER: Run ADBox console TRP-005

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape

Test execution results

Test case step 1: Run the siem-mtad-gat-container container in interactive mode — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-004 Run ADBox console

Attribute Value
test-date 2024-07-24
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.6 TCER: Run ADBox in default mode with a Wazuh connection TRP-006

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run the adbox container without any parameters — 0 = flawless

Test case step 2: Input y after the choice prompt for running in default mode — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

Suggestion: decrease the number of default epochs.

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-005 Run ADBox in default mode with a Wazuh connection

Attribute Value
test-date 2024-07-25
tester AGI
defect-category 0 = flawless
passed-steps 2
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.7 TCER: Run ADBox in default mode without a Wazuh connection TRP-007

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape

Test execution results

Test case step 1: Run the AD driver in default mode — 0 = flawless

Test case step 2: Verify offline data loading — 2 = minor defect: offline data not available due to filename mismatch

Output observed:

Running AD driver in default mode.
JSON file 'detector_input_parameters.json' saved at .../detector_models/.../input/detector_input_parameters.json.
Could not establish a connection with OpenSearch.
The file '.../assets/data/train/wazuh-alerts-*.*-2024.07.*.json' does not exist, returning all default data.

Problem with file names:

wazuh-alerts-.-2024.03..json  wazuh-alerts-.-2024.05..json  wazuh-alerts-.-2024.07..json
wazuh-alerts-.-2024.04..json  wazuh-alerts-.-2024.06..json

Defect summary description

Defect category: 2 = minor defect

Offline data not available (filename mismatch).

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-006 Run ADBox in default mode without a Wazuh connection

Attribute Value
test-date 2024-07-25
tester AGI
defect-category 2 = minor defect
passed-steps 1
failed-steps 1
not-executed-steps 0
release-version 0.1
verification-method T

1.8 TCER: ADBox use case 1 with a Wazuh connection TRP-008

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 1 — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-007 ADBox use case 1 with a Wazuh connection

Attribute Value
test-date 2024-07-25
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.9 TCER: ADBox use case 1 without a Wazuh connection TRP-009

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape

Test execution results

Test case step 1: Run adbox with use case scenario 1 (no Wazuh connection) — 2 = minor defect: offline data not available due to filename mismatch

Output observed:

Running AD driver in default mode.
JSON file 'detector_input_parameters.json' saved at .../detector_models/.../input/detector_input_parameters.json.
Could not establish a connection with OpenSearch.
The file '.../assets/data/train/wazuh-alerts-*.*-2024.07.*.json' does not exist, returning all default data.

Problem with file names:

wazuh-alerts-.-2024.03..json  wazuh-alerts-.-2024.05..json  wazuh-alerts-.-2024.07..json
wazuh-alerts-.-2024.04..json  wazuh-alerts-.-2024.06..json

Defect summary description

Defect category: 2 = minor defect

Offline data not available (filename mismatch).

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-008 ADBox use case 1 without a Wazuh connection

Attribute Value
test-date 2024-07-25
tester AGI
defect-category 2 = minor defect
passed-steps 0
failed-steps 1
not-executed-steps 0
release-version 0.1
verification-method T

1.10 TCER: ADBox use case 2 with a Wazuh connection TRP-010

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 2 — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-009 ADBox use case 2 with a Wazuh connection

Attribute Value
test-date 2024-07-25
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.11 TCER: ADBox use case 3 with a Wazuh connection TRP-011

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 3 — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-011 ADBox use case 3 with a Wazuh connection

Attribute Value
test-date 2024-07-25
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.12 TCER: ADBox use case 4 with a Wazuh connection TRP-012

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 4 — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-013 ADBox use case 4 with a Wazuh connection

Attribute Value
test-date 2024-07-25
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.13 TCER: ADBox use case 5 with a Wazuh connection TRP-013

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 5 — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-015 ADBox use case 5 with a Wazuh connection

Attribute Value
test-date 2024-07-25
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.14 TCER: ADBox use case 5 without a Wazuh connection TRP-014

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 5 (no Wazuh connection) — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Suggestions

For prediction the response should be null if the data are not available.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-016 ADBox use case 5 without a Wazuh connection

Attribute Value
test-date 2024-07-26
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.15 TCER: ADBox use case 4 without a Wazuh connection TRP-015

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 4 (no Wazuh connection) — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Suggestions

For prediction the response should be null if the data are not available.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-014 ADBox use case 4 without a Wazuh connection

Attribute Value
test-date 2024-07-26
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.16 TCER: ADBox use case 3 without a Wazuh connection TRP-016

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 3 (no Wazuh connection) — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-012 ADBox use case 3 without a Wazuh connection

Attribute Value
test-date 2024-07-26
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.17 TCER: ADBox use case 2 without a Wazuh connection TRP-017

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 2 (no Wazuh connection) — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Suggestions

For prediction the response should be null if the data are not available.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-010 ADBox use case 2 without a Wazuh connection

Attribute Value
test-date 2024-07-26
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.18 TCER: ADBox use case 1 without a Wazuh connection TRP-018

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run adbox with use case scenario 1 (no Wazuh connection) — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-008 ADBox use case 1 without a Wazuh connection

Attribute Value
test-date 2024-07-26
tester AGI
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.19 TCER: Run ADBox in default mode without a Wazuh connection TRP-019

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape
  • Wazuh host: 192.168.5.5

Test execution results

Test case step 1: Run the adbox container without any parameters — 0 = flawless

Test case step 2: Input y after the choice prompt for running in default mode — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-006 Run ADBox in default mode without a Wazuh connection

Attribute Value
test-date 2024-07-26
tester AGI
defect-category 0 = flawless
passed-steps 2
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.20 TCER: Install ADBox as dev container TRP-020

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, N/A
  • Hardware deviations: aligned with test case specification, N/A
  • Environment: vm-cyfort-idps-escape

Test execution results

Test case step 1: Clone the repo — 0 = flawless

Test case step 2: Start Docker Desktop — 0 = flawless

Test case step 3: Open project folder in VS Code — 0 = flawless

Test case step 4: Open project in dev container in VS Code — 0 = flawless

Test case step 5: Run poetry install in the container — 0 = flawless

Test case step 6: Run ADBox via its entrypoint — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

None

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-003 Install ADBox as dev container

Attribute Value
test-date 2024-07-29
tester AGI
defect-category 0 = flawless
passed-steps 6
failed-steps 0
not-executed-steps 0
release-version 0.1
verification-method T

1.21 TCER: ADBox shipping install TRP-022

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.

Test case step 1: Running the container with the -s flag

  • 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

N/A

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-017 ADBox shipping install

Attribute Value
test-date 2024-12-10
tester AAT
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.3
verification-method T

1.22 TCER: ADBox deployment TRP-023

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Test case step 1: Clone the repo — 0 = flawless

Test case step 2: Change directory — 0 = flawless

Test case step 3: Build the image — 0 = flawless

Test case step 4: Make the container executable and run it — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

N/A

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-001 Deploy ADBox via Docker and shell scripts

Attribute Value
test-date 2025-01-17
tester AAT
defect-category 0 = flawless
passed-steps 4
failed-steps 0
not-executed-steps 0
release-version 0.3
verification-method T

1.23 TCER: ADBox in dev container TRP-024

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Test case step 1: Clone the repo — 0 = flawless

Test case step 2: Start Docker Desktop — 0 = flawless

Test case step 3: Open project folder in VS Code — 0 = flawless

Test case step 4: Open project in dev container in VS Code — 0 = flawless

Test case step 5: Run poetry install in the container — 0 = flawless

Test case step 6: Run ADBox via its entrypoint — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence / Comments — N/A

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-003 Install ADBox as dev container

Attribute Value
test-date 2025-01-17
tester AAT
defect-category 0 = flawless
passed-steps 6
failed-steps 0
not-executed-steps 0
release-version 0.3
verification-method T

1.24 TCER: ADBox console TRP-025

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Test case step 1: Run the siem-mtad-gat-container container in interactive mode — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence / Comments — N/A

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-004 Run ADBox console

Attribute Value
test-date 2025-01-17
tester AAT
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.3
verification-method T

1.25 TCER: ADBox in default mode with Wazuh TRP-026

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Test case step 1: Run the adbox container without any parameters — 0 = flawless

Test case step 2: Input y after the choice prompt for running in default mode — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence / Comments — N/A

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-005 Run ADBox in default mode with a Wazuh connection

Attribute Value
test-date 2025-01-22
tester AAT
defect-category 0 = flawless
passed-steps 2
failed-steps 0
not-executed-steps 0
release-version 0.3
verification-method T

1.26 TCER: ADBox UC scenario 2 with Wazuh TRP-027

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Test case step 1: Run adbox with use case scenario number 2 — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence / Comments — N/A

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-009 ADBox use case 2 with a Wazuh connection

Attribute Value
test-date 2025-01-23
tester AAT
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.3
verification-method T

1.27 TCER: ADBox UC scenario 3 with Wazuh TRP-028

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Test case step 1: Run adbox with use case scenario number 3 — 0 = flawless

Defect summary description

Defect-free test execution, i.e., defect category: 0 = flawless

Text execution evidence / Comments — N/A

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-011 ADBox use case 3 with a Wazuh connection

Attribute Value
test-date 2025-01-23
tester AAT
defect-category 0 = flawless
passed-steps 1
failed-steps 0
not-executed-steps 0
release-version 0.3
verification-method T

2.0 RADAR test case execution results

2.1 TCER: Setup RADAR foundation TRP-030

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.

Test case step 1: Verify working directory availability

  • 0 = flawless: radar directory available.

Test case step 2: Verify configuration of .env file

  • 0 = flawless: available as indicated in the test case specification.

Test case step 3: Verify Ansible configuration, setup and reachability

  • 0 = flawless: remote agent configuration successful
  • 0 = flawless: remote manager configuration successful
  • 0 = flawless: vault creation and setup for agent and manager successful
  • 0 = flawless: sudo setup for access to controlled node successful
  • 0 = flawless: Ansible reaching nodes in the SSH group successful

Test case step 4: Verify creation and setup of SSL/TLS certificates

  • 0 = flawless: certificates created and stored under config/wazuh_indexer_ssl_certs/

Defect summary description

0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

Any additional informative details not fitting in the above sections.

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-041 Setup RADAR foundation

Attribute Value
test-date 2026-03-10
tester DMA
defect-category 0 = flawless
passed-steps 4
failed-steps 0
not-executed-steps 0
release-version 0.8
verification-method T

2.2 TCER: Build suspicious login TRP-031

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case. The executed command was as follows:

sudo ./build-radar.sh suspicious_login --agent remote --manager local --manager_exists false

In Wazuh Manager

Test case step 2.1: Custom 0310 SSH decoder deployed

  • 0 = flawless: 0310-ssh.xml exists, and is owned by root:wazuh, with mode 0640.

Test case step 2.2: Custom 0310 SSH decoder contains RADAR fields

  • 0 = flawless: decoder contains the decoders sshd-*-with-radar (e.g., including RADAR fields like ASN, country, geo_velocity).

Test case step 2.3: Default 0310 decoder is excluded in ossec.conf

  • 0 = flawless: a <decoder_exclude>0310-ssh_decoders.xml</decoder_exclude> line appears (ideally once) under <ruleset>.

Test case step 2.4: RADAR rules for suspicious_login are in /var/ossec/etc/rules/

  • 0 = flawless: a3-suspicious-login.xml exists with owner root:wazuh, mode 0640.

Test case step 2.5: Suspicious login rules are bound to active responses

  • 0 = flawless: suspicious login rule ID(s) reference the intended <active-response> triggering radar_ar.py.

Test case step 2.6: Agent is registered with the manager

  • 0 = flawless: an entry exists for each agent.

Test case step 2.7: Agent configurations are in place

  • 0 = flawless: agent configurations for suspicious login exist in the <!-- RADAR: suspicious_login agent_config --> block.

Test case step 2.8: Wazuh Manager is running after restart

  • 0 = flawless: the status output indicates the manager is running with no error messages.

In endpoints running agents

Test case step 2.9: RADAR helper directory and script are installed on the SSH agent host

  • 0 = flawless: /opt/radar exists with mode 0755 and owner root:root; radar-helper.py exists with mode 0750 and owner root:root.

Test case step 2.10: radar-helper.service systemd unit is installed, enabled, and running

  • 0 = flawless: the service is active.

Post-test cleanup

Test case step 3: Check the cleanup after successful validations, the command output should be empty

  • 0 = flawless: The command output is empty.

Defect summary description

0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

Any additional informative details not fitting in the above sections.

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-042 Build suspicious login

Attribute Value
test-date 2026-03-10
tester DMA
defect-category 0 = flawless
passed-steps 11
failed-steps 0
not-executed-steps 0
release-version 0.8
verification-method T

2.3 TCER: Build non-whitelist GeoIP detection TRP-032

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case. The executed command was as follows:

./build-radar.sh geoip_detection --agent remote/local --manager remote/local --manager_exists true/false

In Wazuh Manager

Test case step 2.1: Whitelist file exists

  • 0 = flawless: /var/ossec/etc/lists/whitelist_countries exists, its contents match the scenario's whitelist_countries, and its owner is root:wazuh, with mode 0664.

Test case step 2.2: ossec.conf has <list> entry for whitelist_countries

  • 0 = flawless: under <ruleset>, there is a <list>...whitelist_countries</list> entry with the RADAR marker for geoip_detection.

Test case step 2.3: Custom 0310 SSH decoder deployed

  • 0 = flawless: 0310-ssh.xml exists, and is owned by root:wazuh, mode 0640.

Test case step 2.4: Default 0310 decoder is excluded in ossec.conf

  • 0 = flawless: a <decoder_exclude>0310-ssh_decoders.xml</decoder_exclude> line appears (ideally once) under <ruleset>.

Test case step 2.5: RADAR rules for geoip_detection exist

  • 0 = flawless: a2-geoip-detection.xml contains a RADAR_RULES: geoip_detection block; rules use whitelist_countries to detect non-whitelisted countries.

Test case step 2.6: GeoIP non-whitelist rules are bound to active responses

  • 0 = flawless: GeoIP non-whitelist rule ID(s) reference the correct <active-response>.

Test case step 2.7: Active response config files are in place

  • 0 = flawless: ar.yaml and radar_ar.py present in /var/ossec/active-response/bin/.

Test case step 2.8: Agent configurations are in place

  • 0 = flawless: agent configurations for geoip detection exist.

In endpoints running agents

Test case step 2.9: RADAR helper directory and script are installed on the SSH agent host

  • 0 = flawless: /opt/radar exists with mode 0755 and owner root:root; radar-helper.py exists with mode 0750 and owner root:root.

Test case step 2.10: radar-helper.service systemd unit is installed, enabled, and running

  • 0 = flawless: the service is active.

Post-test cleanup

Test case step 3: Check the cleanup after successful validations, the command output should be empty

  • 0 = flawless: The command output is empty.

Defect summary description

0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

Any additional informative details not fitting in the above sections.

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-043 Build non-whitelist GeoIP detection

Attribute Value
test-date 2026-03-10
tester DMA
defect-category 0 = flawless
passed-steps 11
failed-steps 0
not-executed-steps 0
release-version 0.8
verification-method T

2.4 TCER: Build log volume abnormal growth TRP-033

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case. The executed command was as follows:

./build-radar.sh log_volume --agent remote/local --manager remote/local --manager_exists true/false

In Wazuh Manager

Test case step 2.1: RADAR snippet for log_volume is present in ossec.conf

  • 0 = flawless: ossec.conf contains a RADAR: log_volume BEGIN/END block with the log volume-related settings.

Test case step 2.2: RADAR decoders for log_volume exist

  • 0 = flawless: 0001-log-volume.xml contains a block defining decoders that produce a log_bytes field.

Test case step 2.3: RADAR rules for log_volume exist

  • 0 = flawless: a1-log-volume.xml exists in /var/ossec/etc/rules.

Test case step 2.4: OpenSearch template radar-log-volume is present

  • 0 = flawless: HTTP status is 200 and the template JSON exists with mappings for log_bytes as a numeric type.

Test case step 2.5: log_volume rules are bound to active responses

  • 0 = flawless: log volume anomaly rule ID(s) reference radar_ar.py as intended.

Test case step 2.6: Agent configurations are in place

  • 0 = flawless: agent configurations for log volume exist.

In endpoints running agents

Test case step 2.7: RADAR helper directory and script are installed on the SSH agent host

  • 0 = flawless: /opt/radar exists with mode 0755 and owner root:root; radar-helper.py exists with mode 0750 and owner root:root.

Test case step 2.8: radar-helper.service systemd unit is installed, enabled, and running

  • 0 = flawless: the service is active.

Post-test cleanup

Test case step 3: Check the cleanup after successful validations, the command output should be empty

  • 0 = flawless: The command output is empty.

Defect summary description

0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

Any additional informative details not fitting in the above sections.

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-044 Build log volume abnormal growth

Attribute Value
test-date 2026-03-10
tester DMA
defect-category 0 = flawless
passed-steps 9
failed-steps 0
not-executed-steps 0
release-version 0.8
verification-method T

2.5 TCER: Run RADAR for log volume abnormal growth TRP-034

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case. The executed command was as follows:

./run-radar.sh log_volume

In Wazuh Dashboard

Test case step 2.1: The detector LOG_VOLUME_DETECTOR is created in the Wazuh Dashboard with the correct configurations

  • 0 = flawless: In the Anomaly Detection menu, the created detector is visible. The expected configurations are found in config.yaml under the log_volume key.

Test case step 2.2: Monitor creation and trigger to webhook existence

  • 0 = flawless: the monitor is created in the Wazuh Dashboard with a trigger to a RADAR webhook instance. In the Alerting menu, the created Monitor is listed under the Monitors tab.

Defect summary description

0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

Any additional informative details not fitting in the above sections.

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-045 Run RADAR for log volume abnormal growth

Attribute Value
test-date 2026-03-10
tester DMA
defect-category 0 = flawless
passed-steps 2
failed-steps 0
not-executed-steps 0
release-version 0.8
verification-method T

2.6 TCER: DECIPHER-RADAR detection validation for suspicious login TRP-035

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case. The executed command was as follows:

sudo ./simulate-radar.sh suspicious_login --agent remote

Test case step 2.1: The ansible playbook executes successfully

  • 0 = flawless: The simulation was successfully executed.

Test case step 2.2: In the Wazuh Dashboard Discovery page, at least one of rule IDs 210012, 210013, 210020, or 210021 is triggered for the remote agent

  • 0 = flawless: Alerts with one of the rule.id : [210012, 210013, 210020, or 210021] appear in the Wazuh Dashboard.

Test case step 2.3: In Wazuh manager, alert tier verified via grep "Risk computed" /var/ossec/logs/active-responses.log | tail -1

  • Tier 0:
    • 0 = flawless: no email received at EMAIL_TO.
    • 0 = flawless: no FlowIntel case created.
    • 0 = flawless: no mitigation command in /var/ossec/logs/active-responses.log.
  • Tier 1: (Same alerts were generated, but by putting one of the IPs present in the alert in MISP changed the tier from 0 to 1)
    • 0 = flawless: email received at EMAIL_TO with risk score, tier, and FlowIntel case URL.
    • 0 = flawless: FlowIntel case created and accessible at the URL included in the email.
    • 0 = flawless: FlowIntel case contains relevant information about the suspicious login event, related events found in MISP, and a breakdown of the risk score including the assigned tier.
    • 0 = flawless: misp_available: true appears in the FlowIntel case description, confirmed via "DECIPHER analyze completed" log entry matched by decision_id.
    • 0 = flawless: FlowIntel case is tagged with low priority.
    • 0 = flawless: no mitigation command in /var/ossec/logs/active-responses.log.

Defect summary description

0 = flawless

Text execution evidence

FlowIntel case:

Email:

Comments

Any additional informative details not fitting in the above sections.

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-046 DECIPHER-RADAR detection validation for Suspicious login

Attribute Value
test-date 2026-03-10
tester DMA
defect-category 0 = flawless
passed-steps 3
failed-steps 0
not-executed-steps 0
release-version 0.8
verification-method T

2.7 TCER: Detection validation for GeoIP detection TRP-036

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case. The executed command was as follows:

sudo ./simulate-radar.sh geoip_detection --agent remote

Test case step 2.1: The ansible playbook executes successfully

  • 0 = flawless: The ansible playbook executed successfully.

Test case step 2.2: In the Wazuh Dashboard Discovery page, a rule with ID matching 10090* is triggered for the remote agent

  • 0 = flawless: A rule with rule.id '100900' was triggered and appeared in Wazuh Dashboard

Test case step 2.3: In Wazuh manager, alert tier verified via grep "Risk computed" /var/ossec/logs/active-responses.log | tail -1

  • Tier 1:

    • 0 = flawless: email received at EMAIL_TO with risk score, tier.
    • 0 = flawless: no mitigation command in /var/ossec/logs/active-responses.log.
  • Tier 3:

    • 0 = flawless: email received at EMAIL_TO with risk score, and tier.
    • 0 = flawless: mitigation commands from mitigations_tier3 for geoip_detection appear in /var/ossec/logs/active-responses.log.
    • 0 = flawless: firewall-drop: IP address blocked in the agent.

Defect summary description

0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

Any additional informative details not fitting in the above sections.

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-047 Detection validation for GeoIP detection

Attribute Value
test-date 2026-03-10
tester DMA
defect-category 0 = flawless
passed-steps 3
failed-steps 0
not-executed-steps 0
release-version 0.8
verification-method T

2.8 TCER: Detection validation for log volume abnormal growth TRP-037

Relevant test environment and configuration details

  • Software deviations: aligned with test case specification, with controller node running Ubuntu 22.04.5 LTS (Jammy Jellyfish)
  • Hardware deviations: aligned with test case specification (VMs handled via QEMU KVM hypervisor)

Test execution results

Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case. The executed command was as follows:

./simulate-radar.sh log_volume --agent remote

Test case step 2.1: The ansible playbook executes successfully

  • 0 = flawless: The ansible playbook executed successfully.

Test case step 2.2: In the Wazuh Dashboard Discovery page, rule ID 100309 is triggered for the remote agent

  • 0 = flawless: An alert with rule.id 100309 has been triggered and appears in the Wazuh dashboard.

Test case step 2.3: Spike has happened in the agent

  • 0 = flawless: /var/log size increase confirmed.

Test case step 2.4: In Wazuh manager, alert tier verified via grep "Risk computed" /var/ossec/logs/active-responses.log | tail -1

  • Tier 2:
    • 0 = flawless: email received at EMAIL_TO with risk score, tier, and FlowIntel case URL.
    • 0 = flawless: mitigation commands from mitigations_tier2 for log_volume appear in /var/ossec/logs/active-responses.log.

Test case step 2.5: Spike file removed after cleanup_minutes elapsed (or manually if cleanup_minutes is 0)

  • 0 = flawless: Spike file removed after cleanup_minutes elapsed

Defect summary description

0 = flawless

Text execution evidence

See linked files (if any), e.g., screenshots, logs, etc.

Comments

Any additional informative details not fitting in the above sections.

Guide

  • Defect category: 0 = flawless; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
  • Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)

Parent links: TST-048 Detection validation for Log volume abnormal growth

Attribute Value
test-date 2026-03-11
tester DMA
defect-category 0 = flawless
passed-steps 5
failed-steps 0
not-executed-steps 0
release-version 0.8
verification-method T