1 TCER: modelling TRA-001
This test case execution result (TCER) reports the outcome of verifying modelling artifacts.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify linked requirements
- 0 = flawlesss: The data model of SATRAP-DL uses a data modelling language based on type theory, namely TypeQL.
- 0 = flawlesss: SATRAP-DL relies on a database paradigm that allows for knowledge representation based on semantics and PERA model implemented by TypeDB.
- 0 = flawlesss: SATRAP-DL supports querying the CTI SKB based on semantic criteria.
- 0 = flawlesss: The data model of the CTI SKB is extensible and allows for the integration of new information.
- 0 = flawlesss: The data model of the CTI SKB SHALL relies on a type-theoretic polymorphic entity-relation-attribute (PERA) data model to allow for the addition of new entities and relationships without requiring a schema migration.
Test case step 2: Check for alignment between system concept and implemented system
- 0 = flawlesss: alignment confirmed upon reviewing design artifacts and comparing these against the implementation.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
- N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TST-001 TC: Verify data modelling artifacts
Attribute | Value |
---|---|
test-date | 2025-03-25 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 2 |
release-version | 0.1 |
verification-method | R |
2 TCER: SW engineering TRA-002
This test case execution result (TCER) reports the outcome of the verification of naming convention usage and adherence to the SOLID software engineering principles.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify consistent naming convention use
- 1 = insignificant defect: based on a sample of the source files, most functions, classes and variables follow the PEP-8 naming convention consistently. Nevertheless, we did identify one problematic instance in the
log_utils.py
module, see the comments section below for more details.
Test case step 2: Verify adherence to SOLID
- 0 = flawlesss: the 5 SOLID design principles are largely respected by the architectural modules.
Defect summary description
An insignificant defect was detected during test execution, i.e., thus assigning the overall highest defect category from the test step verdicts: 1 = insignificant defect
Please see the comments below for a few relevant observations.
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
The function definition testing(self, ...)
in satrap/commons/log_utils.py
: a function at the module level has a self
parameter in its signature, as opposed to being used in instance methods within class definitions to refer to the instance of the class; see linked file for exact reference.
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect>
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
SOLID principles: Single responsibility principle (SRP), Open-closed principle (OCP), Liskov substitution principle (LSP), Interface integration principle (ISP), Dependency inversion principle (DIP).
satrap/commons/log_utils.py
(line 69)
Parent links: TST-002 TC: Verify software engineering
Attribute | Value |
---|---|
test-date | 2025-03-25 |
tester | AAT |
defect-category | 1 = insignificant defect |
passed-steps | 2 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | R |
3 TCER: STIX and reasoning TRA-003
This test case execution result (TCER) reports the outcome of STIX and reasoning engine usage verification.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify DBMS technology
- 0 = flawlesss: SATRAP-DL uses a DBMS technology that comes with a reasoning engine as a key integral part, namely TypeDB.
Test case step 2: Verify use of STIX 2.1
- 0 = flawlesss: SATRAP-DL uses STIX 2.1 as the default standard format for CTI representation.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TST-003 TC: Verify STIX and reasoning engine
Attribute | Value |
---|---|
test-date | 2025-03-25 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 2 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | R,I |
4 TCER: data model TRA-004
We analyze the SATRAP data model to verify adherence to that of STIX 2.1.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify data model alignment with STIX 2.1
- 1 = insignificant defect: support for ingesting STIX 2.1 is implemented, providing a direct mapping of the imported data to equivalent concepts in the TypeDB database; however, custom and metadata objects are currently missing.
Defect summary description
Assigned defect category: 1 = insignificant defect
STIX 2.1 is currently not complete (custom properties and meta objects currently not handled), but sufficient coverage is provided for the alpha release.
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TST-004 TC: Verify data model
Attribute | Value |
---|---|
test-date | 2025-03-25 |
tester | AAT |
defect-category | 1 = insignificant defect |
passed-steps | 1 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | A |
5 TCER: centralized management TRA-005
We report on our inspection to verify centralized management of system parameters customization via a dedicated configuration file, and of log storage, exception types and error messages.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
SATRAP-DL enables customization of system parameters via a YAML file located at satrap/assets/satrap_params.yml
, capturing logging mode, TypeDB database parameters (host, port, db name) and ETL default source files/paths.
Test case step 1: Verify centralized system parameterization
- 0 = flawlesss: The user-controlled YAML file
satrap_params.yml
captures logging mode.
Test case step 2: Verify centralized parameterization for database connections
- 0 = flawlesss: The user-controlled YAML file
satrap_params.yml
captures TypeDB database parameters (host, port, db name)
Test case step 3: Verify centralized parameterization file for managing file paths
- 0 = flawlesss: The
settings.py
encapsulates paths to various resources used throughout the code.
Test case step 4: Verify designated logs storage location
- 0 = flawlesss: SATRAP-DL stores its logs under
satrap/assets/logs
, with log files organized under subfolders named by date, which in turn contain timestamped logs files capturing the name of the log producing module.
Test case step 5: Verify centralized exception definitions
- 0 = flawlesss: Exceptions are defined in a centralized manner and stored in
satrap/commons/exceptions.py
.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
satrap/assets/satrap_params.yml
(line 3)satrap/settings.py
satrap/commons/exceptions.py
Parent links: TST-005 TC: Verify centralized management
Attribute | Value |
---|---|
test-date | 2025-03-26 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 5 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | I |
6 TCER: code clarity TRA-006
We report on our code inspections to validate the logging feature of the ETL subsystem.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify source code clarity according to the C5-DEC method and guidelines
- 0 = flawlesss: The majority of the SATRAP-DL code base exhibits a high degree of consistency in terms of understandability, readability and being well-documented.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TST-006 TC: Verify code clarity
Attribute | Value |
---|---|
test-date | 2025-03-26 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 1 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | I |
7 TCER: secure programming TRA-007
This test case execution report addresses the validation test case dealing with secure programming aspects and practices.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify components input validation
- 2 = minor defect: while most SATRAP components receiving input perform some form of validation, we have identified a few discrepancies such as those in the ETL
extractor
module. Thefetch
functions in theDownloader
andSTIXExtractor
do not validate the source URLs or file paths against a reference pattern or inclusion in any black or white lists. They do however perform syntactic checks directly and indirectly via thevalidate_file
function in thefile_utils
module.
Test case step 2: Verify sanitization of input and output of data passing across trust boundaries
- 0 = flawlesss: since integration with TIPs or other external sources residing outside the SATRAP trust boundary is currently not implemented as part of the alpha release (requirements for beta release), this sanitization requirements is considered to be satisfied.
Test case step 3: Verify resource liberation
- 0 = flawlesss: the ETL subsystem and various functions making I/O operations correctly release resources, e.g., database network connections and file streams handled via
with
context managers.
Test case step 4: Verify SBOM usage
- 0 = flawlesss: all software dependencies of SATRAP, thanks to use its of Poetry, are listed precisely in an inventory providing a software bill of material (SBOM), that lists all used libraries, their respective versions, along with the corresponding hashes (automatically generated lock file).
Test case step 5: Verify log string sanitization
- 2 = minor defect: log strings are currently not sanitized and validated before being logging to prevent log injection attacks.
Log injection vulnerabilities can emerge when writing invalidated user input to log files can allow an attacker to forge log entries or inject malicious content into the logs; the data can enter an application from an untrusted source (N/A in the alpha release of SATRAP) or it can be written to an application or system log file. (applicable to the alpha release)
A note on log forging (source: OWASP)
In the most benign case, an attacker may be able to insert false entries into the log file by providing the application with input that includes appropriate characters. If the log file is processed automatically, the attacker can render the file unusable by corrupting the format of the file or injecting unexpected characters. A more subtle attack might involve skewing the log file statistics. Forged or otherwise, corrupted log files can be used to cover an attacker’s tracks or even to implicate another party in the commission of a malicious act.
Test case step 6: Verify secret storage
- 0 = flawlesss: Manual and automated scans (SAST) confirm the absence of logged or hardcoded sensitive information in the source code such as passwords or entity identifiers.
Test case step 7: Verify data semantic integrity enforcement
- 0 = flawlesss: the TypeDB engine, together with the SATRAP data model and TypeQL types (
cti-skb-types.tql
) enforce semantic integrity ensuring that relationships and constraints adhere to the intended meaning. These enable benefitting from measures such as data validation with respect to schemas and relationship constraints, technical possibility of automated checks for data redundancy and inference powered by a reasoning engine.
Defect summary description
Various minor issues have been identified, thus assigning the overall highest defect category from the test step verdicts: 2 = minor defect
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
satrap/etl/extract/extractor.py
(line 51)satrap/etl/extract/extractor.py
(line 88)satrap/commons/file_utils.py
(line 33)satrap/commons/file_utils.py
(line 68)satrap/assets/schema/cti-skb-types.tql
Parent links: TST-007 TC: Verify secure programming
Attribute | Value |
---|---|
test-date | 2025-03-26 |
tester | AAT |
defect-category | 2 = minor defect |
passed-steps | 5 |
failed-steps | 2 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | I,R |
8 TCER: MITRE ATT&CK ingestion TRA-008
This test case execution report covers the validation test case specification on the ingestion of the MITRE ATT&CK data set.
Relevant test environment and configuration details
- Software deviations: aligned with test case specification
- Hardware deviations: aligned with test case specification
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify the execution of the SATRAP setup
CLI command
- 0 = flawlesss: the obtained result aligns with the expected outcome described in the linked validation test case specification.
Test case step 2: Verify the execution of the SATRAP etl
CLI command
- 0 = flawlesss: the obtained result aligns with the expected outcome described in the linked validation test case specification.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TST-008 Test setup + MITRE ATT&CK ingestion
Attribute | Value |
---|---|
test-date | 2025-03-27 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 2 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | T |
9 TCER: ETL architecture TRA-009
This test case execution report covers the ETL subsystem; see the linked files for precise references to the cited code modules and classes mentioned below.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify data integration module for orchestration
- 0 = flawlesss: As part of its dedicated ETL subsystem, SATRAP provides a module in charge of orchestrating (
etlorchestrator.py
), another for the ingestion (extractor.py
) of datasets in STIX 2.1 and, finally, another module for the insertion (loader.py
) of content into the SATRAP CTI SKB, powered by TypeDB.
Test case step 2: Verify the integration module for transforming data into STIX 2.1
- 0 = flawlesss: As part of its dedicated ETL subsystem, SATRAP provides a module in charge of transforming (
transformer.py
) the ingested STIX data into the representation language of the CTI SKB schema, namely TypeQL.
Test case step 3: Verify data integration module for database operations and connections
- 0 = flawlesss: SATRAP provides a dedicated data management package containing various related modules, with one in particular (
typedbmanager.py
) in charge of managing database operations and connections.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
satrap/etl/etlorchestrator.py
(line 11)satrap/etl/extract/extractor.py
satrap/etl/transform/transformer.py
satrap/etl/load/loader.py
satrap/datamanagement/typedb/typedbmanager.py
Parent links: TST-009 Verify ETL architecture
Attribute | Value |
---|---|
test-date | 2025-03-27 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 3 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | R |
10 TCER: CTI SKB inference TRA-010
This test case execution report covers the SATRAP automated reasoning and inference capabilities; see the linked files for precise references to the cited files.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify that SATRAP-DL implements inference rules for automated reasoning
- 0 = flawlesss: The SATRAP implementation provides a dedicated rules file (
cti-skb-rules.tql
). By analyzing the CTI SKB inference rule definition file and CTI SKB type definitions files stored in thesatrap/assets/schema
, we confirm the presence of the required artifacts enabling derivation of knowledge over existing relations in the CTI SKB powered by TypeDB.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
satrap/assets/schema/cti-skb-rules.tql
Parent links: TST-010 Verify CTI SKB inference rules
Attribute | Value |
---|---|
test-date | 2025-03-27 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 1 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | A |
11 TCER: Jupyter Notebook frontend TRA-011
Relevant test environment and configuration details
- Software deviations: aligned with test case specification
- Hardware deviations: aligned with test case specification
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Open the source code of SATRAP-DL in Microsoft VS Code Studio using the Dev Containers extension
- 0 = flawlesss: The project loaded successfully, with a bash session being the entry point selected by default, giving access to the SATRAP GNU/Linux container.
Test case step 2: Navigate to the VS Code Studio terminal and activate Python environment
- 0 = flawlesss: The SATRAP container was successfully accessed, and the commands
poetry shell
andpoetry install
also successfully run.
Test case step 3: Navigate to the folder satrap/frontend/quick_start.ipynb
and activate kernel
- 0 = flawlesss: Jupyter Notebook located successfully and kernel activated.
Test case step 4: Run each cell from top to bottom in order
- 0 = flawlesss: All cells executed, and their outputs were successfully compared to the expected output reference file
quick_start-test-reference.ipynb
.
Part 1 on "Starting with simple functions":
- 0 = flawlesss:
satrap = CTIanalysisToolbox(TYPEDB_SERVER_ADDRESS, DB_NAME)
- 0 = flawlesss:
print(satrap.get_sdo_stats())
- 0 = flawlesss:
print(satrap.mitre_attack_techniques())
- 0 = flawlesss:
print(satrap.mitre_attack_mitigations())
- 0 = flawlesss: cell "Get information on a specific MITRE ATT&CK element (technique, group, software, etc.) using its MITRE ATT&CK id."
- 0 = flawlesss: cell "Get information about a STIX object using its STIX id."
- 0 = flawlesss: cell "Retrieve mitigations explicitly associated to a specific technique using its STIX id."
Part 2 on "CTI analysis through automated reasoning":
- 0 = flawlesss: cell "Get statistics on the usage of ATT&CK techniques by groups. The output of this function is the same as when running the command
satrap techniques
on the CLI." - 0 = flawlesss: cell
display(satrap.techniques_usage(infer=True))
- 0 = flawlesss: cell
techniques = satrap.techniques_used_by_group("G0025", infer=True)
- 0 = flawlesss: cell
display(satrap.related_mitigations(group_name="BlackTech"))
Subsection "Explanation of inferred knowledge"
- 0 = flawlesss: 1st cell starting with
rel_explanation = satrap.explain_if_related_mitigation("G0098", "course-of-action--20a2baeb-98c2-4901-bad7-dc62d0a03dea")
. - 0 = flawlesss: 2nd cell starting with
reason = satrap.explain_related_techniques("ZIRCONIUM", "T1059.006")
. - 0 = flawlesss: last cell starting with
dg_explanation = satrap.explain_techniques_used_by_group("G0071", "Domain Groups")
.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
The function signature t1 = satrap.get_attck_concept_info("T1027.001")
contains a typo: attck
-> attack
Functions are named inconsistently: there is a mix of verb-based (preferred as per C5-DEC conventions) and noun-based naming, e.g.,
satrap.get_sdo_stats()
satrap.mitre_attack_techniques()
satrap.mitigations_for_technique()
satrap.get_attck_concept_info()
satrap.search_stix_object()
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
docs/specs/TST/assets/quick_start-test-reference.ipynb
Parent links: TST-011 Test Jupyter notebook frontend
Attribute | Value |
---|---|
test-date | 2025-03-27 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 4 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | T |
12 TCER: ETL logging TRA-012
We ran tests to validate the logging feature of the ETL subsystem.
Relevant test environment and configuration details
- Software deviations: aligned with test case specification
- Hardware deviations: aligned with test case specification
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify ETL logging by running ./satrap.sh etl
to ingest the latest version of the MITRE ATT&CK Enterprise data set
0 = flawlesss: The generated log, i.e., test-evidence-log.txt
, stored in the satrap/assets/logs
folder was checked (stored under date folders and timestamped files according to ETL execution time) and the following items were validated:
- a log entry is generated at the beginning of the job, indicating the start time
- each log entry recording an event comes with a log level: DEBUG, INFO, WARNING, ERROR, CRITICAL
- there is at least a log entry for each ETL phase (i.e., extraction, transformation and loading) describing the status in terms of success/failure and some minimal hint or additional information explaining or complementing the execution status.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
docs/specs/TRA/assets/test-evidence-log.txt
Parent links: TST-012 Test ETL logging
Attribute | Value |
---|---|
test-date | 2025-03-29 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 1 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | T |
13 TCER: CM settings TRA-013
We report on our inspection of the centralized settings file to verify that some of its content is read from a user-controller configuration management file allowing configuration management without requiring software rebuild.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Navigate to the settings.py
- 0 = flawlesss: The file
settings.py
was located atsatrap/settings.py
.
Test case step 2: Navigate to the YAML configuration management
- 0 = flawlesss: The configuration management file
satrap_params.yml
exposed to the user was found atsatrap/assets/satrap_params.yml
.
Test case step 3: Check that the YAML file is read into memory (e.g., in a Python dictionary).
- 0 = flawlesss: this was validated the
settings.py
fileread_yaml(SATRAP_PARAMS_FILE_PATH)
; see linked files section for the precise, automatically retrieved line number.
Test case step 4: Check that at least one settings parameter read into memory
- 0 = flawlesss: parameters in
settings.py
are populated from the in-memory copy of thesatrap_params.yml
file, with an example given below
HOST = satrap_params_dict.get('typedb').get('host','typedb')
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
satrap/settings.py
(line 34)
Parent links: TST-013 Inspect settings for CM
Attribute | Value |
---|---|
test-date | 2025-03-28 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 4 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | I |
14 TCER: SATRAP CLI TRA-014
We report on our tests carried out using the SATRAP command line interface (CLI) to verify that it provides at least the commands specified in the software requirement specification (SRS) that the linked test case specification traces to.
Relevant test environment and configuration details
- Software deviations: aligned with test case specification
- Hardware deviations: aligned with test case specification
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Run ./satrap.sh rules
- 0 = flawlesss: obtained result consistent with the expected outcome specified in the linked test case.
Test case step 2: Run ./satrap.sh stats
- 0 = flawlesss: obtained result consistent with the expected outcome specified in the linked test case.
Test case step 3: Run ./satrap.sh techniques
- 0 = flawlesss: obtained result consistent with the expected outcome specified in the linked test case.
Test case step 4: Run ./satrap.sh mitigations
- 0 = flawlesss: obtained result consistent with the expected outcome specified in the linked test case.
Test case step 5: Run ./satrap.sh search campaign--0c259854-4044-4f6c-ac49-118d484b3e3b
- 0 = flawlesss: obtained result consistent with the expected outcome specified in the linked test case.
Test case step 6: Run ./satrap.sh info_mitre T1027.001
- 0 = flawlesss: obtained result consistent with the expected outcome specified in the linked test case.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TST-014 Test command line interface (CLI)
Attribute | Value |
---|---|
test-date | 2025-03-28 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 6 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | T |
15 TCER: Open-source TIP adoption TRA-015
We reviewed the SATRAP design artifacts to validate its adoption of open-source TIPs by design, specifically MISP and OpenCTI.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify SATRAP adoption of open-source TIPs
- 0 = flawlesss: we confirmed upon reviewing the SATRAP system concept documents that it adopts both MISP and OpenCTI as TIPs.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TST-017 Verify open-source TIP integration
Attribute | Value |
---|---|
test-date | 2025-03-28 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 1 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | R |
16 TCER: release + licensing TRA-016
We report on our inspection of SATRAP release and licensing model.
Relevant test environment and configuration details
- Software deviations: N/A
- Hardware deviations: N/A
Test execution results
Here we report the results in terms of step-wise alignments or deviations with respect to the expected outcome of the covered test case.
Test case step 1: Verify public open-source release of SATRAP source code
- 0 = flawlesss: The entire source code of SATRAP was confirmed to be released on GitHub via a public repository, as per project agreements.
Test case step 2: Verify licenses of 3rd-party libraries
- 0 = flawlesss: SATRAP software library dependencies do not restrict the privileges granted by the license selected for SATRAP-DL.
Defect summary description
Defect-free test execution, i.e., defect category: 0 = flawlesss
Text execution evidence
See linked files (if any), e.g., screenshots, logs, etc.
Comments
N/A
Guide
- Defect category: 0 = flawlesss; 1 = insignificant defect; 2 = minor defect; 3 = major defect; 4 = critical defect
- Verification method (VM): Test (T), Review of design (R), Inspection (I), Analysis (A)
Parent links: TST-018 Verify release and licensing
Attribute | Value |
---|---|
test-date | 2025-03-28 |
tester | AAT |
defect-category | 0 = flawless |
passed-steps | 2 |
failed-steps | 0 |
not-executed-steps | 0 |
release-version | 0.1 |
verification-method | I |