idps-escape

Malware Communication Detection

Part 1: Objectives

Example Scenario: An internal host infected with malware begins making periodic outbound connections to suspicious external IP addresses over uncommon ports, such as 53, 8081, or 1337. These communications follow a beacon-like pattern, often involve low-volume payloads, and occur at regular intervals (e.g., every 60 seconds). These are typical indicators of command-and-control (C2) activity, where the host checks in with an attacker-controlled server for further instructions.

Categorical features: To improve detection accuracy, anomalies are categorized by source IP address (id.orig_h). This ensures that the anomaly model learns a distinct baseline for each internal host. Without this separation, the communication behavior of one host might obscure that of another, masking malicious activity. By assigning a categorical field per host, the detector can catch host-specific deviations, like one system suddenly reaching out to rare IPs or ports.

Part 2: Data preparation and ingestion

2.1 Dataset ingestion

To simulate today’s data, each event’s timestamp is shifted to the current date while preserving its original time. This is done via the ingestion script malware_communication/wazuh_ingest.py, located in the soar-radar folder:

python3 malware_communication/wazuh_ingest.py

• What it does:
  • Shifts the timestamp_readable field to today
  • Computes ISO @timestamp, event_hour, and normalizes numeric fields
  • Bulk indexes data into OpenSearch in daily indices like wazuh-ad-malware-c2-2025.07.17

2.2 Index Pattern & Wazuh Integration

1. In Dashboards Management, create an Index Pattern for wazuh-ad-malware-c2-*.
2. Confirm data visibility in Discover with fields:
   • @timestamp, id.orig_h, id.resp_h, service, proto, orig_bytes, resp_bytes

Part 3: Detector & Feature Configuration

3.1 Create the Anomaly Detector

1. Go to Wazuh Dashboards ➔ OpenSearch Plugins ➔ Anomaly Detection.
2. Click Create detector and fill:
   • Name: malware-c2-detector
   • Description: “Detects host-based outbound C2 activity”
   • Index: wazuh-ad-malware-c2-*
   • Time field: @timestamp
   • Interval: 5m with 1m delay
   • Type: Real-time detection
   • Custom result index: opensearch-ad-plugin-result-malware_communication

3.2 Define Numeric Features

Feature name	Method	Field	Notes
conn_count	count()	uid	Counts outbound connections per host
bytes_sent	sum()	orig_bytes	Total outbound payload volume
bytes_received	sum()	resp_bytes	Total inbound response volume
unique_dst_ips	cardinality()	id.resp_h	Number of distinct contacted IPs
unique_dst_ports	cardinality()	id.resp_p	Number of destination ports used

3.2.1 Workaround for Cardinality

The OpenSearch UI does not support cardinality() directly, which is why we use a custom expression:

 "unique_dst_ips": {      
		"cardinality": {      
			"field": "id.resp_h.keyword"     
		}    
}

 "unique_dst_ports": {      
		"cardinality": {      
			"field": "id.resp_h.keyword"     
		}    
}

3.3 Enable Categorical Field

Under Categorical field, select id.orig_h. This ensures that each host has its own model. Anomalous behavior is then evaluated relative to each host’s typical communication behavior.

3.4 Review & Save

Click Next, validate the features, and confirm the model starts without errors. Once created, the model will begin analyzing live data.

Part 4: Monitor, Webhook & Wazuh Rule Integration

4.1 Create an OpenSearch Monitor

1. From the malware-c2-detector view, click Create monitor.
2. Set:
   • Trigger name: Malware-Communication-Detected
   • Severity: High

   • Condition:

       anomaly_grade ≥ 0.8 AND confidence ≥ 0.85
     

3. Create a Notification Channel:
   • Name: RADAR
   • Type: Webhook (POST)
   • URL: http://192.168.0.28:8888/opensearch-alert

4. Action Payload:

    {
      "monitor": {"name": ""},
      "trigger": {"name": ""},
      "entity": "",
      "periodStart": "",
      "periodEnd":   ""
    }
   

4.2 Webhook & Decoder

1. Deploy ad_alerts_webhook.py in the Wazuh manager
2. Log output to /var/log/ad_alerts.log

3. Add this to ossec.conf:

    <localfile>
        <log_format>syslog</log_format>
        <location>/var/log/ad_alerts.log</location>
    </localfile>
   

4. Add decoders and rules for rule ID 100304 to trigger enrichment

Part 5: Active Response Actions

5.1 Manager-side Active response

• Receives id.orig_h (internal host IP), period_start, period_end
• Queries OpenSearch for all events from that host in the time window
• Groups them by destination IP (id.resp_h)
• For each destination IP:
  • Calls !firewalldrop.sh <dst_ip>
  • Calls !terminate_c2.sh <dst_ip>
• Once per host:
  • Calls !quarantine_host.sh (blocks all outbound except loopback)

1. In ossec.conf on the manager, register and bind only the ad_context_malcom_active_response.py script. Script can be found in /soar-radar/malware_communication/active_responses/.

<ossec_config>  
<!-- 1) Command declaration -->  
	<command>    
		<name>ad_enrich</name>    
		<executable>ad_context__malcom_active_response.py</executable>    
		<timeout_allowed>yes</timeout_allowed>  
	</command>  
	<!-- 2) Active-response binding -->  
	<active-response>    
		<disabled>no</disabled>    
		<command>ad_enrich</command>    
		<location>server</location>    
		<rules_id>100304</rules_id>    
		<timeout>120</timeout>  
	</active-response>
</ossec_config>

• When Wazuh rule 100304 fires, it will run ad_context_malcom_active_response.py.

1. Place a copy of the script from the active_responses directory to /var/ossec/active-response/bin in the Wazuh manager.

Note: remember to update the Wazuh access credentials (username, password) in the script based on your setup. Here we simply default values from the official “factory settings” of Wazuh.

1. Give permissions for execution:

chmod 750 /var/ossec/active-response/bin/ad_context_active_response.py
chown root:wazuh /var/ossec/active-response/bin/ad_context_active_response.py

1. Install dependencies in the Wazuh manager:

python3 -m pip install requests

5.2 Agent-side active responses

• firewalldrop.sh: blocks outbound connection to C2 IP
• terminate_c2.sh: kills processes with connections to C2 IP
• quarantine_host.sh: drops all outbound connections (isolates host)

These scripts are passed JSON via stdin:

• terminate_c2.sh receives the C2 IP via extra_args[0]
• quarantine_host.sh extracts user_keyword (host IP) from alert.data

5.3 Wazuh Integration

1. Install jq

sudo apt update
sudo apt install -y jq

1. Copy the active response scripts:

sudo cp quarantine_host.sh /var/ossec/active-response/bin/
sudo cp terminate_c2.sh /var/ossec/active-response/bin/
sudo cp write_contextual_logs_malcom_active_response.py /var/ossec/active-response/bin/

1. Set proper permissions:

sudo chown root:wazuh /var/ossec/active-response/bin/quarantine_host.sh
sudo chmod 750 /var/ossec/active-response/bin/quarantine_host.sh

sudo chown root:wazuh /var/ossec/active-response/bin/terminate_c2.sh
sudo chmod 750 /var/ossec/active-response/bin/terminate_c2.sh

sudo chown root:wazuh /var/ossec/active-response/bin/write_contextual_logs_malcom_active_response.py
sudo chmod 750 /var/ossec/active-response/bin/write_contextual_logs_malcom_active_response.py

1. Enable Remote Commands

Edit /var/ossec/etc/local_internal_options.conf

wazuh_command.remote_commands=1

1. Register Commands in ossec.conf. Edit /var/ossec/etc/ossec.conf. Under <ossec_config>:

<command>
  <name>quarantine_host.sh</name>
  <executable>quarantine_host.sh</executable>
  <timeout_allowed>yes</timeout_allowed>
</command>

<command>
  <name>terminate_c2.sh</name>
  <executable>terminate_c2.sh</executable>
  <timeout_allowed>yes</timeout_allowed>
</command>

<command>
  <name>write_contextual_logs_malcom_active_response.py</name>
  <executable>write_contextual_logs_malcom_active_response.py</executable>
  <timeout_allowed>no</timeout_allowed>
</command>

Part 6: Active Response Analysis

Our anomaly detection uses thresholds to prioritize and differentiate alerts based on severity. Two tiers are applied:

Tier 1 – Monitor & Investigate:

• anomaly_grade >= 0.8
• confidence >= 0.85
• Action: Log the alert, enrich context, monitor trends. Analysts investigate further via dashboards.

Tier 2 – Active Mitigation:

• anomaly_grade >= 0.95
• confidence >= 0.95
• Action: Automatically trigger active responses such as terminate_c2.sh or quarantine_host.sh.

These levels balance precision and recall. Lowering the threshold improves detection rate but may increase false positives. Conversely, stricter thresholds reduce noise but risk missing early indicators.

To ensure operational trust, all thresholds are empirically tuned using the labeled dataset and real-world behavioral validation.

Part 7. Risk Analysis

In a scenario where an endpoint begins initiating outbound connections to unfamiliar IP addresses on unusual ports—consistent with beaconing behavior to a potential command-and-control (C2) server—the risk is quantified using the following risk model:

R = C × I

• C is the anomaly detection model’s confidence score, used as a proxy for the likelihood of malicious behavior.
• I is the impact severity derived from the Common Vulnerability Scoring System (CVSS).

In the case of C2 communication detection:

• Confidentiality: High risk, as persistent beaconing may enable data exfiltration.
• Integrity: Moderate-to-high risk, due to potential remote command execution or system manipulation.
• Availability: Low or none, as communication volume is typically minimal and stealthy.

To translate this into actionable security tiers, we define thresholds:

• Tier 2: flag for investigation
• Tier 3: urgent containment

Under this framework, high-confidence detection of stealthy C2 communications (e.g., regular outbound connections to unfamiliar hosts) elevates the risk score toward the high end of the scale, often triggering Tier 2 or Tier 3 depending on historical baselines. This method ensures that alert severity reflects both behavioral certainty and standardized impact—adhering to the principle of Likelihood × Impact while enabling consistent, interpretable threat prioritization.

Part 8: Dataset

The dataset originates from Kaggle - Network Malware Detection Connection Analysis. It is derived from the CTU-13 Botnet Traffic dataset, featuring labeled connection records representing benign and malicious communication traces.

This site is open source. Improve this page.