---
title: 'Implementing TDD in IntelliSIEM: Handling Edge Cases and Errors'
url: 'https://bsb.perryconsulting.net/blog/strengthening-the-foundation-with-comprehensive-testing-in-intellisiem'
markdown: 'https://bsb.perryconsulting.net/blog/strengthening-the-foundation-with-comprehensive-testing-in-intellisiem.md'
date: '2024-09-28'
description: "Previous entries in this series covered the initial tool setup and testing of the environment, establishing the structure of the project, initial tests, and connecting to APIs. In this third entry, we're going to enhance our testing approach through a Test-Driven Development (TDD) methodology to build a more robust and reliable system. Initially, our tests covered basic scenarios but didn’t fully account for real-world challenges like invalid configurations, API rate limits, and malformed data. We will expand our test cases significantly to address this, focusing on edge cases and error handling. To see this project in its curr"
taxonomy:
category:
- projects
tag:
- python
- security
---
# Implementing TDD in IntelliSIEM: Handling Edge Cases and Errors
28 SEP 2024 [python](https://bsb.perryconsulting.net/blog/tag:python) [security](https://bsb.perryconsulting.net/blog/tag:security)
*Previous entries in this series covered the initial tool setup and testing of the environment, establishing the structure of the project, initial tests, and connecting to APIs. In this third entry, we're going to* *enhance our testing approach through a Test-Driven Development (TDD) methodology to build a more robust and reliable system.*
Initially, our tests covered basic scenarios but didn’t fully account for real-world challenges like invalid configurations, API rate limits, and malformed data. We will expand our test cases significantly to address this, focusing on edge cases and error handling.
To see this project in its current state, head over to my repo and check it out: [IntelliSIEM on GitHub](https://github.com/perryconsulting/IntelliSIEM)
---
### **Enhancing Our Test Scenarios**
In our initial design, we tested for success and failure of threat data collection from the APIs, but what about other potential issues? Let's expand out our testing to ensure that we're handling other potential data issues such as an empty or corrupt response. Here's the new `tests/test_data_collection.py`:
PYTHON
```
1# IntelliSIEM Copyright 2024, Rob Perry
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14# tests/test_data_collection.py
15
16import pytest
17from src.data_collection import fetch_threat_data, APIError
18
19def test_fetch_threat_data_success(mocker):
20 """
21 Test fetching threat data with valid API keys.
22 """
23 mocker.patch('src.api_client.APIClient.get_data', return_value=[{'id': 'test'}])
24 threats = fetch_threat_data()
25 assert len(threats) > 0
26
27def test_fetch_threat_data_failure(mocker):
28 """
29 Test fetching threat data with invalid API keys.
30 """
31 # Mock API client responses to raise an error
32 mocker.patch('src.api_client.APIClient.get_data', side_effect=APIError("API Error"))
33
34 # Expect the APIError exception to be raised and handle it in the function
35 threats = fetch_threat_data()
36 assert threats == [] # Expecting an empty list as a fallback
37
38def test_fetch_threat_data_empty_response(mocker):
39 """
40 Test fetching threat data with an empty response.
41 """
42 mocker.patch('src.api_client.APIClient.get_data', return_value=[])
43 threats = fetch_threat_data()
44 assert len(threats) == 0 # Expecting empty list when no data is returned
45
46def test_fetch_threat_data_corrupt_data(mocker):
47 """
48 Test fetching threat data with corrupt response data.
49 """
50 mocker.patch('src.api_client.APIClient.get_data', return_value=[{'malformed': 'data'}])
51 threats = fetch_threat_data()
52 assert threats == [] # Expecting empty list on data parsing failure
```
In the same vein, we also need to make sure we can handle things like a missing or invalid config file, so we'll add a new test file `tests/test_config.py`:
PYTHON
```
1# IntelliSIEM Copyright 2024, Rob Perry
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15import pytest
16from src.config import load_config
17from src.error_handling import APIError
18
19def test_load_config_missing_file():
20 """
21 Test loading configuration with a missing file.
22 """
23 with pytest.raises(FileNotFoundError):
24 load_config('config/non_existent_config.yaml')
25
26def test_load_config_invalid_format(mocker):
27 """
28 Test loading configuration file with an invalid format.
29 """
30 mocker.patch('builtins.open', mocker.mock_open(read_data="invalid_yaml"))
31 with pytest.raises(APIError):
32 load_config()
```
APIs can be unpredictable sometimes. We should also make sure we can handle other potential API issues that come up, like timeouts, rate limiting, and invalid responses, so we'll also add a new `tests/test_api_client.py`:
PYTHON
```
1# IntelliSIEM Copyright 2024, Rob Perry
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15import pytest
16import requests
17from src.api_client import APIClient, APIError
18
19def test_api_client_timeout(mocker):
20 """
21 Test API client handling of a timeout error.
22 """
23 mocker.patch('requests.get', side_effect=requests.exceptions.Timeout)
24 client = APIClient("https://example.com/api", {"Authorization": "Bearer token"})
25 # Match a more general pattern, ignoring the specific exception message.
26 with pytest.raises(APIError, match="API request timed out"):
27 client.get_data("endpoint")
28
29def test_api_client_invalid_response(mocker):
30 """
31 Test API client handling of invalid JSON response.
32 """
33 mocker.patch('requests.get', return_value=mocker.Mock(status_code=200, json=mocker.Mock(side_effect=ValueError)))
34 client = APIClient("https://example.com/api", {"Authorization": "Bearer token"})
35 with pytest.raises(APIError, match="Invalid response format."):
36 client.get_data("endpoint")
37
38def test_api_client_rate_limit(mocker):
39 """
40 Test API client handling of rate limiting.
41 """
42 mock_response = mocker.Mock(status_code=429, headers={"Retry-After": "5"})
43 mocker.patch('requests.get', return_value=mock_response)
44 client = APIClient("https://example.com/api", {"Authorization": "Bearer token"})
45 with pytest.raises(APIError, match="Rate limit exceeded."):
46 client.get_data("endpoint")
```
### **Time To Retest Our Code**
Now that we've got our shiny new tests in place, it's time to see how well we did.
BASH
```
pytest /tests
```
I hope you like the color red because there's quite a bit of it. The good news is that because we're letting our tests determine how we implement, all these error messages will point us in exactly the right direction to help ensure our error handling and function implementations are as correct as we can make them.
*Some of you, depending on which version of the project you're on, may notice an error relating to the failure to load APIError from the config.py file. Don't worry; we'll fix that below as we work through the other issues.*
*Note: the snipped output below does not reflect the actual color you may see in your terminal. My preferred dev environment generally uses the various JetBrains tools on Fedora, so my error output is bright red. The plugin I use for this site uses the default VSCode color scheme.*
BASH
```
=========================================================================================================================== short test summary info ============================================================================================================================
FAILED tests/test_api_client.py::test_api_client_timeout - AssertionError: Regex pattern did not match.
FAILED tests/test_api_client.py::test_api_client_invalid_response - ValueError
FAILED tests/test_api_client.py::test_api_client_rate_limit - Failed: DID NOT RAISE <class 'src.error_handling.APIError'>
FAILED tests/test_config.py::test_load_config_invalid_format - Failed: DID NOT RAISE <class 'src.error_handling.APIError'>
FAILED tests/test_data_collection.py::test_fetch_threat_data_corrupt_data - AssertionError: assert [{'malformed'...med': 'data'}] == []
========================================================================================================================= 5 failed, 4 passed in 0.33s ==========================================================================================================================
```
Remember, we implemented the tests before we actually implemented the code to handle those situations! Let's walk through each of these in order and fix them. Note that I will only provide the changes to implement for each file until all changes have been implemented, then I will show the full final version.
### Failure 1: test\_api\_client\_timeout - AssertionError
The regex pattern for the raised `APIError` did not match the actual error message. The simple fact is, we haven't implemented a specific error handler for this yet. Here's the specific test we're looking at:
PYTHON
```
1def test_api_client_timeout(mocker):
2 """
3 Test API client handling of a timeout error.
4 """
5 mocker.patch('requests.get', side_effect=requests.exceptions.Timeout)
6 client = APIClient("https://example.com/api", {"Authorization": "Bearer token"})
7 # Match a more general pattern, ignoring the specific exception message.
8 with pytest.raises(APIError, match="API request timed out"):
9 client.get_data("endpoint")
```
We need to modify the `src/api_client.py get_data` method to raise a specific error message for a timeout error:
PYTHON
```
1except requests.exceptions.Timeout:
2 log_error("API request timed out")
3 raise APIError("API request timed out")
```
If we re-run our tests again, there should be one less to worry about.
### **Failure 2: test\_api\_client\_invalid\_response - ValueError**
This test looks for JSON parsing errors, but the current implementation doesn't specifically catch them:
PYTHON
```
1def test_api_client_invalid_response(mocker):
2 """
3 Test API client handling of invalid JSON response.
4 """
5 mocker.patch('requests.get', return_value=mocker.Mock(status_code=200, json=mocker.Mock(side_effect=ValueError)))
6 client = APIClient("https://example.com/api", {"Authorization": "Bearer token"})
7 with pytest.raises(APIError, match="Invalid response format."):
8 client.get_data("endpoint")
```
Still working in `api_client.py`, we should catch this exception and raise an appropriate `APIError`.
PYTHON
```
1except ValueError:
2 log_error("Invalid JSON response format from API")
3 raise APIError("Invalid response format.") # New error for invalid JSON responses
```
### **Failure 3: test\_api\_client\_rate\_limit - Failed: DID NOT RAISE**
This message is a bit different from the previous two. When our account is rate-limited by the remote service, we should get a specific status code that triggers an error for us to handle. This test case expects an `APIError` to be raised, but the current implementation doesn't specifically handle the `429` status code.
PYTHON
```
1def test_api_client_rate_limit(mocker):
2 """
3 Test API client handling of rate limiting.
4 """
5 mock_response = mocker.Mock(status_code=429, headers={"Retry-After": "5"})
6 mocker.patch('requests.get', return_value=mock_response)
7 client = APIClient("https://example.com/api", {"Authorization": "Bearer token"})
8 with pytest.raises(APIError, match="Rate limit exceeded."):
9 client.get_data("endpoint")
```
Let's go ahead and make sure we handle this case. For now, we want to raise the exception. Later, we may wish to consider refactoring this code to address the situation differently. This is the last error specific to the api\_client.py file, so here's the final code to date:
PYTHON
```
1# IntelliSIEM Copyright 2024, Rob Perry
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15import requests
16from src.error_handling import APIError, log_error
17
18class APIClient:
19 """
20 A client for interacting with various threat intelligence APIs.
21 """
22
23 def __init__(self, base_url, headers):
24 """
25 Initialize the API client with the base URL and headers.
26
27 :param base_url: (str) The base URL of the API.
28 :param headers: (dict) Headers required for API requests.
29 """
30 self.base_url = base_url
31 self.headers = headers
32
33 def get_data(self, endpoint, params=None):
34 """
35 Fetch data from the specified endpoint.
36
37 :param endpoint: (str) The API endpoint to fetch data from.
38 :param params: (dict) Optional query parameters.
39 :return: (dict) JSON response from the API.
40 :raises APIError: If there is an error with the API request.
41 """
42 url = f"{self.base_url}/{endpoint}"
43 try:
44 response = requests.get(url, headers=self.headers, params=params, timeout=10)
45 if response.status_code == 429:
46 retry_after = response.headers.get("Retry-After", "unknown time")
47 log_error(f"Rate limit exceeded. Retry after: {retry_after}")
48 raise APIError("Rate limit exceeded.")
49 response.raise_for_status()
50 return response.json()
51 except requests.exceptions.Timeout:
52 log_error("API request timed out")
53 raise APIError("API request timed out")
54 except requests.exceptions.RequestException as e:
55 log_error(f"API request failed: {e}")
56 raise APIError(f"API request failed: {e}")
57 except ValueError:
58 log_error("Invalid JSON response format from API")
59 raise APIError("Invalid response format.") # New error for invalid JSON responses
```
### **Failure 4: test\_load\_config\_invalid\_format - Failed: DID NOT RAISE**
The `test_load_config_invalid_format` test case expects an `APIError`, but the current `load_config` function doesn't raise it.
PYTHON
```
1def test_load_config_invalid_format(mocker):
2 """
3 Test loading configuration file with an invalid format.
4 """
5 mocker.patch('builtins.open', mocker.mock_open(read_data="invalid_yaml"))
6 with pytest.raises(APIError):
7 load_config()
```
Let's go ahead and update `load_config()` to raise `APIError` for parsing issues. Now, is this truly an API error? Nope. Should there be a separate class of errors relating to the application configuration? Probably. We'll add that to our TODO list, and circle back on the next post when we take a careful look at our error handling strategy. For now, this is the only failure related to `config.py`, so here's the current file:
PYTHON
```
1# IntelliSIEM Copyright 2024, Rob Perry
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15import yaml
16from src.error_handling import log_error, APIError
17
18def load_config(config_file='config/config.yaml'):
19 """
20 Load configuration from the specified YAML file.
21
22 :param config_file: (str) The path to the YAML configuration file.
23 :return: dict: Configuration data.
24 :raises APIError: If there is an error with the configuration file.
25 """
26
27 try:
28 with open(config_file, 'r') as file:
29 return yaml.safe_load(file)
30 except yaml.YAMLError as e:
31 log_error(f"YAML parsing error while loading configuration: {e}")
32 raise APIError(f"Configuration file error: {e}")
33 except FileNotFoundError:
34 log_error(f"Configuration file not found: {config_file}")
35 raise FileNotFoundError(f"Configuration file not found: {config_file}")
```
Now, interestingly, with this particular case there is also an issue with the test itself where `mocker` isn't providing test data quite correctly. To resolve this, we're also going to update our `test_config.py` file to be more specific with our test:
PYTHON
```
1# IntelliSIEM Copyright 2024, Rob Perry
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15import pytest
16from src.config import load_config, APIError
17
18def test_load_config_missing_file():
19 """
20 Test loading configuration with a missing file.
21 """
22 with pytest.raises(FileNotFoundError):
23 load_config('config/non_existent_config.yaml')
24
25def test_load_config_invalid_format(mocker):
26 """
27 Test loading configuration file with an invalid format.
28 """
29 invalid_yaml = """
30 api_keys:
31 alien_vault: !!invalid
32 """
33 mocker.patch('builtins.open', mocker.mock_open(read_data=invalid_yaml))
34 with pytest.raises(APIError, match="Configuration file error"):
35 load_config()
```
### **Failure 5: test\_fetch\_threat\_data\_corrupt\_data - AssertionError**
OK. Last test failure. In this case, another Assertion Error. The `test_fetch_threat_data_corrupt_data` test case expects an empty list for malformed data, but currently, the function does not filter it:
PYTHON
```
1def test_fetch_threat_data_corrupt_data(mocker):
2 """
3 Test fetching threat data with corrupt response data.
4 """
5 mocker.patch('src.api_client.APIClient.get_data', return_value=[{'malformed': 'data'}])
6 threats = fetch_threat_data()
7 assert threats == [] # Expecting empty list on data parsing failure
```
We need to update the `fetch_threat_data` function to handle malformed data. This is the only edit to `data_collection.py`, so here's the full code:
PYTHON
```
1# IntelliSIEM Copyright 2024, Rob Perry
2#
3# Licensed under the Apache License, Version 2.0 (the "License");
4# you may not use this file except in compliance with the License.
5# You may obtain a copy of the License at
6#
7# http://www.apache.org/licenses/LICENSE-2.0
8#
9# Unless required by applicable law or agreed to in writing, software
10# distributed under the License is distributed on an "AS IS" BASIS,
11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12# See the License for the specific language governing permissions and
13# limitations under the License.
14
15from src.api_client import APIClient
16from src.error_handling import APIError, log_error
17from src.config import load_config
18
19config = load_config()
20
21def fetch_threat_data():
22 """
23 Fetch threat data from various sources and return the combined result.
24
25 :return: (list) A list of threat intelligence data.
26 """
27 try:
28 alien_vault = APIClient("https://otx.alienvault.com/api/v1",
29 {"X-OTX-API-KEY": config['api_keys']['alien_vault']})
30 vt = APIClient("https://www.virustotal.com/api/v3", {"x-apikey": config['api_keys']['virus_total']})
31
32 alien_vault_data = alien_vault.get_data("indicators/export")
33 vt_data = vt.get_data("files", params={"limit": 10})
34
35 # Filter out malformed data
36 valid_data = [entry for entry in alien_vault_data + vt_data if 'id' in entry]
37 return valid_data
38 except APIError as e:
39 log_error(f"Failed to fetch threat data: {e}")
40 return [] # Return an empty list on failure
```
With that done, all tests should pass once again!
These additional tests help make IntelliSIEM more resilient, reliable, and capable of handling unexpected situations gracefully. Incorporating a testing framework lays the groundwork for future development, helping ensure that we catch issues early and maintain a high level of code quality. Next up, we'll dive in deeper and make sure we have a robust error-handling structure in place.
---
## Navigation
- Parent: [Blog](https://bsb.perryconsulting.net/blog.md)
- Previous: [IntelliSIEM and Test-Driven Development: Creating Robust Error Handling and Log Rotation](https://bsb.perryconsulting.net/blog/intellisiem-and-test-driven-development-creating-robust-error-handling-and-log-rotation.md)
- Next: [Setting Up the Core of IntelliSIEM – Structure, TDD, and API Connection](https://bsb.perryconsulting.net/blog/setting-up-the-core-of-intellisiem-structure-tdd-and-api-connection.md)