--- title: 'Setting Up the Core of IntelliSIEM – Structure, TDD, and API Connection' url: 'https://bsb.perryconsulting.net/blog/setting-up-the-core-of-intellisiem-structure-tdd-and-api-connection' markdown: 'https://bsb.perryconsulting.net/blog/setting-up-the-core-of-intellisiem-structure-tdd-and-api-connection.md' date: '2024-09-25' description: "Part 1 of this series covered the initial tool setup, and testing of the environment. In Part 2, we're going to dive into structure, tests, and API connections. In this post, I’m excited to dive into the development of IntelliSIEM’s data collection module. This step is crucial as it lays the foundation for how we will gather, process, and analyze threat data from various sources. To see this project in its curr" taxonomy: category: - projects tag: - python - security --- # Setting Up the Core of IntelliSIEM – Structure, TDD, and API Connection 25 SEP 2024 [python](https://bsb.perryconsulting.net/blog/tag:python) [security](https://bsb.perryconsulting.net/blog/tag:security) *Part 1 of this series covered the initial tool setup, and testing of the environment.**In Part 2, we're going to dive into structure, tests, and API connections.* In this post, I’m excited to dive into the development of IntelliSIEM’s data collection module. This step is crucial as it lays the foundation for how we will gather, process, and analyze threat data from various sources. To see this project in its current state, head over to my repo and check it out: [IntelliSIEM on GitHub](https://github.com/perryconsulting/IntelliSIEM) --- ### **Step 1: Project Structure and Setup** To ensure the project remains organized and scalable, I began by designing a clear directory structure with separate folders for configuration, data, source code, and tests: ``` IntelliSIEM/ ├── LICENSE ├── README.md ├── config/ │ └── config_template.yaml ├── data/ ├── src/ │ ├── __init__.py │ ├── data_collection.py │ ├── error_handling.py │ └── api_client.py ├── tests/ │ ├── __init__.py │ └── test_data_collection.py └── requirements.txt ``` With this design in mind, I created the `config`, `data`, `src`, and `tests` subdirectories in the project directory. In the project settings in PyCharm, I also marked `src` as "Sources", `tests` as "Tests", and `config` and `data` as "Resources". I then added `__init__.py` files to the `src/` and `tests/` directories to make them Python packages, ensuring smooth module imports. Finally, I added `requirements.txt` so that end users can easily import the libraries necessary to run the application. BASH ``` mkdir config mkdir data mkdir src mkdir tests touch src/__init__.py touch tests/__init__.py touch requirements.txt ``` ### **Step 2: README and License** Before pushing any code to GitHub, I documented the project setup and usage in the `README.md` file and applied the Apache License 2.0 to protect the code and clarify usage rights. Applying the license involves creating the LICENSE file, populating it with the entire license text, and creating a stub placed at the top of every code file. You'll see that stub as we move through the project. To see the full text of either, check out the full project in the [GitHub repo](https://github.com/perryconsulting/IntelliSIEM). ### **Step 3: Credential Handling** Sensitive information like API keys should never be hard-coded. For this early stage of development, I created a `config/config_template.yaml` file with the basic syntax that we'll build on later to implement a more secure way of handling the keys: YAML ``` 1api_keys: 2 alient_vault: "YOUR_ALIEN_VAULT_API_KEY" 3 virus_total: "YOUR_VIRUS_TOTAL_API_KEY" ``` To prevent the exposure of our own keys, we'll first add an entry to `.gitignore` for `config/config.yaml`. As a reminder, the `.gitignore` should now look something like this: .gitignore ``` 1.venv/ 2.idea/ 3__pychache__/ 4/config/config.yaml ``` Then, copy the template to `config.yaml` and fill in our keys. This will ensure that version control ignores the file to prevent accidental exposure. BASH ``` cp config/config_template.yaml config/config.yaml ``` ### **Step 4: Implementing Test-Driven Development (TDD)** OK. I can already hear the groans, "Why are we wasting time on this?" or "TDD is too hard!". Yeah, I get it, but the crux of the matter is that test-driven development helps catch issues early. I wrote some initial test cases using `pytest` and `pytest-mock` for the data collection module. First things first, we need to make sure we've added `pytest` to our project. Let's utilize our newly created `requirements.txt` file to make sure we have everything we need so far. Open it up in the editor and it should look like this: TXT ``` requests pandas matplotlib seaborn fpdf reportlab json5 pyyaml pytest pytest-mock beautifulsoup4 lxml sqlalchemy ``` Save that and open the console again and we'll test it: BASH ``` pip install -r requirements.txt ``` Now that we have all our libraries installed, we're going to create our first tests. Remember, in TDD we always expect the tests to fail first. Here’s a sample of the tests I wrote in `tests/test_data_collection.py`: PYTHON ``` 1# IntelliSIEM Copyright 2024, Rob Perry 2# 3# Licensed under the Apache License, Version 2.0 (the "License"); 4# you may not use this file except in compliance with the License. 5# You may obtain a copy of the License at 6# 7# http://www.apache.org/licenses/LICENSE-2.0 8# 9# Unless required by applicable law or agreed to in writing, software 10# distributed under the License is distributed on an "AS IS" BASIS, 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12# See the License for the specific language governing permissions and 13# limitations under the License. 14 15from src.data_collection import fetch_threat_data, APIError 16 17def test_fetch_threat_data_success(mocker): 18 """ 19 Test fetching threat data with valid API keys. 20 """ 21 mocker.patch('src.api_client.APIClient.get_data', return_value=[{'id': 'test'}]) 22 threats = fetch_threat_data() 23 assert len(threats) > 0 24 25def test_fetch_threat_data_failure(mocker): 26 """ 27 Test fetching threat data with invalid API keys. 28 """ 29 # Mock API client responses to raise an error 30 mocker.patch('src.api_client.APIClient.get_data', side_effect=APIError("API Error")) 31 32 # Expect the APIError exception to be raised and handle it in the function 33 threats = fetch_threat_data() 34 assert threats == [] # Expecting an empty list as a fallback 35 ``` Now, as I said, we anticipate that this is going to fail and we can verify that by running our first tests. BASH ``` pytest /tests ``` Congratulations, you've failed your first test! Now, we can move on to the implementation and start passing. To keep this brief, I'm going to assume that you are consistently running your tests every time you make a change until all your tests pass, so I won't keep covering that aspect repeatedly. ### **Step 5: Implementing the Data Collection Module** The data collection module is the heart of IntelliSIEM. It fetches data from various threat intelligence APIs. Following our tests logically, we're first importing from `src.data_collection`, but we haven't created that file yet. Let's do that now in `src/data_collection.py`: PYTHON ``` 1# IntelliSIEM Copyright 2024, Rob Perry 2# 3# Licensed under the Apache License, Version 2.0 (the "License"); 4# you may not use this file except in compliance with the License. 5# You may obtain a copy of the License at 6# 7# http://www.apache.org/licenses/LICENSE-2.0 8# 9# Unless required by applicable law or agreed to in writing, software 10# distributed under the License is distributed on an "AS IS" BASIS, 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12# See the License for the specific language governing permissions and 13# limitations under the License. 14 15from src.api_client import APIClient 16from src.error_handling import APIError, log_error 17from src.config import load_config 18 19config = load_config() 20 21def fetch_threat_data(): 22 """ 23 Fetch threat data from various sources and return the combined result. 24 25 :return: (list) A list of threat intelligence data. 26 """ 27 try: 28 alien_vault = APIClient("https://otx.alienvault.com/api/v1", 29 {"X-OTX-API-KEY": config['api_keys']['alien_vault']}) 30 vt = APIClient("https://www.virustotal.com/api/v3", {"x-apikey": config['api_keys']['virus_total']}) 31 32 alien_vault_data = alien_vault.get_data("indicators/export") 33 vt_data = vt.get_data("files", params={"limit": 10}) 34 35 return alien_vault_data + vt_data 36 except APIError as e: 37 log_error(f"Failed to fetch threat data: {e}") 38 return [] 39 ``` Continuing logically, we need to implement the `APIClient` class next. The `APIClient` class manages the API requests and provides error handling. We'll build more robust error handling as we build better tests throughout the project. Here is the current src/api\_client.py: PYTHON ``` 1# IntelliSIEM Copyright 2024, Rob Perry 2# 3# Licensed under the Apache License, Version 2.0 (the "License"); 4# you may not use this file except in compliance with the License. 5# You may obtain a copy of the License at 6# 7# http://www.apache.org/licenses/LICENSE-2.0 8# 9# Unless required by applicable law or agreed to in writing, software 10# distributed under the License is distributed on an "AS IS" BASIS, 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12# See the License for the specific language governing permissions and 13# limitations under the License. 14 15import requests 16from src.error_handling import APIError, log_error 17 18class APIClient: 19 """ 20 A client for interacting with various threat intelligence APIs. 21 """ 22 23 def __init__(self, base_url, headers): 24 """ 25 Initialize the API client with the base URL and headers. 26 27 :param base_url: (str) The base URL of the API. 28 :param headers: (dict) Headers required for API requests. 29 """ 30 self.base_url = base_url 31 self.headers = headers 32 33 def get_data(self, endpoint, params=None): 34 """ 35 Fetch data from the specified endpoint. 36 37 :param endpoint: (str) The API endpoint to fetch data from. 38 :param params: (dict) Optional query parameters. 39 :return: (dict) JSON response from the API. 40 :raises APIError: If there is an error with the API request. 41 """ 42 url = f"{self.base_url}/{endpoint}" 43 try: 44 response = requests.get(url, headers=self.headers, params=params, timeout=10) 45 response.raise_for_status() 46 return response.json() 47 except requests.exceptions.RequestException as e: 48 log_error(f"API request failed: {e}") 49 raise APIError(f"API request failed: {e}") 50 ``` I'm going to touch on error handling in Step 6, so before we move on to that, let's ensure that we've implementing loading the API keys from the `config.yaml` file. We take care of that in `config.py`: PYTHON ``` 1# IntelliSIEM Copyright 2024, Rob Perry 2# 3# Licensed under the Apache License, Version 2.0 (the "License"); 4# you may not use this file except in compliance with the License. 5# You may obtain a copy of the License at 6# 7# http://www.apache.org/licenses/LICENSE-2.0 8# 9# Unless required by applicable law or agreed to in writing, software 10# distributed under the License is distributed on an "AS IS" BASIS, 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12# See the License for the specific language governing permissions and 13# limitations under the License. 14 15import yaml 16 17def load_config(config_file='config/config.yaml'): 18 """ 19 Load configuration from the specified YAML file. 20 21 :param config_file: The path to the YAML configuration file. 22 :return: dict: Configuration data. 23 """ 24 25 with open(config_file, 'r') as file: 26 return yaml.safe_load(file) 27 ``` ### **Step 6: Error Handling and Logging** To ensure smooth operation and easier debugging, I implemented a custom `APIError` class and a logging mechanism. Is it robust? Well, not really. As we work on really making this module more robust, we'll improve this class and enhance the error handling and reporting. For now, we're really interested in logging errors so that we can make sure we handle them gracefully when we reach a release-ready version. Here is the current `src/error_handling.py`: PYTHON ``` 1# IntelliSIEM Copyright 2024, Rob Perry 2# 3# Licensed under the Apache License, Version 2.0 (the "License"); 4# you may not use this file except in compliance with the License. 5# You may obtain a copy of the License at 6# 7# http://www.apache.org/licenses/LICENSE-2.0 8# 9# Unless required by applicable law or agreed to in writing, software 10# distributed under the License is distributed on an "AS IS" BASIS, 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12# See the License for the specific language governing permissions and 13# limitations under the License. 14 15import logging 16 17logging.basicConfig(level=logging.ERROR, filename='data/error.log', format='%(asctime)s %(levelname)s: %(message)s') 18 19class APIError(Exception): 20 """ 21 Custom exception for API errors. 22 """ 23 pass 24 25def log_error(message): 26 """ 27 Log error messages to a file. 28 29 :param message: (str) The error message to log. 30 """ 31 logging.error(message) ``` This helps track any issues that arise during API calls or data processing. Hopefully, all your tests have passed. We have the basic implementation of the data collector, but currently, our tests only cover the basic success and failure states of the collector. In the next post, we will improve and enhance our tests. Once we have robust tests in place, we'll move into better error handling and then revisit the API key handling. --- ## Navigation - Parent: [Blog](https://bsb.perryconsulting.net/blog.md) - Previous: [Implementing TDD in IntelliSIEM: Handling Edge Cases and Errors](https://bsb.perryconsulting.net/blog/strengthening-the-foundation-with-comprehensive-testing-in-intellisiem.md) - Next: [Getting Started with IntelliSIEM: Setting Up the Development Environment](https://bsb.perryconsulting.net/blog/getting-started-with-intellisiem-setting-up-the-development-environment.md)