Quad9 analyzer errors

[mlodic]

While running an analysis for "www.test.com" with the analyzers Quad9_Malicious_Detector and Quad9_DNS, both failed with the following errors:

intelowl_celery_worker_default    | Traceback (most recent call last):
intelowl_celery_worker_default    |   File "/opt/deploy/intel_owl/api_app/classes.py", line 297, in start
intelowl_celery_worker_default    |     _result = self.run()
intelowl_celery_worker_default    |               ^^^^^^^^^^
intelowl_celery_worker_default    |   File "/opt/deploy/intel_owl/api_app/analyzers_manager/observable_analyzers/dns/dns_malicious_detectors/quad9_malicious_detector.py", line 41, in run
intelowl_celery_worker_default    |     quad9_answer = self._quad9_dns_query(observable)
intelowl_celery_worker_default    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_celery_worker_default    |   File "/opt/deploy/intel_owl/api_app/analyzers_manager/observable_analyzers/dns/dns_malicious_detectors/quad9_malicious_detector.py", line 83, in _quad9_dns_query
intelowl_celery_worker_default    |     resolutions.extend([resolution.address for resolution in answer])
intelowl_celery_worker_default    |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_celery_worker_default    |   File "/opt/deploy/intel_owl/api_app/analyzers_manager/observable_analyzers/dns/dns_malicious_detectors/quad9_malicious_detector.py", line 83, in <listcomp>
intelowl_celery_worker_default    |     resolutions.extend([resolution.address for resolution in answer])
intelowl_celery_worker_default    |                         ^^^^^^^^^^^^^^^^^^
intelowl_celery_worker_default    | AttributeError: 'CNAME' object has no attribute 'address'
intelowl_celery_worker_default    | Traceback (most recent call last):
intelowl_celery_worker_default    |   File "/opt/deploy/intel_owl/api_app/classes.py", line 297, in start
intelowl_celery_worker_default    |     _result = self.run()
intelowl_celery_worker_default    |               ^^^^^^^^^^
intelowl_celery_worker_default    |   File "/opt/deploy/intel_owl/api_app/analyzers_manager/observable_analyzers/dns/dns_resolvers/quad9_dns_resolver.py", line 48, in run
intelowl_celery_worker_default    |     json_response = quad9_response.json()
intelowl_celery_worker_default    |                     ^^^^^^^^^^^^^^^^^^^^^
intelowl_celery_worker_default    |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 832, in json
intelowl_celery_worker_default    |     return jsonlib.loads(self.content, **kwargs)
intelowl_celery_worker_default    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_celery_worker_default    |   File "/usr/local/lib/python3.11/json/__init__.py", line 341, in loads
intelowl_celery_worker_default    |     s = s.decode(detect_encoding(s), 'surrogatepass')
intelowl_celery_worker_default    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
intelowl_celery_worker_default    | UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 0: invalid continuation byte

[nishantxscooby]

Hi @mlodic, I’ve just finished working on the Capa_Info permission fix and would like to take a look at this issue next if no one is already on it.

May I get assigned so I can start investigating the Quad9 analyzer errors?

[ManaswibRane]

Hi @mlodic , I’ve submitted a PR addressing this issue. Could you please review it?
#3164

[mlodic]

You need to get assigned before opening a PR and you need to have completed your previous assignment.

[ManaswibRane]

Hi @mlodic
I’ve already prepared a fix for this issue .
Could you please assign this issue to me? Thanks!

[ManaswibRane]

Hi @mlodic ,

The proposed fixes for the Quad9 analyzers have been implemented in PR #3179 . This should address and resolve the issue. Screenshots and JSON outputs are attached for verification.

Sorry for the trouble, and thanks for your patience!

[ManaswibRane]

Hi @mlodic
While working on fixing the UnicodeDecodeError in Quad9DNSResolver, I discovered a related issue that causes the analyzer to always return empty results.
The Problem:
The Quad9DNSResolver class (api_app/analyzers_manager/observable_analyzers/dns/dns_resolvers/quad9_dns_resolver.py) inherits from DoHMixin and attempts to parse the response as JSON:

json_response = quad9_response.json()
for answer in json_response.get("Answer", []):
if (data := answer.get("data")) is not None:
resolutions.append(data)

However, DoHMixin (api_app/analyzers_manager/observable_analyzers/dns/doh_mixin.py) sets the header to request wire format (binary DNS messages), not JSON:
pythonheaders: dict = {"Accept": "application/dns-message"}
This mismatch causes the JSON parsing to fail silently (or throw errors), resulting in empty resolution lists even for valid domains.

I’ve just started contributing, so please let me know if this is indeed an issue or if there’s something I should handle differently.

[mlodic]

Yes, that's an issue. As you can see in the similar Quad9MaliciousDetector analyzer, another method is used to decode the content:
dns_response = dns.message.from_wire(quad9_response.content)

we should do the same there. Can you add this change and test it in your actual open PR?

Thanks reporting this.

[mlodic]

actually, those 2 analyzers shared some code, it would be better if the _quad9_dns_query would be shared among them to avoid getting these errors, considering that those 2 analyzers are performing the queries in the same way because it is the same service.