Feature/support all pb fields [WIP] #10

ExaneServerTeam · 2021-01-05T11:54:13Z

Hello,

This PR is a major rework on the Protobuf Parsing.
The Published Json is quite changed.

It supports:

Almost all fields (originalRequestorSubnet is missing / No use & test case on my side)
Decodes of RRs & rdata
Split functions for Common/Query/Response/RRs data

My entended use is to provide required info for our SIEM with RPZ support on the DNS side (PDNS-Recursor)

* Add specific functions to parse Response, RRs and common fields * Extended use of dnspython lib (especialy for rdata mgmt)

* Add a dedicated Query Field parser * Move from/to parsing to the Common parser

Habbie · 2021-01-05T12:33:20Z

The Published Json is quite changed.

How compatible is it to the old format, for consumers?

Almost all fields (originalRequestorSubnet is missing / No use & test case on my side)

This field is important to many users, can you add it anyway? :)

originalRequestorSubnet support is loosely based on ProtobufLogger.py in PDNS source tree.

ExaneServerTeam · 2021-01-05T15:00:49Z

The Published Json is quite changed.

How compatible is it to the old format, for consumers?

I'll try to raise a diff or something. BRB on the subject.

Almost all fields (originalRequestorSubnet is missing / No use & test case on my side)

This field is important to many users, can you add it anyway? :)

PR updated with the support of originalRequestorSubnet. It is based on Protobuf.py implementation in PDNS source tree.

ExaneServerTeam · 2021-01-05T15:46:40Z

JSON Produced by this PR

The main difference I see from current layout:

Query related data have been moved to query dict
return_code has moved to response dict

{
    "dns_message": "CLIENT_RESPONSE",
    "message_id": "f344c6f134c14351b87588d3e63bc8bf",
    "socket_family": "IPv4",
    "socket_protocol": "UDP",
    "from_address": "10.0.0.1",
    "to_address": "10.0.80.2",
    "bytes": 247,
    "dns_id": 34516,
    "from_port": 54462,
    "to_port": 53,
    "response": {
        "return_code": "NOERROR",
        "rrs": [
            {
                "name": "a767.dscg3.akamai.net.",
                "type": "A",
                "class": "IN",
                "ttl": 18,
                "rdata": {
                    "address": "92.122.122.152"
                },
                "udr": false
            },
            {
                "name": "a767.dscg3.akamai.net.",
                "type": "A",
                "class": "IN",
                "ttl": 18,
                "rdata": {
                    "address": "92.122.122.147"
                },
                "udr": false
            }
        ]
    },
    "query_time": "2021-01-05T15:32:16.574470+00:00",
    "response_time": "2021-01-05T15:32:16.575857+00:00",
    "latency": 0.001387,
    "query": {
        "name": "ctldl.windowsupdate.com.",
        "type": "A",
        "class": "IN"
    }
}

Reference JSON / Current one

{
    "dns_message": "AUTH_QUERY",
    "socket_family": "IPv6",
    "socket protocol": "UDP",
    "from_address": "0.0.0.0",
    "to_address": "184.26.161.130",
    "query_time": "2020-05-29 13:46:23.322",
    "response_time": "1970-01-01 01:00:00.000",
    "latency": 0,
    "query_type": "A",
    "query_name": "a13-130.akagtm.org.",
    "return_code": "NOERROR",
    "bytes": 4
}

Tested on the following query types (PowerDNS recursor export fields): + A + AAAA + CNAME + TXT + MX + NS + PTR + SRV Not tested: 'SPF': is deprecated, I don't know if there is still any usage. cf: https://tools.ietf.org/html/rfc7208#page-11 A small remark: * Transmitted MX & SRV fields are incomplete (Missing priority/ports ... integers). This may need an upstream fix.

ExaneServerTeam · 2021-01-08T13:56:02Z

Hello, this version works a lot better. The previous had plenty of error while parsing rdata (Sorry my dev env required some firewall opening at my company to be operational).

It has been tested so far on:

A
AAAA
CNAME
TXT
MX
NS
PTR
SRV

Not done:
'SPF': Almost impossible to test (Deprecated)
cf: https://tools.ietf.org/html/rfc7208#page-11

ExaneServerTeam · 2021-01-21T12:50:28Z

@Habbie @dmachard Any thought or remarks ?

I have this version run for the past days and it works fine on my side with ~400 clients so far.

lwhitworth · 2021-05-05T14:43:42Z

FWIW I've a similar requirement and have implemented this code. I'm pushing to the receiver from DNSDist and had to remove the lines:

        "appliedPolicyTrigger": "applied_policy_trigger",  # 8
        "appliedPolicyHit": "applied_policy_hit",  # 9

As I was getting:

May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: 2021-05-05 15:28:06,897 Task exception was never retrieved
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: future: <Task finished coro=<cb_onpayload() done, defined at /usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py:278> exception=ValueError('Protocol message DNSResponse has no field appliedPolicyHit.',)>
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: Traceback (most recent call last):
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]:   File "/usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py", line 308, in cb_onpayload
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]:     parse_pb_msg_response(dns_pb2, dns_msg)
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]:   File "/usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py", line 203, in parse_pb_msg_response
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]:     elif dns_pb2.response.HasField(key):
May 05 15:28:06 pdnsr_host pdns_protobuf_receiver[2852]: ValueError: Protocol message DNSResponse has no field appliedPolicyHit.

and

May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: 2021-05-05 15:26:32,257 Task exception was never retrieved
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: future: <Task finished coro=<cb_onpayload() done, defined at /usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py:279> exception=ValueError('Protocol message DNSResponse has no field appliedPolicyTrigger.',)>
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: Traceback (most recent call last):
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]:   File "/usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py", line 309, in cb_onpayload
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]:     parse_pb_msg_rrs(dns_pb2, dns_msg)
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]:   File "/usr/local/lib/python3.6/dist-packages/pdns_protobuf_receiver/receiver.py", line 204, in parse_pb_msg_response
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]:     if isinstance(val, str):
May 05 15:26:32 pdnsr_host pdns_protobuf_receiver[1785]: ValueError: Protocol message DNSResponse has no field appliedPolicyTrigger.

Other than that though it seems to be a step forward definitely. I'm seeing response.rrs.rdata.address fields as expected which is definitely useful for us.

DNSdist is not in line with PDNS Recursor Protobuf Message version. This is require catching the Exception raised (And silently ignored)

ExaneServerTeam · 2021-05-20T09:41:27Z

@lwhitworth Hello I have been late on this.
Looks like the PDNS Recursor PB messages are not in line with current dnsdist (1.5.1 at least) release.
The last commit should fix the issue. Could you give it a try ?

ExaneServerTeam added 3 commits January 5, 2021 11:50

Support all protobuf fields v1

f6db954

* Add specific functions to parse Response, RRs and common fields * Extended use of dnspython lib (especialy for rdata mgmt)

V1.1: Rework layout

b5db6a0

* Add a dedicated Query Field parser * Move from/to parsing to the Common parser

Black the code

4910d5d

Support originalRequestorSubnet and requestorId

6d979d9

originalRequestorSubnet support is loosely based on ProtobufLogger.py in PDNS source tree.

Fix issue with Protobof Message Version

a4cf405

DNSdist is not in line with PDNS Recursor Protobuf Message version. This is require catching the Exception raised (And silently ignored)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/support all pb fields [WIP] #10

Feature/support all pb fields [WIP] #10

Uh oh!

ExaneServerTeam commented Jan 5, 2021

Uh oh!

Habbie commented Jan 5, 2021

Uh oh!

ExaneServerTeam commented Jan 5, 2021 •

edited

Loading

Uh oh!

ExaneServerTeam commented Jan 5, 2021 •

edited

Loading

Uh oh!

ExaneServerTeam commented Jan 8, 2021

Uh oh!

ExaneServerTeam commented Jan 21, 2021

Uh oh!

lwhitworth commented May 5, 2021 •

edited

Loading

Uh oh!

ExaneServerTeam commented May 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Feature/support all pb fields [WIP] #10

Are you sure you want to change the base?

Feature/support all pb fields [WIP] #10

Uh oh!

Conversation

ExaneServerTeam commented Jan 5, 2021

Uh oh!

Habbie commented Jan 5, 2021

Uh oh!

ExaneServerTeam commented Jan 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ExaneServerTeam commented Jan 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

JSON Produced by this PR

Reference JSON / Current one

Uh oh!

ExaneServerTeam commented Jan 8, 2021

Uh oh!

ExaneServerTeam commented Jan 21, 2021

Uh oh!

lwhitworth commented May 5, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ExaneServerTeam commented May 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ExaneServerTeam commented Jan 5, 2021 •

edited

Loading

ExaneServerTeam commented Jan 5, 2021 •

edited

Loading

lwhitworth commented May 5, 2021 •

edited

Loading