add missing tests and update docs (#659)

* add missing tests
* update docs

README.md

@@ -1,9 +1,9 @@
 <p align="center">
 <img src="https://goreportcard.com/badge/github.com/dmachard/go-dns-collector" alt="Go Report"/>
 <img src="https://img.shields.io/badge/go%20version-min%201.20-green" alt="Go version"/>
-<img src="https://img.shields.io/badge/go%20tests-440-green" alt="Go tests"/>
+<img src="https://img.shields.io/badge/go%20tests-446-green" alt="Go tests"/>
 <img src="https://img.shields.io/badge/go%20bench-19-green" alt="Go bench"/>
-<img src="https://img.shields.io/badge/go%20lines-39765-green" alt="Go lines"/>
+<img src="https://img.shields.io/badge/go%20lines-40169-green" alt="Go lines"/>
 </p>
 
 <p align="center">

docs/_integration/elasticsearch/README.md

@@ -1,7 +1,7 @@
 
 # DNS-collector with Elastic and Kibana
 
-- Download the `[docker-compose](https://github.com/dmachard/go-dnscollector/blob/doc_atags/docs/_integration/elasticsearch/docker-compose.yml)` file
+- Download the [`docker-compose`](https://github.com/dmachard/go-dnscollector/blob/doc_atags/docs/_integration/elasticsearch/docker-compose.yml) file
 
 - Create the `data` folder.
 

docs/_integration/fluentd/README.md

@@ -1,7 +1,7 @@
 
 # DNS-collector with Fluentd
 
-- Download the `[docker-compose](https://github.com/dmachard/go-dnscollector/blob/doc_atags/docs/_integration/fluentd/docker-compose.yml)` file
+- Download the [`docker-compose`](https://github.com/dmachard/go-dnscollector/blob/doc_atags/docs/_integration/fluentd/docker-compose.yml) file
 
 - Create the `data` folder.
 

docs/_integration/kafka/README.md

@@ -1,7 +1,7 @@
 
 # DNS-collector with Kafka
 
-- Download the `[docker-compose](https://github.com/dmachard/go-dnscollector/blob/doc_atags/docs/_integration/kafka/docker-compose.yml)` file
+- Download the [`docker-compose`](https://github.com/dmachard/go-dnscollector/blob/doc_atags/docs/_integration/kafka/docker-compose.yml) file
 
 - Create the `data` folder.
 

docs/collectors/collector_afpacket.md

@@ -15,10 +15,13 @@ sudo setcap cap_net_admin,cap_net_raw=eip go-dnscollector
 
 Options:
 
-* `port` (int) filter on source and destination port.
-* `device` (str) interface name to sniff.
-  > if value is empty, bind on all interfaces.
-* `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
+* `port` (int)
+  > filter on source and destination port.
+
+* `device` (str)
+  > Interface name to sniff. If value is empty, bind on all interfaces.
+
+* `chan-buffer-size` (int)
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
 
 Defaults:

docs/collectors/collector_dnstap.md

@@ -7,34 +7,47 @@ The traffic can be a tcp or unix DNStap stream. TLS is also supported.
 
 Options:
 
-- `listen-ip` (str)
+* `listen-ip` (str)
   > Set the local address that the server will bind to. If not provided, the server will bind to all available network interfaces (0.0.0.0).
-- `listen-port` (int)
+
+* `listen-port` (int)
   > Set the local port that the server will listen on. If not provided, use the default port.
-- `sock-path` (str)
+
+* `sock-path` (str)
   > Specify the path for the Unix socket to be created.
-- `tls-support` (bool)
+
+* `tls-support` (bool)
   > Enables or disables TLS (Transport Layer Security) support. If set to true, TLS will be used for secure communication.
-- `tls-min-version` (str)
+
+* `tls-min-version` (str)
   > Specifies the minimum TLS version that the server will support.
-- `cert-file` (str)
+
+* `cert-file` (str)
   > Specifies the path to the certificate file to be used for TLS. This is a required parameter if TLS support is enabled.
-- `key-file`(str)
+
+* `key-file`(str)
   > Specifies the path to the key file corresponding to the certificate file. This is a required parameter if TLS support is enabled.
-- `sock-rcvbuf` (int)
+
+* `sock-rcvbuf` (int)
   > This advanced parameter allows fine-tuning of network performance by adjusting the amount of data the socket can receive before signaling to the sender to slow down. Sets the socket receive buffer in bytes SO_RCVBUF.
   > Set to zero to use the default system value.
-- `reset-conn` (bool)
+
+* `reset-conn` (bool)
   > Set whether to send a TCP Reset to force the cleanup of the connection on the remote side when the server exits.
-- `chan-buffer-size` (int)
+
+* `chan-buffer-size` (int)
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
-- `disable-dnsparser"` (bool)
+  
+* `disable-dnsparser"` (bool)
   > Disable the minimalist DNS parser. Some JSON keys should not be available, such as `dns.id`, `dns.flags`, ...
-- `extended-support` (bool)
+
+* `extended-support` (bool)
   > Decode the extended extra field sent by DNScollector. If this setting is enabled, DNScollector will expect receiving the specific [protobuf structure](./../../dnsutils/extended_dnstap.proto) in the extra field, which must be sent by another DNS collector.
   > This field will contain additional metadata generated by various transformations such as filtering, ATags, and others.
-- `compression` (string)
-  > Specifies the compression algorithm to use.  Compression for DNStap messages: `none`, `gzip`, `lz4`, `snappy`, `zstd`.
+
+* `compression` (string)
+  > Specifies the compression algorithm to use.
+  > Compression for DNStap messages: `none`, `gzip`, `lz4`, `snappy`, `zstd`.
 
 Defaults:
 
@@ -67,20 +80,30 @@ For config examples, take a look to the following [one](../_examples/use-case-12
 
 Options:
 
-- `listen-ip` (str)
-  > Set the local address that the server will bind to. If not provided, the server will bind to all available network interfaces (0.0.0.0).
-- `listen-port` (int)
+* `listen-ip` (str)
+  > Set the local address that the server will bind to. 
+  > If not provided, the server will bind to all available network interfaces (0.0.0.0).
+
+* `listen-port` (int)
   > Set the local port that the server will listen on. If not provided, use the default port.
-- `sock-path` (str)
+
+* `sock-path` (str)
   > Specify the path for the Unix socket to be created.
-- `tls-support` (bool)
-  > Enables or disables TLS (Transport Layer Security) support. If set to true, TLS will be used for secure communication.
-- `tls-min-version` (str)
+
+* `tls-support` (bool)
+  > Enables or disables TLS (Transport Layer Security) support.
+  > If set to true, TLS will be used for secure communication.
+
+* `tls-min-version` (str)
   > Specifies the minimum TLS version that the server will support.
-- `cert-file` (str)
-  > Specifies the path to the certificate file to be used for TLS. This is a required parameter if TLS support is enabled.
-- `key-file`(str)
-  > Specifies the path to the key file corresponding to the certificate file. This is a required parameter if TLS support is enabled.
+
+* `cert-file` (str)
+  > Specifies the path to the certificate file to be used for TLS.
+  > This is a required parameter if TLS support is enabled.
+
+* `key-file`(str)
+  > Specifies the path to the key file corresponding to the certificate file. 
+  > This is a required parameter if TLS support is enabled.
 
 Defaults
 

docs/collectors/collector_fileingestor.md

@@ -14,15 +14,19 @@ For config examples, take a look to the following links:
 
 Options:
 
-- `watch-dir` (str) directory to watch for pcap files ingest.
+* `watch-dir` (str)
   > Specifies the directory where pcap files are monitored for ingestion.
-- `watch-mode` (str) watch the directory pcap or dnstap file.
-  >  `*.pcap` extension or dnstap stream with `*.fstrm` extension are expected.
-- `pcap-dns-port` (int) dns source or destination port.
-  > Expects a port number use for DNS communication.
-- `delete-after:` (boolean) delete pcap file after ingest.
+
+* `watch-mode` (str)
+  >  Watch the directory pcap or dnstap file. `*.pcap` extension or dnstap stream with `*.fstrm` extension are expected.
+
+* `pcap-dns-port` (int)
+  > Expects a source or destination port number use for DNS communication.
+
+* `delete-after:` (boolean)
   > Determines whether the pcap file should be deleted after ingestion.
-- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
+
+* `chan-buffer-size` (int)
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
 
 Defaults:

docs/collectors/collector_powerdns.md

@@ -4,26 +4,39 @@ Collector to logging protobuf streams from PowerDNS servers. The DNS-collector h
 
 Settings:
 
-- `listen-ip` (str) local address to bind to.
-  > Set the local address that the server will bind to. If not provided, the server will bind to all available network interfaces (0.0.0.0).
-- `listen-port` (int) local port to bind to.
+* `listen-ip` (str)
+  > Set the local address that the server will bind to. 
+  > If not provided, the server will bind to all available network interfaces (0.0.0.0).
+
+* `listen-port` (int)
   > Set the local port that the server will listen on. If not provided, use the default port.
-- `tls-support` (bool) set to true to enable TLS.
-  > Enables or disables TLS (Transport Layer Security) support. If set to true, TLS will be used for secure communication.
-- `tls-min-version` (str) Minimun TLS version to use.
+
+* `tls-support` (bool)
+  > Enables or disables TLS (Transport Layer Security) support.
+  > If set to true, TLS will be used for secure communication.
+
+* `tls-min-version` (str)
   > Specifies the minimum TLS version that the server will support.
-- `cert-file` (str) path to a certificate server file to use.
-  > Specifies the path to the certificate file to be used for TLS. This is a required parameter if TLS support is enabled.
-- `key-file`(str) path to a key server file to use.
-  > Specifies the path to the key file corresponding to the certificate file. This is a required parameter if TLS support is enabled.
-- `sock-rcvbuf` (int) sets the socket receive buffer in bytes SO_RCVBUF.
-  > This advanced parameter allows fine-tuning of network performance by adjusting the amount of data the socket can receive before signaling to the sender to slow down.
+
+* `cert-file` (str)
+  > Specifies the path to the certificate file to be used for TLS.
+  > This is a required parameter if TLS support is enabled.
+
+* `key-file`(str)
+  > Specifies the path to the key file corresponding to the certificate file.
+  > This is a required parameter if TLS support is enabled.
+
+* `sock-rcvbuf` (int)
+  > This advanced parameter allows fine-tuning of network performance by adjusting the amount of data the socket can receive before signaling to the sender to slow down. Sets the socket receive buffer in bytes SO_RCVBUF.
   > Set to zero to use the default system value.
-- `reset-conn` (bool) reset TCP connection on exit.
+
+* `reset-conn` (bool)
   > Set whether to send a TCP Reset to force the cleanup of the connection on the remote side when the server exits.
-- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
+
+* `chan-buffer-size` (int)
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
-- `add-dns-payload` (bool) generate and add fake DNS payload.
+
+* `add-dns-payload` (bool)
   > PowerDNS protobuf message does not contain a DNS payload; use this setting to add a raw DNS payload.
 
 Defaults:
@@ -46,15 +59,15 @@ Defaults:
 
 If you logs your DNS traffic in basic text format, you can use the specific directives:
 
-- `powerdns-tags[:INDEX]`: get all tags separated by comma, or the tag according to the provided INDEX
-- `powerdns-original-request-subnet`: get original request subnet like edns subclient
-- `powerdns-applied-policy`: get applied policy
-- `powerdns-applied-policy-hit`: get applied policy hit
-- `powerdns-applied-policy-kind`: get applied policy kind
-- `powerdns-applied-policy-trigger`: get applied policy trigger
-- `powerdns-applied-policy-type`: get applied policy type
-- `powerdns-metadata[:KEY]`: get  all metadata separated by comma or specific one if a valid [KEY](https://dnsdist.org/rules-actions.html#RemoteLogAction) is provided
-- `powerdns-http-version`: http version used with DoH queries
+* `powerdns-tags[:INDEX]`: get all tags separated by comma, or the tag according to the provided INDEX
+* `powerdns-original-request-subnet`: get original request subnet like edns subclient
+* `powerdns-applied-policy`: get applied policy
+* `powerdns-applied-policy-hit`: get applied policy hit
+* `powerdns-applied-policy-kind`: get applied policy kind
+* `powerdns-applied-policy-trigger`: get applied policy trigger
+* `powerdns-applied-policy-type`: get applied policy type
+* `powerdns-metadata[:KEY]`: get  all metadata separated by comma or specific one if a valid [KEY](https://dnsdist.org/rules-actions.html#RemoteLogAction) is provided
+* `powerdns-http-version`: http version used with DoH queries
 
 Configuration example:
 
@@ -110,7 +123,7 @@ Example to enable logging in your **pdns-recursor**
 lua-config-file=/etc/pdns-recursor/recursor.lua
 ```
 
-*****/etc/pdns-recursor/recursor.lua*
+*/etc/pdns-recursor/recursor.lua*
 
 ```lua
 protobufServer("<dnscollectorip>:6001", {exportTypes={pdns.A, pdns.AAAA, pdns.CNAME}})

docs/collectors/collector_tail.md

@@ -10,13 +10,16 @@ Enable the tail by provided the path of the file to follow
 
 Options:
 
-* `file-path`: (string) file to follow.
+* `file-path` (string)
   > Specifies the path to the file that will be monitored.
-* `time-layout`: (string)  Use the exact layout numbers.
+
+* `time-layout` (string)
   > Specifies the layout format for time representation, following the layout numbers defined in https://golang.org/src/time format.go.
-* `pattern-query`: (string) regexp pattern for queries.
+
+* `pattern-query` (string)
   > Specifies the regular expression pattern used to match queries.
-* `pattern-reply`: (string) regexp pattern for replies.
+
+* `pattern-reply` (string)
   > Specifies the regular expression pattern used to match replies.
 
 Defaults:

docs/collectors/collector_tzsp.md

@@ -5,11 +5,14 @@ Its primary purpose is to suppport DNS packet capture from Mikrotik brand device
 
 Options:
 
-- `listen-ip` (str) listen on ip.
-- `listen-port` (int) listening on port.
-- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
-  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
+* `listen-ip` (str)
+  > Set the local address that the server will bind to.
+
+* `listen-port` (int)
+  > Set the local port that the server will bind to.
 
+* `chan-buffer-size` (int)
+  > Specifies the maximum number of packets that can be buffered before dropping additional packets.
 
 Defaults:
 

docs/collectors/collector_xdp.md

@@ -16,9 +16,10 @@ sudo setcap cap_sys_resource,cap_net_raw,cap_perfmon+ep go-dnscollector
 
 Options:
 
-- `device` (str) interface name to sniff.
-  > Interface to use for XDP.
-- `chan-buffer-size` (int) incoming channel size, number of packet before to drop it.
+* `device` (str)
+  > Interface name to use for XDP sniffing.
+
+* `chan-buffer-size` (int)
   > Specifies the maximum number of packets that can be buffered before dropping additional packets.
 
 Defaults:

docs/extended_dnstap.md

@@ -5,6 +5,8 @@ These features can be only used between two `DNS-collector` instance.
 
 ## Compression
 
+> ref: https://github.com/dmachard/go-dnscollector/issues/490
+
 DNSTAP messages are highly compressible. They can be sent in reasonably large blocks, which enables significant compression for transmission over long-haul network links. While DNSTAP does not natively support compression, it seems not unreasonable that `DNS-collector` could have a configurable compression flag that would mark a stream as being compressed with one of the different models of compression that are supported in other areas of the code currently. This would allow a much more efficient transmission of DNSTAP-based messages through various components.
 
 The following codec are supported:
@@ -14,8 +16,6 @@ The following codec are supported:
 - snappy
 - std
 
-> ref: https://github.com/dmachard/go-dnscollector/issues/490
-
 ## Extended metadata
 
 DNSTAP message can be extended by incorporating additional metadata added through transformations, such as filtering, geo, ATags.
@@ -33,7 +33,7 @@ The following transformers are supported:
 
 DNSTAP messages contains sensitive data. `DNS-collector` have a configurable flag to enable TLS encryption.
 
-## Configuration examples
+## Configuration
 
 How to enable it on the collector side ?
 

docs/loggers/logger_clickhouse.md

@@ -5,11 +5,20 @@ Clickhouse client to remote ClickHouse server
 
 Options:
 
-- `url`: (string) Clickhouse server url
-- `user`: (string) Clickhouse database user
-- `password`: (string) Clickhouse database user password
-- `table`: (string) Clickhouse table name
-- `database`: (string) Clickhouse database name
+* `url` (string)
+  > Clickhouse server url
+
+* `user` (string)
+  > Clickhouse database user
+
+* `password` (string)
+  > Clickhouse database user password
+
+* `table` (string)
+  > Clickhouse table name
+
+* `database` (string)
+  > Clickhouse database name
 
 Defaults: