Grep command produces different results

[franklyIdontgiveadamn]

Running a grep command with the same search phrases produces different results from your php code. The php total for "failed" is the same, but grep shows 220 sent witnesses, while the php shows only 213. No witnesses were sent between runs.

/var/log/miner # date
Thu Jun 9 03:08:18 UTC 2022
/var/log/miner # /opt/miner/processlogs.php

Using logs in folder /var/log/miner/

General Witnesses Overview

Total witnesses = 213 (9.65/hour)
Succesfully delivered = 209 (98.12%)
Failed = 4 (1.88%)
/var/log/miner # date
Thu Jun 9 03:09:09 UTC 2022
/var/log/miner # grep "miner_onion_server_light:decrypt:" console.log* | grep "sending witness at RSSI"|wc -l
220
/var/log/miner # grep "@miner_poc_grpc_client_statem:send_report" console.log* | grep "failed to submit report"|wc -l
4
/var/log/miner # grep "miner_onion_server_light:decrypt:" console.log|grep "sending witness at RSSI"|sort|tail
2022-06-09 01:03:58.969 153 [info] <0.2102.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -110, Frequency: 904.7, SNR: -1.8
2022-06-09 01:09:52.240 153 [info] <0.2102.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -88, Frequency: 904.1, SNR: 10.8
2022-06-09 01:16:29.374 201 [info] <0.2096.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -103, Frequency: 904.5, SNR: 6.5
2022-06-09 01:22:46.038 201 [info] <0.2096.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -97, Frequency: 904.3, SNR: 5.0
2022-06-09 01:23:22.905 201 [info] <0.2096.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -118, Frequency: 903.9, SNR: -7.5
2022-06-09 02:17:19.792 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -116, Frequency: 904.9, SNR: -7.5
2022-06-09 02:22:54.526 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -115, Frequency: 903.9, SNR: -6.0
2022-06-09 02:24:42.024 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -110, Frequency: 904.9, SNR: -4.0
2022-06-09 02:50:24.910 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -115, Frequency: 903.9, SNR: -3.8
2022-06-09 03:00:33.104 203 [info] <0.2075.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -119, Frequency: 904.7, SNR: -11.8
/var/log/miner # date
Thu Jun 9 03:11:25 UTC 2022

[franklyIdontgiveadamn]

processlogs.php fails to count all witness reports compared with grepping.
I've tried to indicate there is a problem with your code, but perhaps I didn't present conclusive evidence, so here is another try at proving there is a problem.

I ran "processlogs.php -l" on my sensecap m1, and it produced a list of 108 witness reports, including 2 fails.
See: http://ix.io/41Vq

Grep produced a report of 109 witness reports, including the two fails.
See: http://ix.io/41Vj

Your code is missing one successful submission in this example, and it is in the middle of the data set. You can compare time stamps to confirm that it's missing. This is the missing case:

2022-07-08 00:13:30.808 202 [info] <0.2143.0>@miner_onion_server_light:decrypt:{230,13} sending witness at RSSI: -115, Frequency: 904.9, SNR: -4.0

I run 1 grep to tally all reports, and a different grep (not shown here, but shown in my first comment) to capture the failed submissions.

Comparing the reports is a little tricky, because my failed witness submissions are shown as of the time they were submitted, but your report shows the time when they failed, so the order is not quite the same for the failed submissions.

I've compared using grep and your code many times, and almost every time there is a discrepancy in the total submitted, although the number of fails has always agreed. I believe your code is missing some of the valid submissions.

[LTBAA]

how did you run "processlogs.php -l" on your sensecap m1? the command does not work with mine

[franklyIdontgiveadamn]

I didn't do anything special. I opened a shell in the miner, then installed php-cli and used wget as below:
/opt/miner # apk add php-cli
/opt/miner # wget -O processlogs.php https://raw.githubusercontent.com/inigoflores/helium-miner-log-analyzer/main/processlogs.php; chmod a+x processlogs.php

Then:
/opt/miner # ./processlogs.php -l

If you're new to linux, perhaps you didn't use the ./ before processlogs.php?

I recommend that you try the grep commands outside the balena shell instead these days. processlogs.php just showed 543 witness reports, but grep shows 595 currently. Now that chain sync is over, the discrepancy problem is worse.

root@6f56cd9:~# grep "miner_onion_server_light:decrypt:" /mnt/data/docker/volumes/1901628_miner-log/_data/console.log* | grep "sending witness at RSSI"|cut --complement -d ":" -f1|sort|wc -l

root@6f56cd9:~# grep "@miner_poc_grpc_client_statem:send_report" /mnt/data/docker/volumes/1901628_miner-log/_data/console.log* | grep "failed to submit report"|cut --complement -d ":" -f1|sort|wc -l