GeoIP stopped working as before 😕 #184

dusatvoj · 2021-05-22T18:16:16Z

Logstash information:

Please include the following information:

Logstash version (e.g. bin/logstash --version) 7.12.1
Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) - DEB package from elastic repo
How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes) - systemd
How was the Logstash Plugin installed - bundled in DEB package

JVM (e.g. java -version): java 11.0.11

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

JVM version (java -version)
JVM installation source (e.g. from the Operating System's package manager, from source, etc).
Value of the JAVA_HOME environment variable if set.

... I'm using Debian repo's openjdk-11

OS version (uname -a if on a Unix-like system): Linux elk-node-1 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
I have problem that cca 1/4 events from apache / nginx / haproxy have tag _geoip_lookup_failure (some of them are RFC 1918 IPs) but the rest of events not have that tag. Below is image of Index pattern data with data types. In past (I think ~3 months ago) there were different types of some fields (geoip.ip changed from ip to text and geoip.location changed from geo_point into object). This behavior causes Field type conflict ... Due to Field type conflict I can't load map layer and even on new index patterns (eg. I've recently added new haproxy into different space and pattern) I can't create map layers bcs of wrong field type for geoip mapping

My logstash setup (only these basic fields related to geoip plugin:

geoip {
    source => "event.clientip"
}

Expected behavior is "no change of datatype" and "working out of the box as before"

Steps to reproduce:

IDK ... maybe upgrade of ELK VM (including upgrade of ELK)? Can't rememeber (and I've found out this shit a week after it started happening)

Provide logs (if relevant):
N/A 😕

The text was updated successfully, but these errors were encountered:

hectorqin · 2021-05-24T09:32:59Z

I meet this problem too! Please resolve it，thanks!

es logs are as below:

[logstash-2021.05.24][0] mapping update rejected by primary
java.lang.IllegalArgumentException: mapper [geoip.ip] cannot be changed from type [ip] to [text]
        at org.elasticsearch.index.mapper.FieldMapper.checkIncomingMergeType(FieldMapper.java:281) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:266) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:46) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:516) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:476) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:503) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:309) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:476) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:304) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.Mapping.merge(Mapping.java:107) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.MapperService.mergeMappings(MapperService.java:523) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.MapperService.mergeMappings(MapperService.java:513) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:389) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:384) ~[elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:281) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:115) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:168) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.12.1.jar:7.12.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.12.1.jar:7.12.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]

jsvd · 2021-05-24T09:48:05Z

What version of the geoip are you using? (you can find with bin/logstash-plugin list --verbose geoip)

dusatvoj · 2021-05-24T09:51:01Z

@jsvd The version which is bundled with Logstash 7.12.1 DEB package ... logstash-filter-geoip (6.0.5)

dusatvoj · 2021-05-24T09:52:49Z

But It's happening about 3 months as I said above. I've created this issues when I found out that nobody cares (so maybe nobody knows about the potential bug or smth is wrong in my deployement)

jsvd · 2021-05-24T10:55:39Z

I'm trying to replicate on my side. Here's a few tests:

Index the same document with a geoip lookup with Logstash 7.9.0, then with Logstash 7.12.1. Using Elasticsearch 7.12.1

Logstash command used for both:

echo "1.1.1.1" | bin/logstash -e "filter { geoip { source => message  } } output { elasticsearch {} }"

No indexing issues, mapping of the "geoip.ip" field remained type "ip":

❯ curl -s localhost:9200/logstash-*/_mapping | jq ".\"logstash-2021.05.24-000001\".mappings.properties.geoip.properties.ip"
{
  "type": "ip"
}

✅ template correct, both documents indexed

Send failed lookup message first, then a valid lookup document:

echo "111111" | bin/logstash -e "filter { geoip { source => message  } } output { elasticsearch {} }"

then

echo "1.1.1.1" | bin/logstash -e "filter { geoip { source => message  } } output { elasticsearch {} }"

✅ template correct, both documents indexed

From there I can't replicate the template change of client.ip from "ip" to "text". is there sample data or documents that can be shared here? Are you able to replicate with a few curl requests or logstash configurations?

hectorqin · 2021-05-24T12:35:09Z

The version of this plugin is also 6.0.5

logstash configuration is as blew:

input {
    redis {
        data_type   => "list"
        key         => "logstash-log-list"
        host        => "127.0.0.1"
    }
}
filter {
    mutate {
        lowercase => [ "type" ]
    }
    if [clientPosition][cip] {
        geoip {
            source => "[clientPosition][cip]"
            database => "/usr/local/logstash/GeoLite2-City_20210507/GeoLite2-City.mmdb"
        }
    }
    if [timestamp] {
        date {
            match => ["timestamp", "yyyy-MM-dd HH:mm:ss", "yyyy-M-d HH:mm:ss", "ISO8601", "UNIX"]
        }
        mutate {
            remove_field => [ "timestamp" ]
        }
    }
    if ![type] {
        drop{}
    }
    if [type] =~ "test" {
        mutate { add_field => { "[@metadata][target_index]" => "logstash-%{type}-%{+YYYY.MM}" } }
    } else if [type] =~ "^dap" {
        mutate { add_field => { "[@metadata][target_index]" => "logstash-%{type}-%{+YYYY.MM.dd}" } }
    } else {
        mutate { add_field => { "[@metadata][target_index]" => "logstash-%{type}-%{+YYYY.MM}" } }
    }
}
output {
    stdout {
        codec => "rubydebug"
    }
    elasticsearch {
        hosts              => ["https://127.0.0.1:9200"]
        cacert             => "/usr/local/logstash/config/elasticsearch-ca.pem"
        index              => "%{[@metadata][target_index]}"
        document_type      => "%{type}"
        ilm_enabled        => false
        sniffing           => true
    }
}

Test data

127.0.0.1:6379> LPUSH logstash-log-list '{"type": "log", "clientPosition":{"cip":"223.101.75.206"}}'

Error comes again

Could not load '.aprc' from ENV['HOME']: no implicit conversion of nil into String
{
              "type" => "log",
    "clientPosition" => {
        "cip" => "223.101.75.206"
    },
             "geoip" => {
              "latitude" => 31.2449,
                    "ip" => "223.101.75.206",
          "country_name" => "China",
         "country_code2" => "CN",
              "location" => {
            "lon" => 121.3948,
            "lat" => 31.2449
        },
              "timezone" => "Asia/Shanghai",
         "country_code3" => "CN",
           "region_name" => "Shanghai",
             "city_name" => "Putuo",
             "longitude" => 121.3948,
           "region_code" => "SH",
        "continent_code" => "AS"
    },
        "@timestamp" => 2021-05-24T12:32:08.785Z,
          "@version" => "1"
}
[2021-05-24T12:32:09,493][WARN ][logstash.outputs.elasticsearch][main][6aceaa243065e491686cfa938155a077bb1ee954222efab248a34080ff331a5b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-log-2021.05", :routing=>nil, :_type=>"log"}, #<LogStash::Event:0x54ab69a7>], :response=>{"index"=>{"_index"=>"logstash-log-2021.05", "_type"=>"log", "_id"=>"4rJcnnkB8VlYSZD9cuLF", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [@version] cannot be changed from type [keyword] to [text]"}}}}

Check template is ok

curl -k -s -XGET  'https://127.0.0.1:9200/logstash-*/_mapping'  | jq ".\"logstash-log-2021.05\".mappings.properties.geoip.properties.ip"
{
  "type": "ip"
}

Template full

{
    "mappings" : {
      "dynamic_templates" : [
        {
          "message_field" : {
            "path_match" : "message",
            "match_mapping_type" : "string",
            "mapping" : {
              "norms" : false,
              "type" : "text"
            }
          }
        },
        {
          "string_fields" : {
            "match" : "*",
            "match_mapping_type" : "string",
            "mapping" : {
              "fields" : {
                "keyword" : {
                  "ignore_above" : 256,
                  "type" : "keyword"
                }
              },
              "norms" : false,
              "type" : "text"
            }
          }
        }
      ],
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "keyword"
        },
        "geoip" : {
          "dynamic" : "true",
          "properties" : {
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            }
          }
        }
      }
    }
  }

jsvd · 2021-05-24T13:13:36Z

This may be related to a change in the ES output related to the handling of the document_type logstash-plugins/logstash-output-elasticsearch#994

Can y'all confirm that you're explicitly setting document_type => value is the ES output? I can see that @hectorqin is

hectorqin · 2021-05-24T13:21:35Z

After i remove the document_type setting, it works, thanks!

dusatvoj · 2021-05-24T16:04:22Z

I do not have explicitly set document_type :/

elasticsearch {
    hosts => "https://127.0.0.1:9200"
    ssl => "true"
    cacert => "<CERT_PATH>"
    ssl_certificate_verification => "false"
    index => "logstash-%{[app][name]}-%{[app][env]}-%{+YYYY.MM.dd}"
    ilm_policy => "<JUST_HOT_TO_WARM_PHASE>"
    user => "XXXXXXXXXX"
    password => "XXXXXXXXXX"
}

I have trouble trying inline logstash filter bcs SSL and auth so I haven't tried that. If it's necessary I can try that but I need a bit of help with multiple options in inline filter definition.

dusatvoj · 2021-05-24T18:04:10Z

The elasticsearch output plugin gives mi this mapping (copied from kibana) ... no document_type set:

{
  "geoip": {
    "properties": {
      "city_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "continent_code": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "country_code2": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "country_code3": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "country_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "dma_code": {
        "type": "long"
      },
      "ip": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "latitude": {
        "type": "float"
      },
      "location": {
        "properties": {
          "lat": {
            "type": "float"
          },
          "lon": {
            "type": "float"
          }
        }
      },
      "longitude": {
        "type": "float"
      },
      "postal_code": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "region_code": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "region_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "timezone": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

jsvd · 2021-05-25T10:30:32Z

@dusatvoj in ES 7.x the type won't be included in the response unless ?include_type_name=true is included in the request.
For example:

❯ curl -s "localhost:9200/_mapping?pretty" | head -n5                     
{
  "logstash-2021.05.24-000001" : {
    "mappings" : {
      "dynamic_templates" : [
        {

❯ curl -s "localhost:9200/_mapping?include_type_name=true&pretty" | head -n5
{
  "logstash-2021.05.24-000001" : {
    "mappings" : {
      "_doc" : {
        "dynamic_templates" : [

dusatvoj · 2021-05-25T12:03:05Z

So you mean output like this? https://pastebin.com/nmhpxdFX

curl --insecure --user X:Y -XGET "https://localhost:9200/logstash-moo-hosting-all-2021.05.25/_mapping?include_type_name=true&pretty"

jsvd · 2021-05-25T12:57:20Z

Thank you can you also obtain the template with the type? e.g.

curl --insecure --user X:Y -XGET -s https://localhost:9200/_template/logstash?include_type_name=true

dusatvoj · 2021-05-25T21:05:39Z

@jsvd https://localhost:9200/_template/logstash?include_type_name=true&pretty:

{
  "logstash" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_doc" : {
        "dynamic_templates" : [
          {
            "message_field" : {
              "path_match" : "message",
              "mapping" : {
                "norms" : false,
                "type" : "text"
              },
              "match_mapping_type" : "string"
            }
          },
          {
            "string_fields" : {
              "mapping" : {
                "norms" : false,
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              },
              "match_mapping_type" : "string",
              "match" : "*"
            }
          }
        ],
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "geoip" : {
            "dynamic" : true,
            "properties" : {
              "ip" : {
                "type" : "ip"
              },
              "latitude" : {
                "type" : "half_float"
              },
              "location" : {
                "type" : "geo_point"
              },
              "longitude" : {
                "type" : "half_float"
              }
            }
          },
          "@version" : {
            "type" : "keyword"
          }
        }
      }
    },
    "aliases" : { }
  }
}

kaisecheng · 2021-11-04T14:21:51Z

@dusatvoj This issue will be transferred to elasticsearch-output as the problem is not related to geoip-filter. geoip-filter gives location details of the IP address and keeps the output geoip.ip as a string of ip. In your case, the field type in elasticsearch changed. It could be an issue of elasticsearch upgrade or elasticsearch template.

dusatvoj · 2021-11-04T14:27:22Z

@kaisecheng ok, thank you for info

kaisecheng · 2021-11-04T14:50:06Z

new issue created logstash-plugins/logstash-output-elasticsearch#1050

dusatvoj added the bug Something isn't working label May 22, 2021

kaisecheng added bug and removed bug Something isn't working bug labels May 28, 2021

roaksoax assigned kaisecheng Nov 2, 2021

kaisecheng mentioned this issue Nov 4, 2021

Field type of geoip.ip and geoip.location changed after ELK upgrade logstash-plugins/logstash-output-elasticsearch#1050

Open

kaisecheng closed this as completed Nov 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GeoIP stopped working as before 😕 #184

GeoIP stopped working as before 😕 #184

dusatvoj commented May 22, 2021 •

edited

Loading

hectorqin commented May 24, 2021

jsvd commented May 24, 2021

dusatvoj commented May 24, 2021

dusatvoj commented May 24, 2021

jsvd commented May 24, 2021 •

edited

Loading

hectorqin commented May 24, 2021 •

edited

Loading

jsvd commented May 24, 2021 •

edited

Loading

hectorqin commented May 24, 2021

dusatvoj commented May 24, 2021

dusatvoj commented May 24, 2021

jsvd commented May 25, 2021

dusatvoj commented May 25, 2021

jsvd commented May 25, 2021

dusatvoj commented May 25, 2021 •

edited

Loading

kaisecheng commented Nov 4, 2021

dusatvoj commented Nov 4, 2021

kaisecheng commented Nov 4, 2021

GeoIP stopped working as before 😕 #184

GeoIP stopped working as before 😕 #184

Comments

dusatvoj commented May 22, 2021 • edited Loading

hectorqin commented May 24, 2021

jsvd commented May 24, 2021

dusatvoj commented May 24, 2021

dusatvoj commented May 24, 2021

jsvd commented May 24, 2021 • edited Loading

hectorqin commented May 24, 2021 • edited Loading

jsvd commented May 24, 2021 • edited Loading

hectorqin commented May 24, 2021

dusatvoj commented May 24, 2021

dusatvoj commented May 24, 2021

jsvd commented May 25, 2021

dusatvoj commented May 25, 2021

jsvd commented May 25, 2021

dusatvoj commented May 25, 2021 • edited Loading

kaisecheng commented Nov 4, 2021

dusatvoj commented Nov 4, 2021

kaisecheng commented Nov 4, 2021

dusatvoj commented May 22, 2021 •

edited

Loading

jsvd commented May 24, 2021 •

edited

Loading

hectorqin commented May 24, 2021 •

edited

Loading

jsvd commented May 24, 2021 •

edited

Loading

dusatvoj commented May 25, 2021 •

edited

Loading