Logstash

Filebeat + Elasticsearch + Kibana 轻量日志收集与展示系统 https://wzyboy.im/post/1111.html?utm_source=tuicool&utm_medium=referral 提到 beat -> logstash -> elk 可以 beat -> elk ingest plugs ( Elasticsearch Ingest Node ) Elasticsearch Ingest Node 是 Elasticsearch 5.0 起新增的功能。在 Ingest Node 出现之前，人们通常会在 ES 前置一个 Logstash Indexer，用于对数据进行预处理。有了 Ingest Node 之后，Logstash Indexer 的大部分功能就可以被它替代了，grok, geoip 等 Logstash 用户所熟悉的处理器，在 Ingest Node 里也有。对于数据量较小的 ES 用户来说，省掉一台 Logstash 的开销自然是令人开心的，对于数据量较大的 ES 用户来说，Ingest Node 和 Master Node, Data Node 一样也是可以分配独立节点并横向扩展的，也不用担心性能瓶颈。 目前 Ingest Node 已支持数十种处理器，其中的 script 处理器具有最大的灵活性。 与 /_template 类似，Ingest API 位于 /_ingest 下面。用户将 pipeline 定义提交之后，在 Beats 中即可指定某 pipeline 为数据预处理器。 ...

1、filebeat /var/log/secure 2、 filter { grok { #type => "syslog" match => ["message", "%{SYSLOGBASE} Failed password for (invalid user |)%{USERNAME:username} from %{IP:src_ip} port %{BASE10NUM:port} ssh2"] add_tag => "ssh_brute_force_attack" } grok { #type => "syslog" match => ["message", "%{SYSLOGBASE} Accepted password for %{USERNAME:username} from %{IP:src_ip} port %{BASE10NUM:port} ssh2"] add_tag => "ssh_sucessful_login" } geoip { source => "src_ip" target => "geoip" add_tag => [ "ssh-geoip" ] add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] add_field => [ "geoipflag", "true" ] } }

filter json { source => “message” } This mean is Try to use json format transfer log, then put some data to message filed. So some filed just be setting, and some data set to message. ．Use this to check mach and log https://grokconstructor.appspot.com/do/match https://blog.johnwu.cc/article/elk-logstash-grok-filter.html https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns This is geth log for example A： INFO [11-14|09:58:17.730] Generating DAG in progress epoch=1 percentage=99 elapsed=4m8.643s INFO [11-15|01:41:33.455] Generating DAG in progress epoch=1 percentage=9 elapsed=27.614s B： INFO [11-15|01:19:44.590] Loaded most recent local fast block number=0 hash=656134…58fded td=1 age=49y7mo1h, Loaded most recent local fast block ...

https://www.rosehosting.com/blog/install-and-configure-the-elk-stack-on-ubuntu-16-04/ https://www.elastic.co/guide/en/logstash/current/configuration.html https://dotblogs.com.tw/supershowwei/2016/05/25/185741 install finish 1、/etc/logstash/conf.d/ put some logstash conf 2、ubuntu have logstash listen error, so nano /etc/logstash/startup.options LS_USER = root 3、/usr/share/logstash/bin# ./system-install reuse LS_USER for config 注意： mutate { add_field => { “logTime” => “%{+YYYY-MM-dd} %{time}” }