ethereum-etl export data every time write over file. Filebeat always reload file then ELK receive repeat records… So .env STARTBLOCK=01205866 ENDBLOCK=01205888 startetl.sh IP_PORT is go-ethereum-node1 outside ip & port. here is docker-machine source get file that path need to be careful #!/bin/bash IP_PORT=192.168.99.100:18545 ETH_METHOD=eth_blockNumber BLOCKNUMBER_JSON_HEX=$(curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"'$ETH_METHOD'","params":[],"id":1}' $IP_PORT | jq '.result' | tr -d '"') BLOCKNUMBER_DEX=$(printf "%08d\n" $BLOCKNUMBER_JSON_HEX) printf "\n===== Now Geth BlockerNumber =====\n" printf "HEX: %s\n" $BLOCKNUMBER_JSON_HEX printf "DEC: %s\n" $BLOCKNUMBER_DEX source .env #source ~/ethereum_etl/.env printf "\n===== .env BLOCK VARS =====\n" printf "STARTBLOCK: %s\n" $STARTBLOCK printf "ENDBLOCK: %s\n" $ENDBLOCK if [[ $BLOCKNUMBER_DEX =~ ^?[0-9]+$ ]] || [ -z $BLOCKNUMBER_DEX ]; then printf "NOT Number Geth BLOCKNUMBER, check Geth\n" exit fi if [[ $STARTBLOCK =~ ^?[0-9]+$ ]] || [ -z $STARTBLOCK ]; then printf "NOT Number .env STARTBLOCK, check .env\n" exit fi if [[ $ENDBLOCK =~ ^?[0-9]+$ ]] || [ -z $ENDBLOCK ]; then printf "NOT Number .env ENDBLOCK, check .env\n" exit fi #printf "\n%s\n" $BLOCKNUMBER_DEX #printf "%s" $STARTBLOCK #printf "\n%s\n" $((10#$BLOCKNUMBER_DEX - 10#$STARTBLOCK)) if (( (( 10#$BLOCKNUMBER_DEX - 10#$STARTBLOCK -1 )) > 0 )); then printf "\n===== Change .env =====\n" printf "startblock: %s => %s\n" $STARTBLOCK $((10#$ENDBLOCK+1)) printf "endblock: %s => %s\n" $ENDBLOCK $BLOCKNUMBER_DEX STARTBLOCK=$(printf "%08d\n" $((10#$ENDBLOCK+1)) ) echo STARTBLOCK=$STARTBLOCK > ~/ethereum_etl/.env echo ENDBLOCK=$BLOCKNUMBER_DEX >> .env #echo STARTBLOCK=$STARTBLOCK > ~/ethereum_etl/.env #echo ENDBLOCK=$BLOCKNUMBER_DEX >> ~ethereum_etl/.env fi docker-compose up --build ethereum_etl docker-compose.yml version: '3.3' services: go-ethereum-node1: build: context: go-ethereum-node1/ volumes: #- ./go-ethereum/keystore:/root/.ethereum/devchain/keystore:rw - ./go-ethereum-node1/genesis/poa_for_dev.json:/root/genesis/poa_for_dev.json:ro - alldata:/root/.ethereum/devchain - /etc/localtime:/etc/localtime:ro #- ./go-ethereum/log:/root/log:rw entrypoint: /root/start.sh ports: - "18545:8545" - "30313:30303" - "30313:30303/udp" networks: - etl ethereum_etl: build: context: ethereum-etl/ env_file: .env volumes: - alldata:/ethereum-etl/output:rw #- /root/go/src/github.com/ethereum/go-ethereum/build/bin/data:/ethereum-etl/ipc #restart: unless-stopped networks: - etl volumes: alldata: networks: etl: driver: bridge ethereum_etl DOCKERFILE FROM python:3.6-alpine MAINTAINER Eric Lim ENV PROJECT_DIR=ethereum-etl RUN apk add unzip RUN wget https://github.com/blockchain-etl/ethereum-etl/archive/develop.zip \ && unzip develop.zip && rm develop.zip RUN mv ethereum-etl-develop /$PROJECT_DIR WORKDIR /$PROJECT_DIR RUN apk add --no-cache gcc musl-dev #for C libraries: RUN pip install --upgrade pip && pip install -e /$PROJECT_DIR/ #CMD ["export_all", "-s", "01990000", "-e", "99999999", "-p", "http://xxx.xxx.xxx.xxx:8545", "-o", "output"] #CMD ["sh","-c", "echo startblock=$STARTBLOCK endblock=$ENDBLOCK"] CMD ["sh","-c","python ethereumetl export_all -s $STARTBLOCK -e $ENDBLOCK -p http://xxx.xxx.xxx.xxx:8545 -o output"] go-ethereum-node1 see https://sueboy.blogspot.com/search?q=poa ...

https://github.com/sueboy19/oldelk

{"type":"log","@timestamp":"2019-01-21T08:57:51Z","tags":["status","plugin:elasticsearch@6.5.2","error"],"pid":1,"state":"red","message":"Status changed from yellow to red - Request Timeout after 3000ms","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"} 1. First use oss FROM docker.elastic.co/elasticsearch/elasticsearch-oss:6.5.2 FROM docker.elastic.co/kibana/kibana-oss:6.5.2 So Not X-pack problem 2. Truly Problem is connect to elasticsearch failed Even login kibana docker then ping elasticsearch docker that get response. But kibana logs always get this error message. kibana website message “Kibana server is not ready yet” Try to get docker elasticsearch ip, change docker kibana.yml like example： ...

https://morphyhu.szitcare.com/wordpress/?p=1314 New docker use DOCKER-USER Important!! check host route & login docker container check route. and iptables -L -t nat POSTROUTING MASQUERADE 172.17.0.0/16 must same submask. EX: host route have 172.17.0.0, 172.18.0.0, 172.19.0.0, 172.20.0.0 docker insdie route use 172.18.0.0 iptables MASQUERADE use 172.17.0.0 Docker Internet is failed. So add iptables -t nat -A POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE #启动后默认增加的规则 iptables -N DOCKER iptables -N DOCKER-ISOLATION-STAGE-1 iptables -N DOCKER-ISOLATION-STAGE-2 iptables -N DOCKER-USER iptables -t nat -N DOCKER iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER iptables -t nat -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE iptables -t nat -A DOCKER -i docker0 -j RETURN iptables -A FORWARD -j DOCKER-USER iptables -A FORWARD -j DOCKER-ISOLATION-STAGE-1 iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT iptables -A FORWARD -o docker0 -j DOCKER iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT iptables -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 iptables -A DOCKER-ISOLATION-STAGE-1 -j RETURN iptables -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP iptables -A DOCKER-ISOLATION-STAGE-2 -j RETURN iptables -A DOCKER-USER -j RETURN #docker run --name smokeping -d -p 82:80 -e PUID=1000 -e PGID=1000 -e TZ=Asia/Shanghai -v /data/smokeping/data:/data -v /data/smokeping/config:/config linuxserver/smokeping #启动上述镜像后默认增加的规则 iptables -t nat -A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE iptables -t nat -A DOCKER ! -i docker0 -p tcp -m tcp --dport 82 -j DNAT --to-destination 172.17.0.2:80 iptables -A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT sudo iptable ...

https://github.com/taskrabbit/elasticsearch-dump Output elasticsearch->json docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-logs --output=/tmp/etl-logs.json --type=mapping docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-transactions --output=/tmp/etl-transactions.json --type=mapping docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-receipts --output=/tmp/etl-receipts.json --type=mapping docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-contracts --output=/tmp/etl-contracts.json --type=mapping docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-blocks --output=/tmp/etl-blocks.json --type=mapping docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-logs --output=/tmp/etl-logs-data.json --type=data docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-transactions --output=/tmp/etl-transactions-data.json --type=data docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-receipts --output=/tmp/etl-receipts-data.json --type=data docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-contracts --output=/tmp/etl-contracts-data.json --type=data docker run --rm -ti -u root -v /mnt/sda1/var/lib/docker/volumes/xxxxxxx/_data/data:/tmp taskrabbit/elasticsearch-dump --input=http://192.168.99.101:9200/filebeat-6.5.4-etl-blocks --output=/tmp/etl-blocks-data.json --type=data Innput json->elasticsearch docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-contracts.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-contracts --type=mapping docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-contracts-data.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-contracts --type=data docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-receipts.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-receipts --type=mapping docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-receipts-data.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-receipts --type=data docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-transactions.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-transactions --type=mapping docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-transactions-data.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-transactions --type=data docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-blocks.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-blocks --type=mapping docker run --rm -ti -u root -v data:/tmp taskrabbit/elasticsearch-dump --input=/tmp/etl-blocks-data.json --output=http://xxx.xxx.xxx.xxx:9200/filebeat-6.5.4-etl-blocks --type=_data