Docker

ethereum-etl export data every time write over file. Filebeat always reload file then ELK receive repeat records… So .env STARTBLOCK=01205866 ENDBLOCK=01205888 startetl.sh IP_PORT is go-ethereum-node1 outside ip & port. here is docker-machine source get file that path need to be careful #!/bin/bash IP_PORT=192.168.99.100:18545 ETH_METHOD=eth_blockNumber BLOCKNUMBER_JSON_HEX=$(curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"'$ETH_METHOD'","params":[],"id":1}' $IP_PORT | jq '.result' | tr -d '"') BLOCKNUMBER_DEX=$(printf "%08d\n" $BLOCKNUMBER_JSON_HEX) printf "\n===== Now Geth BlockerNumber =====\n" printf "HEX: %s\n" $BLOCKNUMBER_JSON_HEX printf "DEC: %s\n" $BLOCKNUMBER_DEX source .env #source ~/ethereum_etl/.env printf "\n===== .env BLOCK VARS =====\n" printf "STARTBLOCK: %s\n" $STARTBLOCK printf "ENDBLOCK: %s\n" $ENDBLOCK if [[ $BLOCKNUMBER_DEX =~ ^?[0-9]+$ ]] || [ -z $BLOCKNUMBER_DEX ]; then printf "NOT Number Geth BLOCKNUMBER, check Geth\n" exit fi if [[ $STARTBLOCK =~ ^?[0-9]+$ ]] || [ -z $STARTBLOCK ]; then printf "NOT Number .env STARTBLOCK, check .env\n" exit fi if [[ $ENDBLOCK =~ ^?[0-9]+$ ]] || [ -z $ENDBLOCK ]; then printf "NOT Number .env ENDBLOCK, check .env\n" exit fi #printf "\n%s\n" $BLOCKNUMBER_DEX #printf "%s" $STARTBLOCK #printf "\n%s\n" $((10#$BLOCKNUMBER_DEX - 10#$STARTBLOCK)) if (( (( 10#$BLOCKNUMBER_DEX - 10#$STARTBLOCK -1 )) > 0 )); then printf "\n===== Change .env =====\n" printf "startblock: %s => %s\n" $STARTBLOCK $((10#$ENDBLOCK+1)) printf "endblock: %s => %s\n" $ENDBLOCK $BLOCKNUMBER_DEX STARTBLOCK=$(printf "%08d\n" $((10#$ENDBLOCK+1)) ) echo STARTBLOCK=$STARTBLOCK > ~/ethereum_etl/.env echo ENDBLOCK=$BLOCKNUMBER_DEX >> .env #echo STARTBLOCK=$STARTBLOCK > ~/ethereum_etl/.env #echo ENDBLOCK=$BLOCKNUMBER_DEX >> ~ethereum_etl/.env fi docker-compose up --build ethereum_etl docker-compose.yml version: '3.3' services: go-ethereum-node1: build: context: go-ethereum-node1/ volumes: #- ./go-ethereum/keystore:/root/.ethereum/devchain/keystore:rw - ./go-ethereum-node1/genesis/poa_for_dev.json:/root/genesis/poa_for_dev.json:ro - alldata:/root/.ethereum/devchain - /etc/localtime:/etc/localtime:ro #- ./go-ethereum/log:/root/log:rw entrypoint: /root/start.sh ports: - "18545:8545" - "30313:30303" - "30313:30303/udp" networks: - etl ethereum_etl: build: context: ethereum-etl/ env_file: .env volumes: - alldata:/ethereum-etl/output:rw #- /root/go/src/github.com/ethereum/go-ethereum/build/bin/data:/ethereum-etl/ipc #restart: unless-stopped networks: - etl volumes: alldata: networks: etl: driver: bridge ethereum_etl DOCKERFILE FROM python:3.6-alpine MAINTAINER Eric Lim ENV PROJECT_DIR=ethereum-etl RUN apk add unzip RUN wget https://github.com/blockchain-etl/ethereum-etl/archive/develop.zip \ && unzip develop.zip && rm develop.zip RUN mv ethereum-etl-develop /$PROJECT_DIR WORKDIR /$PROJECT_DIR RUN apk add --no-cache gcc musl-dev #for C libraries: RUN pip install --upgrade pip && pip install -e /$PROJECT_DIR/ #CMD ["export_all", "-s", "01990000", "-e", "99999999", "-p", "http://xxx.xxx.xxx.xxx:8545", "-o", "output"] #CMD ["sh","-c", "echo startblock=$STARTBLOCK endblock=$ENDBLOCK"] CMD ["sh","-c","python ethereumetl export_all -s $STARTBLOCK -e $ENDBLOCK -p http://xxx.xxx.xxx.xxx:8545 -o output"] go-ethereum-node1 see https://sueboy.blogspot.com/search?q=poa ...

https://morphyhu.szitcare.com/wordpress/?p=1314 New docker use DOCKER-USER Important!! check host route & login docker container check route. and iptables -L -t nat POSTROUTING MASQUERADE 172.17.0.0/16 must same submask. EX: host route have 172.17.0.0, 172.18.0.0, 172.19.0.0, 172.20.0.0 docker insdie route use 172.18.0.0 iptables MASQUERADE use 172.17.0.0 Docker Internet is failed. So add iptables -t nat -A POSTROUTING -s 172.18.0.0/16 ! -o docker0 -j MASQUERADE #启动后默认增加的规则 iptables -N DOCKER iptables -N DOCKER-ISOLATION-STAGE-1 iptables -N DOCKER-ISOLATION-STAGE-2 iptables -N DOCKER-USER iptables -t nat -N DOCKER iptables -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER iptables -t nat -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE iptables -t nat -A DOCKER -i docker0 -j RETURN iptables -A FORWARD -j DOCKER-USER iptables -A FORWARD -j DOCKER-ISOLATION-STAGE-1 iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT iptables -A FORWARD -o docker0 -j DOCKER iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT iptables -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2 iptables -A DOCKER-ISOLATION-STAGE-1 -j RETURN iptables -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP iptables -A DOCKER-ISOLATION-STAGE-2 -j RETURN iptables -A DOCKER-USER -j RETURN #docker run --name smokeping -d -p 82:80 -e PUID=1000 -e PGID=1000 -e TZ=Asia/Shanghai -v /data/smokeping/data:/data -v /data/smokeping/config:/config linuxserver/smokeping #启动上述镜像后默认增加的规则 iptables -t nat -A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE iptables -t nat -A DOCKER ! -i docker0 -p tcp -m tcp --dport 82 -j DNAT --to-destination 172.17.0.2:80 iptables -A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT sudo iptable ...

docker-compose.yml version: '3.3' services: ethereum_etl: build: context: . env_file: .env volumes: - /var/log/hardblue/etl/:/ethereum-etl/output:rw #- /root/go/src/github.com/ethereum/go-ethereum/build/bin/data:/ethereum-etl/ipc #restart: unless-stopped networks: - etl networks: etl: driver: bridge .env STARTBLOCK=00000000 DOCKERFILE FROM python:3.6-alpine MAINTAINER Eric Lim ENV PROJECT_DIR=ethereum-etl RUN apk add unzip RUN wget https://github.com/blockchain-etl/ethereum-etl/archive/develop.zip \ && unzip develop.zip && rm develop.zip RUN mv ethereum-etl-develop /$PROJECT_DIR WORKDIR /$PROJECT_DIR RUN apk add --no-cache gcc musl-dev #for C libraries: RUN pip install --upgrade pip && pip install -e /$PROJECT_DIR/ #CMD ["export_all", "-s", "01990000", "-e", "99999999", "-p", "http://xxx.xxx.xxx.xxx:8545", "-o", "output"] #CMD ["sh","-c", "echo startblock=$STARTBLOCK endblock=$ENDBLOCK"] CMD ["sh","-c","python ethereumetl export_all -s $STARTBLOCK -e $ENDBLOCK -p http://xxx.xxx.xxx.xxx:8545 -o output"] crontab -e ...

https://stackoverflow.com/questions/33017329/where-is-a-log-file-with-logs-from-a-container get container id docker ps |grep < container name > get container log /var/lib/docker/containers/< container id >/< container id >-json.log

底層採用PVE(Proxmox)，建立KVM VM，KVM VM使用docker-machine或是自建用boot2docker iso做好backup，使用時直接restore docker 執行時 Restart policies：always，這樣VM開機就會自動執行，不需要另外再設定。 Volume 直接用docker-compose YAML volume減少權限問題，資料也會儲存在VM內，使用VM後，備份是以VM為主，不用擔心資料遺失 一個VM放多個docker、整組docker-compose 或 一個VM放單一功能docker可以自己決定 VM可以被快照保護，一但出問題，還原後、開完機，VM內的docker會自動起服務，資料也是存在的，減少docker backup&resotre問題 為了管理docker方便，第一次建立VM時，執行Portainer always，可以backup，將來每一個VM都可以利用圖形界面查看docker運作情況 缺點： 硬碟空間使用受限於KVM，擴充大小有可能需要重開機 線上移轉只能以VM，不能用docker，但docker設計上也不走線上移轉 為什麼需要用到docker呢？ 因為現在很多專案有直接提供docker使用，確實可以減少大量安裝時間， 但volume資料儲存、權限卻是最大的問題，上述架構可以解決這部份顧慮。