elasticsearch数据持久化

elasticSearch数据持久化，默认情况不启用数据保存，故数据一般会几分钟就消失，按以下步骤保存索引数据。

#不同的集群名字不能相同。
cluster.name: es_vm_test
node.name: vmmaster
network.host: 0.0.0.0
http.port: 9200
#数据索引保存
path.data: /home/abc/elk-5.5.1/elkdata/data
path.logs: /home/abc/elk-5.5.1/elkdata/log
#关闭登录验证
xpack.security.enabled: false

logstash抓取nginx日志

以下是基于elk+lnmp开源进行测试验证。
也可以参考官网的实现方法：https://kibana.logstash.es/content/logstash/plugins/codec/json.html
https://kibana.logstash.es/content/logstash/plugins/codec/multiline.html
在官网文档中，有较多应用场景：
https://kibana.logstash.es/content/
https://kibana.logstash.es/content/logstash/examples/

1.抓取nginx日志

input {
    file {
        # path => ["/home/wwwlogs/h5.vim.vim.com.log", "/home/wwwlogs/h5.vim.vim.com2.log"]
	path => "/home/wwwlogs/h5.vim.vim.com.log"
        exclude => "*.zip"
        type => "java"
        add_field => [ "domain", "h5.vim.vim.com" ]
        codec => multiline {
                      pattern => "^\s+"
                      what => previous
              }
    }
    file {
        # path => ["/home/wwwlogs/h5.api.vim.vim.com.log", "/home/wwwlogs/h5.api.vim.vim.com2.log"]
	path => "/home/wwwlogs/h5.api.vim.vim.com.log"
        exclude => ["*.zip", "*.gz"]
        type => "java"
        add_field => [ "domain", "h5.api.vim.vim.com" ]
        codec => multiline {
                        pattern => "^\s+"
                        what => previous
                 }
    }
}
filter {

}
output {
    stdout { 
		codec => rubydebug 
	}
    elasticsearch {
        hosts => ["0.0.0.0:9200"]
        index => "logstash-%{domain}-%{+YYYY.MM.dd}"
    }
}

2.定期清理索引

#!/bin/bash

# --------------------------------------------------------------
# This script is to delete ES indices older than specified days.
# Version: 1.0
# --------------------------------------------------------------

function usage() {
        echo "Usage: `basename $0` -s ES_SERVER -d KEEP_DAYS [-w INTERVAL]"
}


PREFIX='logstash-'
WAITTIME=2
NOW=`date  +%s.%3N`
LOGPATH=/apps/logs/elasticsearch


while getopts d:s:w: opt
do
        case $opt in
        s) SERVER="$OPTARG";;
        d) KEEPDAYS="$OPTARG";;
        w) WAITTIME="$OPTARG";;
        *) usage;;
        esac
done

if [ -z "$SERVER" -o -z "$KEEPDAYS" ]; then
        usage
fi

if [ ! -d $LOGPATH ]; then
        mkdir -p $LOGPATH
fi


INDICES=`curl -s $SERVER/_cat/indices?h=index | grep -P '^logstash-.*\d{4}.\d{2}.\d{2}' | sort`
for index in $INDICES
do
        date=`echo $index | awk -F '-' '{print $NF}' | sed 's/\./-/g' | xargs -I{} date -d {} +%s.%3N`
        delta=`echo "($NOW-$date)/86400" | bc`
        if [ $delta -gt $KEEPDAYS ]; then
                echo "deleting $index" | tee -a $LOGPATH/es_delete_indices.log
                curl -s -XDELETE $SERVER/$index | tee -a $LOGPATH/es_delete_indices.log
                echo | tee -a $LOGPATH/es_delete_indices.log
                sleep $WAITTIME
        fi
done

intel推出dpdk网络开发包

intel推出的dpdk网络开发包
http://dpdk.org/

DPDK is a set of libraries and drivers for fast packet processing.
It is designed to run on any processors. The first supported CPU was Intel x86 and it is now extended to IBM POWER and ARM.
It runs mostly in Linux userland. A FreeBSD port is available for a subset of DPDK features.
DPDK is an Open Source BSD licensed project. The most recent patches and enhancements, provided by the community, are available in master branch.
Main libraries

multicore framework
huge page memory
ring buffers
poll-mode drivers for networking , crypto and eventdev

vmware在Centos7共享文件夹

1.重装vmware-tools工具,把VMwareTools-10.0.1-3160059.tar.gz包拷到任意目录并解压。

执行./vmware-install.pl安装，后续过程不断按回车，使用默认值，即可。

2.在root权限下，执行

mount -t vmhgfs .host:/ /mnt/hgfs
如果报错 ERROR: can not mount filesystem: No such device，则改用如下命令：
vmhgfs-fuse .host:/ /mnt/hgfs，这个是当前测试通过的

3.配置启动项【vim /etc/rc.local】，使用随系统启动，默认开启。

vmhgfs-fuse .host:/ /mnt/hgfs
cd /mnt/hgfs目录下，检查是否OK.需要root权限

ELK的启动和停止脚本 

#!/bin/sh

path_base=`pwd`
path_elastic="$path_base/elasticsearch-5.5.1/bin/elasticsearch"
path_kibana="$path_base/kibana-5.5.1/bin/kibana"
path_logstash="$path_base/logstash-5.5.1/bin/logstash"

mode=$1

kibana_process=`ps -ef | grep "kibana"| grep -v grep`
elastic_process=`ps -ef | grep "elasticsearch"| grep -v grep|grep -v controller`

case "$mode" in
   'start')
	echo "it's ready to start op...."
	if test -n "$elastic_process"; then
		echo "---the elasticsearch had already started.."
	else
		`$path_elastic -d`
	fi

	if test -n "$kibana_process"; then
		echo "---the kibana had already started.."
	else
		`nohup $path_kibana > /dev/null 2>&1 &`
	fi
	echo 'success to start.'
	echo '---1--you can test logstash by way bellow----'
	echo 'cd logstash-5.5.1 && bin/logstash -f ./config/test.conf && cd ..'
	echo 'or cd logstash-5.5.1 && bin/logstash -f ./config/h5.api.vip.com.conf && cd ..'
	;;
   'stop')
	echo "it's ready to check process..."
	if test -n "$kibana_process"; then
		echo "had find kibana process.."
		`echo $kibana_process | awk '{print ($2)}' | xargs kill -9`
	fi
	if test -n "$elastic_process"; then
		echo "had find elasticsearch process.."
		`echo $elastic_process | awk '{print ($2)}' | xargs kill -9`
	fi
	echo 'success to kill.'
	;;
    *)
	basename=`basename "$0"`
	echo "Usage: $basename  {start|stop}  [ ELK server options ]"
	exit 1
	;;
esac
exit 1

数据库性能压测 

mysqltest]$ mysqlslap -S /tmp/mysqltest/mysql.sock -uroot --create=/tmp/mysqltest/user_cart.sql --create-schema=test --query=/tmp/mysqltest/cart1.sql --concurrency=1024 --iterations=3
Benchmark
Average number of seconds to run all queries: 59.774 seconds
Minimum number of seconds to run all queries: 59.570 seconds
Maximum number of seconds to run all queries: 60.040 seconds
Number of clients running queries: 1024
Average number of queries per client: 954
QPS=16343