分类目录归档:mongo

mongo分片及副本集搭建例子

https://github.com/senssei/mongo-cluster-docker

基于上面仓库,改造了一个副本集。
https://github.com/kxtry/mongo-cluster-docker

1.副本集docker-compose脚本

version: '3'
services:
    mgoprimary:
        container_name: "mgoprimary"
        image: mongo:4.2
        ports:
            - "27017:27017"
        command: mongod --replSet vmlocal --port 27017  --oplogSize 16
        links:
            - mgoreplica1:mgoreplica1
            - mgoreplica2:mgoreplica2
        restart: always
        extra_hosts:
            - "mgo.db.com:{{extern_ip}}"
        volumes:
            - /etc/localtime:/etc/localtime:ro
            - /etc/timezone:/etc/timezone:ro
            - ./data/mgoprimary/data:/data/db
 
    mgoreplica1:
        container_name: "mgoreplica1"
        image: mongo:4.2
        ports:
            - "27018:27017"
        command: mongod --replSet vmlocal --port 27017  --oplogSize 16
        restart: always
        extra_hosts:
            - "mgo.db.com:{{extern_ip}}" 
        volumes:
            - /etc/localtime:/etc/localtime:ro
            - /etc/timezone:/etc/timezone:ro
            - ./data/mgoreplica1/data:/data/db
 
    mgoreplica2:
        container_name: "mgoreplica2"
        image: mongo:4.2
        ports:
            - "27019:27017"
        command: mongod --replSet vmlocal --port 27017  --oplogSize 16        
        restart: always
        extra_hosts:
            - "mgo.db.com:{{extern_ip}}"
        volumes:
            - /etc/localtime:/etc/localtime:ro
            - /etc/timezone:/etc/timezone:ro
            - ./data/mgoreplica2/data:/data/db
 
    setup-vmlocal:
        container_name: "setup-vmlocal"
        image: mongo:4.2
        depends_on:
            - "mgoprimary"
            - "mgoreplica1"
            - "mgoreplica2"
        extra_hosts:
            - "mgo.db.com:{{extern_ip}}"
        links:
            - mgoprimary:mgoprimary
            - mgoreplica1:mgoreplica1
            - mgoreplica2:mgoreplica2
        volumes:
            - ./scripts:/scripts
        environment: 
            - MONGO1=mgo.db.com:27017
            - MONGO2=mgo.db.com:27018
            - MONGO3=mgo.db.com:27019
            - RS=vmlocal
        entrypoint: [ "/scripts/setup.sh" ]

3.与docker-compose.yml相同子目录下的./scripts/setup.sh脚本

#!/bin/bash 
 
mongodb1="${MONGO1}"
mongodb2="${MONGO2}"
mongodb3="${MONGO3}"
 
echo "Waiting for startup.."
until mongo --host ${mongodb1} --eval 'quit(db.runCommand({ ping: 1 }).ok ? 0 : 2)' &>/dev/null; do
  printf '.'
  sleep 1
done
 
echo "Started.."
 
echo setup.sh time now: `date +"%T" `
mongo --host ${mongodb1} <<EOF
   var cfg = {
        "_id": "${RS}",
        "protocolVersion": 1,
        "members": [
            {
                "_id": 0,
                "host": "${mongodb1}"
            },
            {
                "_id": 1,
                "host": "${mongodb2}"
            },
            {
                "_id": 2,
                "host": "${mongodb3}"
            }
        ]
    };
    rs.initiate(cfg, { force: true });
    rs.reconfig(cfg, { force: true });
EOF

3.构建脚本build_cluster.sh

#!/bin/sh                                                                                                                                                                                                   
 
path_current=`pwd`                                                                                                                                                                                          
path_script=$(cd "$(dirname "$0")"; pwd)                                                                                                                                                                    
 
mode=$1                                                                                                                                                                                                     
extern_ip=$2                                                                                                                                                                                                
 
case "$mode" in                                                                                                                                                                                             
     'build')                                                                                                                                                                                               
        if [ -f $path_script/docker-compose.yml ]; then                                                                                                                                                     
           echo 'the docker-compose.yml had been exist. if you want to continue, remove it.'                                                                                                                
           exit 1                                                                                                                                                                                           
        fi                                                                                                                                                                                                  
        /bin/cp $path_script/docker-compose.yml.template $path_script/docker-compose.yml && echo "$path_script/docker-compose.yml" | xargs /bin/sed -i "s#{{extern_ip}}#$extern_ip#g"                       
        echo 'success to build the file.'                                                                                                                                                                   
     ;;                                                                                                                                                                                                     
     *)                                                                                                                                                                                                     
       basename=`basename "$0"`                                                                                                                                                                             
       echo "Usage: $basename  {build}{extern_ip}"                                                                                                                                                          
       echo "---$basename build 192.168.10.104"                                                                                                                                                             
     ;;                                                                                                                                                                                                     
esac

mongo的测试脚本

conn = new Mongo("mongodb://wps_credit:2f25da558f6e79c452a61bea4a8d762f@10.100.2.216:27017,10.100.2.217:27017,10.100.2.218/wps_credit?maxPoolSize=300&replicaSet=kae-mongo-test42");
db = conn.getDB("wps_credit");
 
var t1 = new Date();
printjson(db.getCollectionNames());
 
var t2 = new Date().getTime();
printjson(t1, t2, t2 - t1);

mongodb脚本执行

1.编写脚本:job.js,内容如下:

conn = new Mongo("mongodb://abc:b=abc@10.0.1.183:27017,10.0.1.184:27017/wps_credit?maxPoolSize=300&replicaSet=c45134ec-6a5f-461e-8745-7081b46b0d87");
db = conn.getDB("abc");
 
var result = db.transaction_5.find().limit(10);
while(result.hasNext()) {
    printjson(result.next());
}
var tm = new Date();
printjson(db.transaction_6.count());
db.transaction_6.remove({});
printjson(db.transaction_7.count());
db.transaction_7.remove({});
printjson(db.transaction_8.count());
db.transaction_8.remove({});
 
var t2 = new Date().getTime() - t1;
printjson(t2);

2.执行它

mongo -nodb job.js

mongodb主从数据库同步

简单点说,就是从库都是新的mongodb,通过主从切换完成空间的清理。
————————-
1)先删除数据,remove不会阻塞住整个db;
找业务不忙的时间操作。
db.collection1.remove({})
db.collection2.remove({})

2)然后,secondary上的数据重新同步,这样secondary删除的空间就释放了。
同步完之后进行一次主从切换,Secondary升级为Primary

3)新Secondary再同样进行重新同步数据,同样释放空间了
4)drop掉需要清理的空表
db.collection1.drop()
db.collection2.drop()

VC2017的运行时库提取

如果你是用Visual Studio 2015和2017来编写C或C++程序,那么就已经是基于UCRT的。
为了方便提取运行时库,请安装EveryThing搜索工具。
工具下载地址:https://www.voidtools.com/zh-cn/downloads/
以VC2017运行时库为例。
VC2017运行时库【UCRT库】包括以下几种文件。
Microsoft.VC141.CRT:
1.在EveryThing的输入界面搜索:Microsoft.VC141.CRT

2.再搜索ucrt,在Redist的子目录下,才是正常的UWP或ucrt库,文件一般是41个。

Mongo常见操作

Mongo后台建唯一索引
db.orders.ensureIndex({userid:1,reqid:1},{unique:true,background:true})
备分数据库
mongodump -h 10.0.1.106 –port 27017 -u root -p xxxxxxx -d abc_gold -o /data/backup/abc_gold –authenticationDatabase admin
数据恢复
mongorestore -h 10.0.1.107 –port 27017 -u root -p yyyyyyyyy -d abc_gold /data/backup/abc_gold –drop –authenticationDatabase admin

mgo慢查2

1)如果只是查看,可以直接登录mongodb,查询

mongo -authenticationDatabase admin -u ksc_admin -p xxxx –authenticationMechanism SCRAM-SHA-1

>use local

> db.oplog.rs.find({ ts : { $lt : Timestamp(1546401000, 1), $gt : Timestamp(1546399200, 1) } })

其中Timestamp是时间戳,与日期时间格式转化可以使用 https://tool.lu/timestamp

2)提取到文件,json格式

编辑一个.js文件如下:

vi aa.js

var c = db.oplog.rs.find({ ts : { $lt : Timestamp(1546401000, 1), $gt : Timestamp(1546399200, 1) } })
while(c.hasNext()) {
printjson(c.next());
}

使用mongo shell执行这个文件,结果写到文件result.js

mongo 10.81.5.45:27017/local -authenticationDatabase admin -u ksc_admin -p xxxx –authenticationMechanism SCRAM-SHA-1 aa.js> result.js

这样这段时间产生的oplog就以json格式写到了result.js文件里,可以使用vi查看
提取对应时间段的oplog看看
MongoDB复制集正确的连接方式

mongodb://user:password@host1:port1,host2:port2/database?authSource=admin[&option1][&option2][&option3]…
举例如下:
mongodb://username:password@10.0.0.1:27017,10.0.0.2:27017/dbname?authSource=admin&replicaSet=”xxxxxxxxxxxxxxxxxxx”&readPreference=secondaryPreferred

username:password –用户名,密码
host:port需要把主从都写上,用逗号隔开 都写上、都写上、都写上
dbname: 数据库名
authSource=admin 通过admin库对登录的用户名和密码进行认证

可选项:
replicaSet=”xxxxxxxxxxxxxxxxxxx” 复制集的名称,在我们云数据库上指 实例ID, 可以写上
readPreference=secondaryPreferred 设置读写分离

mongodb慢查询

通过 db.system.profile.find() 查看当前所有的慢查询日志,这个需要use db,然后查db下的慢查询

查询当前时间段最慢的一条

db.system.profile.find().limit(1).sort({ millis : -1 }).pretty()

查看最近的一条:

db.system.profile.find().sort({$natural: -1}).limit(1)
查看最老的一条:
db.system.profile.find().sort({$natural: 1}).limit(1)
查询某个时间段的慢查询

切换到对应的数据库下:

var c = db.system.profile.find({ ts : { $gt : ISODate(“2019-02-15T00:00:00.197Z”), $lt : ISODate(“2019-02-23T08:43:02.197Z”) } })
while(c.hasNext()) {
printjson(c.next());
}