123456789from pykafka import KafkaClientclient = KafkaClient(hosts="localhost:9092")topic = client.topics['maoyan_wish']consumer = topic.get_simple_consumer(consumer_group='test', auto_commit_enable=True, consumer_id='test')for message in consumer: if message is not None: print(message.offset, message.value)
这个程序貌似是不会停止的
123456789101112#!/usr/bin/python# -*- coding: UTF-8 -*-import timefrom pymongo import MongoReplicaSetClientfrom pymongo import MongoClient# 连接单机# single mongo# c = MongoClient(host="192.168.89.151", port=27017) # okay# 连接集群# mongo clusterc = MongoClient('mongodb://192.168.89.151,192.168.89.152,192.168.89.153')#okayprint c.database_names()
通过将用户添加到docker用户组可以将sudo去掉，命令如下
12345sudo groupadd docker #添加docker用户组sudo gpasswd -a $USER docker #将登陆用户加入到docker用户组中newgrp docker #更新用户组
在metastroe
（1）修改表字段注解和表注解
12alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8;alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
（2）修改分区字段注解
12alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8 ;alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8;
（3）修改索引注解
1alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
修改hive-site.xml配置文件
123 ...
CREATE TABLE tmp.COSTITEM( NAME STRING,ORDERDATE DATE,COST STRING);
– 数据加INSERT INTO tmp.COSTITEM VALUES (‘jack’,’2020-01-01’,’10’);INSERT INTO tmp.COSTITEM VALUES (‘tony’,’2020-01-02’,’15’);INSERT INTO tmp.COSTITEM VALUES (‘jack’,’2020-02-03’,’23’);INSERT INTO tmp.COSTITEM VALUES (‘tony’,’2020-01-04’,’29’);INSERT INTO tmp.COSTITEM VALUES (‘jack’,’2020-01-05’,’46’);INSERT INTO tmp.COSTITEM VALUES (‘jack’,’2020-04-06’,’42’);INSERT INTO tmp.COSTITEM VALUES (‘tony’,’2020-01-07’,’50’); ...
1. 请说下HDFS读写流程HDFS写流程1）client客户端发送上传请求，通过RPC与namenode建立通信，namenode检查该用户是否有上传权限，以及上传的文件是否在hdfs对应的目录下重名，如果这两者有任意一个不满足，则直接报错，如果两者都满足，则返回给客户端一个可以上传的信息2）client根据文件的大小进行切分，默认128M一块，切分完成之后给namenode发送请求第一个block块上传到哪些服务器上3）namenode收到请求之后，根据网络拓扑和机架感知以及副本机制进行文件分配，返回可用的DataNode的地址
注：Hadoop 在设计时考虑到数据的安全与高效, 数据文件默认在 HDFS 上存放三份, 存储策略为本地一份，同机架内其它某一节点上一份, 不同机架的某一节点上一份4）客户端收到地址之后与服务器地址列表中的一个节点如A进行通信，本质上就是RPC调用，建立pipeline，A收到请求后会继续调用B，B在调用C，将整个pipeline建立完成，逐级返回client5）client开始向A上发送第一个block（先从磁盘读取数据然后放到本地内存缓存），以p ...
archlinux设置固定ip
cd /etc/netctl/
Ip adds 查看网卡id
vim 网卡id
TYPE=EthernetPROXY_METHOD=noneBROWSER_ONLY=noBOOTPROTO=”static” # 使用静态IP地址，默认为dhcpIPADDR=”192.168.28.5” # 设置的静态IP地址NETMASK=”255.255.255.0” # 子网掩码GATEWAY=”192.168.28.1” # 网关地址DNS1=”192.168.28.1” # DNS服务器DEFROUTE=yesIPV4_FAILURE_FATAL=noIPV6INIT=yesIPV6_AUTOCONF=yesIPV6_DEFROUTE=yesIPV6_FAILURE_FATAL=noIPV6_ADDR_GEN_MODE=stable-pr ...
查找大文件1find / -type f -print0 | xargs -0 du -h | sort -rh | head -n 10
精确查找1find / -type f -name "*.conf" | xargs grep -ri "10.0.0.74" -i
MinicondaMiniconda is a free minimal installer for conda. It is a small, bootstrap version of Anaconda that includes only conda, Python, the packages they depend on, and a small number of other useful packages, including pip, zlib and a few others. Use the conda install command to install 720+ additional conda packages from the Anaconda repository.
See if Miniconda is right for you.
System requirementsLicense: Free use and redistribution under the terms of the EULA for Miniconda.Operating syste ...
