在我的学士学位论文中,我尝试使用http连接将机器数据(在本例中是用python脚本发送的历史数据)发送到kafka。我使用的是运行在windows系统docker中的合流平台。
我尝试使用python脚本将数据发送到rest代理。起初,我得到了关于我能够解析的数据类型的错误响应。
import pandas as pd
import csv, os, json, requests, time, datetime, copy, sys
if len(sys.argv) > 1:
bgrfc_value = str(sys.argv[1])
else:
print("No arguments for bgrfc given, defaulting to 'false'")
bgrfc_value = 'false'
if len(sys.argv) > 2:
filePath = str(sys.argv[2])
else:
filePath = "path"
if len(sys.argv) > 3:
batchSize = int(float(str(sys.argv[3])))
else:
batchSize = 10
# Build skeleton JSON
basejson = {"message": {"meta" : "", "data": ""}}
# metajson = [{'meta_key' : 'sender', 'meta_value': 'OPCR'},
# {'meta_key' : 'receiver', 'meta_value': 'CAT'},
# {'meta_key' : 'message_type', 'meta_value': 'MA1SEK'},
# {'meta_key' : 'bgrfc', 'meta_value': bgrfc_value}]
# basejson['message']['meta'] = metajson
url = "http://127.0.0.1:8082/"
headers = {'Content-Type':'application/json','Accept':'application/json'}
def assign_timestamps(batch):
newtimestamps = []
oldtimestamps = []
# Batch timestamps to list, add 10 newly generated timestamps to a list
for item in batch['tag_tsp'].values.tolist():
newtimestamps.append(datetime.datetime.now())
oldtimestamps.append(datetime.datetime.strptime(str(item), "%Y%m%d%H%M%S.%f"))
# Sort old timestamps without sorting the original array to preserve variance
temp = copy.deepcopy(oldtimestamps)
temp.sort()
mrtimestamp = temp[0]
# Replicate variance of old timestamps into the new timestamps
for x in range(batchSize):
diff = mrtimestamp - oldtimestamps[x]
newtimestamps[x] = newtimestamps[x] - diff
newtimestamps[x] = newtimestamps[x].strftime("%Y%m%d%H%M%S.%f")[:-3]
# Switch old timestamps with new timestamps
batch['tag_tsp'] = newtimestamps
return batch
# Build and send JSON, wait for a sec
def build_json(batch):
assign_timestamps(batch)
batchlist = []
for index, row in batch.iterrows():
batchlist.append(row.to_dict())
basejson['message']['data'] = batchlist
print(basejson)
req = requests.post(url, json = json.loads(json.dumps(basejson)), headers = headers)
print(req.status_code)
time.sleep(1)
while(True):
df = pd.read_csv(filePath, sep=";", parse_dates=[2], decimal=",", usecols = ['SENSOR_ID', 'KEP_UTC_TIME', 'VALUE'], dtype={'SENSOR_ID': object})
df = df[::-1]
df.rename(columns={'SENSOR_ID' : 'ext_id', 'KEP_UTC_TIME' : 'tag_tsp', 'VALUE' : 'tag_value_int'}, inplace=True)
# Fill list with batches of 10 rows from the df
list_df = [df[ i:i + batchSize] for i in range(0, df.shape[0], batchSize)]
for batch in list_df:
build_json(batch)
脚本发送数据,但作为响应,我得到状态代码500。
2条答案
按热度按时间3j86kqsm1#
标题值不正确。你需要设置
Accept
以及Content-type
两个标题如下:此外,数据的结构应如下所示:
例如:
eqqqjvef2#
我相信输入“value”的数据一定是字符串。像这样的方法会奏效:
如果在阅读主题时收到有趣的消息,请尝试使用base64编码对消息进行编码。编码后的原始json字符串应如下所示: