是否有可能在不运行谷歌应用程序引擎webservice的情况下运行处理用户数据的脚本?没有webservice的大查询cron作业
对于较小的脚本它工作得很好,但是当我的脚本持续约40分钟我收到提示:DeadlineExceededError
我临时的解决办法是使用Windows在Windows VM调度器和命令行用python脚本
编辑:添加代码
jobs = []
jobs_status = []
jobs_error = []
# The project id whose datasets you'd like to list
PROJECT_NUMBER = 'project'
scope = ('https://www.googleapis.com/auth/bigquery',
'https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/drive',
'https://spreadsheets.google.com/feeds')
credentials = ServiceAccountCredentials.from_json_keyfile_name('client_secrets.json', scope)
# Create the bigquery api client
service = googleapiclient.discovery.build('bigquery', 'v2', credentials=credentials)
def load_logs(source):
body = {"rows": [
{"json": source}
]}
response = service.tabledata().insertAll(
projectId=PROJECT_NUMBER,
datasetId='test',
tableId='test_log',
body=body).execute()
return response
def job_status():
for job in jobs:
_jobId = job['jobReference']['jobId']
status = service.jobs().get(projectId=PROJECT_NUMBER, jobId=_jobId).execute()
jobs_status.append(status['status']['state'])
if 'errors' in status['status'].keys():
query = str(status['configuration']['query']['query'])
message = str(status['status']['errorResult']['message'])
jobs_error.append({"query": query, "message": message})
return jobs_status
def check_statues():
while True:
if all('DONE' in job for job in job_status()):
return
def insert(query, tableid, disposition):
job_body = {
"configuration": {
"query": {
"query": query,
"useLegacySql": True,
"destinationTable": {
"datasetId": "test",
"projectId": "project",
"tableId": tableid
},
"writeDisposition": disposition
}
}
}
r = service.jobs().insert(
projectId=PROJECT_NUMBER,
body=job_body).execute()
jobs.append(r)
return r
class MainPage(webapp2.RequestHandler):
def get(self):
query = "SELECT * FROM [gdocs_users.user_empty]"
insert(query, 'users_data_p1', "WRITE_TRUNCATE")
check_statues()
query = "SELECT * FROM [gdocs_users.user_empty]"
insert(query, 'users_data_p2', "WRITE_TRUNCATE")
query = "SELECT * FROM [gdocs_users.user_%s]"
for i in range(1, 1000):
if i <= 600:
insert(query % str(i).zfill(4), 'users_data_p1', "WRITE_APPEND")
else:
insert(query % str(i).zfill(4), 'user_data_p2', "WRITE_APPEND")
for error in jobs_error:
load_logs(error)
app = webapp2.WSGIApplication([
('/', MainPage),
], debug=True)
谢谢BrettJ!此解决方案可以工作 - 但请注意,App引擎在等待BigQuery返回时没有做任何工作。一个更好的方法是在工作排队后立即返回,然后再回来查看工作是否完成。 –
@BrettJ谢谢你的帮助。我想补充一点如果有人会尝试,你将不得不在'cron.yaml'中添加'target:your_service'来决定你想要使用的服务 – Mat