google app engine - cron job throwing DeadlineExceededError -


i working on google cloud project in free trial mode. have cron job fetch data data vendor , store in data store. wrote code fetch data couple of weeks ago , working fine of sudden , started receiving error "deadlineexceedederror: overall deadline responding http request exceeded" last 2 days. believe cron job supposed timeout after 60 minutes idea why getting error?.

cron task

def run():   try:     config = cron.config     actual_data_source = config['xxx']['xxxx']     original_data_source = actual_data_source      company_list = cron.rest_client.load(config, "companies", '')      if not company_list:         logging.info("company list empty")         return "ok"      row in company_list:         company_repository.save(row,original_data_source, actual_data_source)      return "ok"

repository code

  def save( dto, org_ds , act_dp):   try:     key = 'fin/%s' % (dto['ticker'])     company = companyinfo(id=key)     company.stock_code = key     company.ticker = dto['ticker']     company.name = dto['name']     company.original_data_source = org_ds     company.actual_data_provider = act_dp     company.put()     return company      except exception:     logging.exception("company_repository: error occurred saving company      record ")     raise

restclient

  def load(config, resource, filter):     try:     username = config['xxxx']['xxxx']     password = config['xxxx']['xxxx']     headers = {"authorization": "basic %s" % base64.b64encode(username + ":"      + password)}      if filter:         from_date = filter['from']         to_date = filter['to']         ticker = filter['ticker']         start_date = datetime.strptime(from_date, '%y%m%d').strftime("%y-%m-%d")         end_date = datetime.strptime(to_date, '%y%m%d').strftime("%y-%m-%d")      current_page = 1     data = []      while true:        if (filter):         url = config['xxxx']["endpoints"][resource] % (ticker, current_page, start_date, end_date)       else:         url = config['xxxx']["endpoints"][resource] % (current_page)        response = urlfetch.fetch(             url=url,             deadline=60,             method=urlfetch.get,             headers=headers,             follow_redirects=false,          )       if response.status_code != 200:             logging.error("xxxx received status code %d!" % (response.status_code))             logging.error("error happend url: %s headers %s", url, headers)             return 'sorry, xxxx api request failed', 500        db = json.loads(response.content)        if not db['data']:             break        data.extend(db['data'])        if db['total_pages'] == current_page:             break        current_page += 1      return data except exception:      logging.exception("error occured xxxx api request")      raise

i'm guessing same question this, more code: deadlineexceedederror: overall deadline responding http request exceeded

i modified code write database after each urlfetch. if there more pages, relaunches in deferred task, should before 10 minute timeout.

uncaught exceptions in deferred task cause retry, mindful of that.

it unclear me how actual_data_source & original_data_source worked, think should able modify part.

crontask

def run(current_page=0):   try:     config = cron.config     actual_data_source = config['xxx']['xxxx']     original_data_source = actual_data_source      data, more = cron.rest_client.load(config, "companies", '', current_page)      row in data:           company_repository.save(row, original_data_source, actual_data_source)      # fetch rest     if more:         deferred.defer(run, current_page + 1)   except exception e:      logging.exception("run() experienced error: %s" % e)

restclient

  def load(config, resource, filter, current_page):     try:         username = config['xxxx']['xxxx']         password = config['xxxx']['xxxx']         headers = {"authorization": "basic %s" % base64.b64encode(username + ":"          + password)}          if filter:             from_date = filter['from']             to_date = filter['to']             ticker = filter['ticker']             start_date = datetime.strptime(from_date, '%y%m%d').strftime("%y-%m-%d")             end_date = datetime.strptime(to_date, '%y%m%d').strftime("%y-%m-%d")              url = config['xxxx']["endpoints"][resource] % (ticker, current_page, start_date, end_date)         else:             url = config['xxxx']["endpoints"][resource] % (current_page)          response = urlfetch.fetch(                 url=url,                 deadline=60,                 method=urlfetch.get,                 headers=headers,                 follow_redirects=false,          )         if response.status_code != 200:                 logging.error("xxxx received status code %d!" % (response.status_code))                 logging.error("error happend url: %s headers %s", url, headers)                 return [], false          db = json.loads(response.content)          return db['data'], (db['total_pages'] != current_page)       except exception e:          logging.exception("error occured xxxx api request: %s" % e)          return [], false

Comments

Post a Comment

Popular posts from this blog

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

i'm implementing inapp billing in android app. works well, however, i'm trying decouple broadcast receiver activity manifest. suggestion in android's trivialdrive sample: // important: dynamically register broadcast messages updated purchases. // register receiver here instead of <receiver> in manifest // because call getpurchases() @ startup, therefore can ignore // broadcasts sent while app isn't running. // note: registering listener in activity bad idea, done here // because sample. regardless, receiver must registered after // iabhelper setup, before first call getpurchases(). currently there's class extends broadcastreceiver : public class iabbroadcastreceiver extends broadcastreceiver { /** * intent action receiver should filter for. */ public static final string action = "com.android.vending.billing.purchases_updated"; private final iabbroadcastlistener mlistener; public iabbroadcastreceiver(iabbroadcastlistener listener) { m...
Read more

python Tkinter Capturing keyboard events save as one single string -

is possible store keyboard events 1 string? code below stores 1 char , prints it. card reader or bar code reader, contains collection of character/ string not 1 character @ time. goal save char pressed text variable. from tkinter import * root = tk() def key(event): text= event.char text+= event.char print ("pressed", text) def callback(event): frame.focus_set() print ("clicked at", event.x, event.y) frame = frame(root, width=100, height=100) frame.bind("<key>", key) frame.bind("<button-1>", callback) frame.pack() root.mainloop() currently, creating text variable , everytime key() function gets called, , text stores last character typed. you can define text module level variable , , use module level text inside key function - text = '' def key(event): global text text+= event.char print("pressed", text)
Read more

sql server - Why does Linq-to-SQL add unnecessary COUNT()? -

here tables: create table customer ( cid int primary key, name varchar(32) ) create table ord ( oid int primary key, cid int foreign key references customer, address varchar(20) ) here's linq-to-sql statement: var aaa = c in db.customer select new { c, o = c.ord.tolist() }; and here's inexplicable query generated linq-to-sql: select [t0].[cid] [cid], [t0].[name] [name], [t1].[oid] [oid], [t1].[cid] [cid2], [t1].[address] [address], (select count(*) [dbo].[ord] [t2] [t2].[cid] = [t0].[cid]) [value] [dbo].[customer] [t0] left outer join [dbo].[ord] [t1] on [t1].[cid] = [t0].[cid] order [t0].[cid], [t1].[oid] i want understand how rid of count(*) part. it's entirely uncalled for! linq needs count(*) in order determine number of entries returned o = c.ord.tolist() .
Read more