python - Problems in scraping and saving file by eventlet -

-

i can use evenlet scrap img website failed save them domestic directory. following code. 1 familiar i/o operation in tasklets model? thanks

import pyquery import eventlet eventlet.green import urllib2  #fetch img urls............ works fine  print "loading page..." html=urllib2.urlopen("http://www.meinv86.com/meinv/yuanchuangmeinvzipai/").read() print "parsing urls..." d=pyquery.pyquery(html) count=0 urls=[] url='' in d('img'):  count=count+1  print i.attrib["src"]  urls.append(i.attrib["src"])   def fetch(url):  try:   print "start feteching %s" %(url)   urlfile = urllib2.urlopen(url)   size=int(urlfile.headers['content-length'])   print 'downloading %s, total file size: %d' %(url,size)   data = urlfile.read()   print 'download complete - %s' %(url)  ########################################## #file save won't work    f=open("/head2/"+url+".jpg","wb")   f.write(body)     f.close()   print "file saved" ##########################################      return data   except:   print "fail download..."     pool = eventlet.greenpool()  body in pool.imap(fetch, urls):   print "done"

make sure url suitable filename e.g.:

import hashlib import os  def url2filename(url, ext=''):     return hashlib.md5(url).hexdigest() + ext # removes '\/'  # ... open(os.path.join("/head2", url2filename(url, '.jpg')), 'wb') f:      f.write(body) print "file saved"

note: don't want write files top-level directory such '/head2'.

you consider urllib.urlretrieve().


Comments

Post a Comment

Popular posts from this blog

php - What is the difference between $_SERVER['PATH_INFO'] and $_SERVER['ORIG_PATH_INFO']? -

-
what’s difference between $_server['path_info'] , $_server['orig_path_info'] ? how use them? when run print_r($_server) , path_info , orig_path_info not present in array. why not? how can enable them? i have read php manual on them, still don’t understand them. the path_info variable present if invoke php script this: http://www.example.com/phpinfo.php/hello_there it's /hello_there part after .php script. if don't invoke url that, there won't $_server["path_info"] environment variable. the porig_ prefix uncommon. path_info standard cgi-environment variable, , should never prefixed. did read that? (there issues around php3/php4 if invoked php interpreter via cgi-bin/ - hardly has such setups today.) for reference: http://www.ietf.org/rfc/rfc3875
Read more

fortran - Function return type mismatch -

-
i'm attempting recode old c++ program in fortran make use of lapack (i'm aware c++ have lapack++, i'm having lot of trouble installing it, gave up). i didn't have problems compilation, when had variables declared real . when started coding section of program required lapack, found parameters passed dsyev need double precision . tried change double precision (including changing hard coded numbers double precision counterparts, i.e. 0.0 -> 0.0d0) when try compile, following error of functions , subroutines i've written: error: return type mismatch of function <function> @ (1) (real(4)/real(8)) i'm not sure coming from, in program has been changed double precision. for example, i've declared following: double precision :: alpha(3),d(3),zeta1,zeta2 double precision :: a1(3),a2(3),d1(3),d2(3) double precision :: pi pi = 3.14159265359d0 alpha = (/0.109818d0, 0.405771d0, 2.22766d0/) d = (/0.444635d0, 0.535328d0, 0.154329d0 /) 10 i=1,3 ...
Read more

queue - mq_receive: message too long -

-
i implementing communication between 2 processes using queue. problem when call function mq_receive, error: message long. i have done following: struct mq_attr attr; long size = attr.mq_msgsize; .... // initializing queue "/gateway" int rc = mq_receive(gateway, buffer, size, &prio); if print size value, size=1, while when print same size program (got same mechanism), not long integer ( -1217186280 )... how can solve error?....so while size = 1, believe it's right "message long" why 1? p.s. have tried put : int rc = mq_receive(gateway, buffer, sizeof(buffer), &prio); but no result. it seems need read docs more carefully. when call mq_receive should pass size of destination buffer. this size must greater mq_msgsize attribute of queue . in addition, seems have error in queue attributes initialisation makes proper mq_receive call impossible. here standard message queue session: fill mq_attr struct ( doc ): struct mq_a...
Read more