最近帮朋友做一些网络营销的工作,需要抓取一些客源信息,用以辅助营销,我感觉很多做电商的朋友,尤其是微商朋友们,应该都是比较需要各种客源信息的吧,今天,小编我就借此机会把利用python多进程的方式抓取黄页海量的客源信息的方法分享给大家,希望大家能多多关注本人,多讨论关于爬虫和数据挖掘分析相关的问题。

该代码已经上传至本人的github,喜欢的朋友可到我的公众号获取。

一、特别提到的库:

多进程:pool

mogo数据库:pymongo

自然语言编码转换:codecs

二、特别推荐—-jupyter

做开发没有趁手的兵器是不行的,对于python脚本的开发以及做一些数据挖掘和分析这方面工作来说,jupyter notebook这个工具是一个非常合适的工具,它可以保存你写的阶段性的小脚本,同时,又不至于像ipython那样把有价值的历史代码被滚屏所淹没,关于这个工具的其他优点,大家可以自行百度,总之,这是一个python开发的一个神器,借助它可以快速的写出很多高效的代码出来。

请点击输入图片描述

​三、主要代码:

这次抓取的站点是黄页88网,网站放置行业信息的结构是这样的,按照行业来抓取的话,首先是分为各种行业,每种行业下面又可以获取到各个省份的该行业的信息。

请点击输入图片描述

请点击输入图片描述

为了能够按照行业进行抓取,我们可以先把首页出现的行业分类抓取下来,保存为字典,这一步的方法就不仔细说了,请自行解决。我已经保存好了字典,将起保存在一个单独的配置文件中,方便调用时使用。

请点击输入图片描述

获取各个行业对应的每个省的url,我们可以写一个函数来完成,返回一个url的列表。

请点击输入图片描述

有了这个url以后,就可以直接得到内容了,其中翻页的时候需要注意url中控制页码是通过pn变量实现的,也需要对最后一页做一下判断,否则就会形成死循环。这一步就已经把数据写入了txt文件中。

请点击输入图片描述

最后是执行爬虫的的代码,这里用到了多线程,同时需要存入mongodb的数据库。

请点击输入图片描述

由于自媒体发布的时候贴代码不太方便,感兴趣的朋友关注我的同名订阅号获取github地址吧!

82 对 “python多进程快速批量爬取黄页海量信息并保存文本和数据库教程”的想法;

  1. Hello, Neat post. There is an issue together with your website in web explorer, may test this?
    IE still is the marketplace leader and a big component to other
    folks will miss your wonderful writing due to this problem.

  2. I’ve been surfing online more than 4 hours today, yet
    I never found any interesting article like yours.
    It is pretty worth enough for me. In my opinion, if all website owners and bloggers made good
    content as you did, the internet will be a lot more useful
    than ever before.

  3. Ahaa, its fastidious conversation regarding this article at
    this place at this website, I have read all that,
    so at this time me also commenting at this place.

  4. Today, I went to the beach front with my children. I
    found a sea shell and gave it to my 4 year old daughter
    and said “You can hear the ocean if you put this to your ear.” She put the shell to her ear and screamed.
    There was a hermit crab inside and it pinched her ear.
    She never wants to go back! LoL I know this is entirely
    off topic but I had to tell someone!

  5. Hi there! I could have sworn I’ve visited this web site before but after going through
    many of the posts I realized it’s new to me. Nonetheless, I’m certainly pleased
    I stumbled upon it and I’ll be book-marking it and checking back
    often!

  6. I’m not sure why but this weblog is loading very
    slow for me. Is anyone else having this issue or is it a issue on my
    end? I’ll check back later on and see if the problem still exists.

  7. you’re truly a just right webmaster. The site loading pace
    is incredible. It sort of feels that you are doing any unique
    trick. Moreover, The contents are masterwork. you have performed a excellent
    activity in this subject!

  8. My spouse and I absolutely love your blog and find almost all of your post’s to
    be precisely what I’m looking for. Do you offer guest writers
    to write content for you personally? I wouldn’t mind composing a post or elaborating on a few of the subjects you write in relation to here.
    Again, awesome website!

  9. Simply want to say your article is as astonishing.

    The clearness in your post is just spectacular and i could assume you’re an expert on this subject.
    Fine with your permission let me to grab your feed
    to keep updated with forthcoming post. Thanks a million and please
    continue the gratifying work.

  10. This design is spectacular! You certainly know how to keep
    a reader entertained. Between your wit and your videos, I was almost moved to
    start my own blog (well, almost…HaHa!) Great job.
    I really loved what you had to say, and more than that, how
    you presented it. Too cool!

  11. Hello, There’s no doubt that your website may be having
    browser compatibility problems. When I take a look
    at your web site in Safari, it looks fine but when opening in I.E., it
    has some overlapping issues. I simply wanted to provide you
    with a quick heads up! Besides that, great website!

  12. Hello there, I found your site by way of Google whilst looking for
    a related subject, your site came up, it looks good.
    I’ve bookmarked it in my google bookmarks.

    Hi there, simply was alert to your weblog through
    Google, and located that it’s really informative. I’m going to watch out for brussels.
    I’ll appreciate should you continue this in future.
    Numerous other people shall be benefited from your writing.
    Cheers!

  13. Good day! I know this is kinda off topic but I was wondering if you knew where I could
    locate a captcha plugin for my comment form? I’m
    using the same blog platform as yours and I’m having problems finding
    one? Thanks a lot!

  14. Greetings! This is my first comment here so I just wanted
    to give a quick shout out and say I truly enjoy reading through your blog posts.

    Can you suggest any other blogs/websites/forums that deal with the
    same topics? Thanks for your time!

  15. I really like your blog.. very nice colors & theme.
    Did you make this website yourself or did you hire someone
    to do it for you? Plz reply as I’m looking to construct my own blog and would like to find
    out where u got this from. cheers

  16. Have you ever thought about adding a little bit more
    than just your articles? I mean, what you say is fundamental and
    all. But think of if you added some great graphics or videos to give your posts more, “pop”!
    Your content is excellent but with images and clips, this blog could certainly be one
    of the best in its field. Awesome blog!

  17. After I initially commented I seem to have clicked on the -Notify me when new comments are added- checkbox and from now on whenever a comment is added I recieve
    4 emails with the exact same comment. There has to be a way you are
    able to remove me from that service? Thanks a lot!

  18. Hi, I do believe this is a great blog. I stumbledupon it 😉
    I’m going to come back yet again since I book-marked it.
    Money and freedom is the greatest way to change, may
    you be rich and continue to guide other people.

  19. A fascinating discussion is definitely worth comment. I do
    believe that you should publish more about this issue, it
    might not be a taboo matter but generally people don’t discuss such issues.
    To the next! Best wishes!!

  20. We’re a group of volunteers and starting a new scheme in our community.

    Your site offered us with valuable info
    to work on. You’ve done an impressive job and our entire community will be grateful to
    you.

  21. You really make it appear really easy together with your presentation but
    I to find this matter to be actually one thing which I believe I’d never understand.

    It sort of feels too complex and very large for me. I’m
    taking a look ahead in your subsequent submit,
    I’ll attempt to get the hang of it!

  22. I love your blog.. very nice colors & theme.
    Did you create this website yourself or did you hire someone to
    do it for you? Plz answer back as I’m looking to construct my own blog and would like to find out where u got this from.
    cheers

  23. hey there and thank you for your information – I’ve certainly
    picked up something new from right here. I did however
    expertise a few technical points using this website, since I experienced
    to reload the web site lots of times previous to I could get it to load correctly.
    I had been wondering if your web hosting is OK?
    Not that I am complaining, but slow loading instances times will sometimes affect your placement in google and can damage your quality score if ads
    and marketing with Adwords. Well I’m adding this RSS to my e-mail
    and could look out for much more of your respective fascinating content.
    Ensure that you update this again very soon.

  24. Good day I am so thrilled I found your weblog, I really found you by mistake, while I was searching on Aol for
    something else, Nonetheless I am here now and would just like to say cheers for a fantastic post and a
    all round entertaining blog (I also love the theme/design), I don’t have time to read it all at the minute
    but I have saved it and also added in your RSS feeds, so when I have time I will be back to read a great deal more,
    Please do keep up the awesome b.

  25. This design is steller! You obviously know how to keep a reader entertained.
    Between your wit and your videos, I was almost moved to start my own blog (well, almost…HaHa!) Wonderful job.
    I really loved what you had to say, and more than that, how you presented
    it. Too cool!

  26. Hey! I know this is kinda off topic but I was wondering if you knew
    where I could locate a captcha plugin for my comment form?

    I’m using the same blog platform as yours and I’m having problems
    finding one? Thanks a lot!

  27. I don’t even know how I ended up here, but I thought this post was good.
    I do not know who you are but certainly you’re going to a famous blogger if
    you are not already 😉 Cheers!

  28. Hey there! Quick question that’s completely off topic.
    Do you know how to make your site mobile friendly?
    My blog looks weird when browsing from my iphone4. I’m trying to find a
    theme or plugin that might be able to resolve this problem.

    If you have any suggestions, please share. Cheers!

  29. An outstanding share! I’ve just forwarded this onto a colleague who had been conducting
    a little research on this. And he in fact ordered me dinner simply because I stumbled upon it for
    him… lol. So let me reword this…. Thank YOU for the meal!!
    But yeah, thanx for spending some time to discuss this topic here on your
    web page.

  30. Hey I know this is off topic but I was wondering if you knew of
    any widgets I could add to my blog that automatically tweet my
    newest twitter updates. I’ve been looking for a plug-in like this
    for quite some time and was hoping maybe you would have some experience with something like this.

    Please let me know if you run into anything. I truly enjoy reading your blog and
    I look forward to your new updates.

  31. I must thank you for the efforts you have put in penning this website.
    I really hope to view the same high-grade blog posts by you later on as
    well. In fact, your creative writing abilities has inspired me to get my own website now ;
    )

  32. This is the right website for anybody who wishes to understand this topic.
    You realize a whole lot its almost hard to argue with you (not that I actually will need to…HaHa).
    You certainly put a new spin on a subject which has been discussed for a long time.
    Great stuff, just wonderful!

  33. Hey there! I’m at work surfing around your blog from my new apple iphone!
    Just wanted to say I love reading your blog and look forward to all your posts!

    Keep up the superb work!

发表评论

电子邮件地址不会被公开。 必填项已用*标注