Jump to content
UBot Underground

orkney

Members
  • Content Count

    3
  • Joined

  • Last visited

Posts posted by orkney

  1. I left the group as I couldn't keep up with all of the messages, and now when I try to re-enter it says:

     

    "You have been removed from this group and will not be able to re-enter".

     

    I left of my own choosing and didn't do anything to get banned etc. Is it possible for a mod to add me? Skype id: william.villiers [at] googlemail.com

     

    Thanks :D


     
     
    Copying image
     
  2. Hi all,

     

    I've built my first bot which is ready to crawl just less than 1000 pages. I've dropped a "wait 3 seconds" command in the loop so as to not hammer the server, but I've not used a proxy or anything. The scrape will take around 1 hour to complete which is good with me, but I'm wondering if my crawl rate is still too high? Should I be using a list of proxies to scrape 5-10 pages per second to get the job done quicker and avoid raising red flags with the server admins?

     

    Just looking for any tips from the community on what you might call best practices.

     

    Any tips or plugins which you find essential in making bot creation even simpler etc?

     

    Is there anything I should be looking into which isn't covered in the (excellent) tutorial videos which would improve my workflow? eg. Define was in the title of one of the tut vids but it wasn't actually covered in the video so I've not looked into that yet but guessing I should soon. I'm using Regex a lot but keep seeing mentions of XPath - is this a preferred solution to pattern matching?

     

    All tips for a newbie welcome B)
    Thanks
    Will

×
×
  • Create New...