HUGE speed boost tip for scraping with the basic UBOT package.

[Ira_Hayes 4]

Check out this video to see the speed at which I scrapped my last project (3-4 complete page scrapes per second at times). Video attached.

 

I wanted to see if I could scrape faster by using this command.

set(#data, $read file("http://www.example.com"), "Global")

 

That takes the contents of a URL, and loads it into a variable.

 

Then you just use regex to find what you need. My script has about a dozen lists and multiple nested regex, replace, trim functions, and it's still blazing fast on a virtual machine.

 

Here is an example of the code used to get the title of the off road trail I am scraping.

 

add item to list(%title, $replace($replace($find regular expression(#data, "<meta name=\"TrailName\" content=\"[^\".]*\" />"), "<meta name=\"TrailName\" content=\"", $nothing), "\" />", $nothing), "Don\'t Delete", "Global")

 

That looks tricky in code view, but paste that into your ubot studio and it will make more sense to you.

 

I hope this helps, this is how I will scrape everything in the future.

Speed.mov

[gandensang 11]

can't download attacment

[Legend 181]

downloaded fine here... http://ubotstudio.com/forum/public/style_emoticons/default/blink.gif

[Praney Behl 314]

I think if you are just trying scrape meta description it'll be easier to use:

 

set(#data, "{$title},{$meta description},{$meta keywords}", "Global")

 

Just my .02c

 

Praney

[odeesuba 24]

Interesting idea , worth exploring.

 

Thanks for sharing.

[Ira_Hayes 4]

I think if you are just trying scrape meta description it'll be easier to use:

 

set(#data, "{$title},{$meta description},{$meta keywords}", "Global")

 

Just my .02c

 

Praney

I wasn't scraping keywords or descriptions. These were all meta tags, but not the keyword or description tag.

[JustinF 0]

This thread was a tremendous help tonight, thank you! My scraper was virtually unusable even with sockets and multithreading until I tried your tip, and now it is blazing fast.

[Legend 181]

Definitely... good stuff... thanks! http://ubotstudio.com/forum/public/style_emoticons/default/smile.gif

[whoami 26]

$meta keywords doesnt help on all pages..

Any other suggestion for scraping keywords?

[sktan7 12]

I cannot download the attachment