Cleaning Scraped Text

[Brutal 164]

Hi guys,

 

I'm having trouble figuring out how to do this and would really appreciate any help that anyone could offer.

 

I am scraping a list of urls, and I want to remove (or replace with nothing) all google related urls and leave all non-google related urls.

 

Any ideas?

 

Here's a snippet of the list i'm working with...

http://www.google.com/ http://www.google.com/search?hl=en&q=xOn3HOP739w&bav=on.2,or.r_qf.&bvm=pv.xjs.s.en_US.M4-36_38X9A.O&um=1&ie=UTF-8&tbm=isch&source=og&sa=N&tab=wi http://maps.google.com/maps?hl=en&q=xOn3HOP739w&bav=on.2,or.r_qf.&bvm=pv.xjs.s.en_US.M4-36_38X9A.O&um=1&ie=UTF-8&sa=N&tab=wl https://play.google.com/?hl=en&q=xOn3HOP739w&bav=on.2,or.r_qf.&bvm=pv.xjs.s.en_US.M4-36_38X9A.O&um=1&ie=UTF-8&sa=N&tab=w8 http://www.youtube.com/results?hl=en&q=xOn3HOP739w&bav=on.2,or.r_qf.&bvm=pv.xjs.s.en_US.M4-36_38X9A.O&um=1&ie=UTF-8&sa=N&tab=w1 http://translate.google.com/?hl=en&q=xOn3HOP739w&bav=on.2,or.r_qf.&bvm=pv.xjs.s.en_US.M4-36_38X9A.O&um=1&ie=UTF-8&sa=N&tab=wT http://www.google.com/search?hl=en&q=xOn3HOP739w&bav=on.2,or.r_qf.&bvm=pv.xjs.s.en_US.M4-36_38X9A.O&um=1&ie=UTF-8&tbo=u&tbm=bks&source=og&sa=N&tab=wp http://www.google.com/search?hl=en&q=xOn3HOP739w&bav=on.2,or.r_qf.&bvm=pv.xjs.s.en_US.M4-36_38X9A.O&um=1&ie=UTF-8&tbo=u&tbm=shop&source=og&sa=N&tab=wf http://www.google.com/finance?hl=en&q=xOn3HOP739w&bav=on.2,or.r_qf.&bvm=pv.xjs.s.en_US.M4-36_38X9A.O&um=1&ie=UTF-8&sa=N&tab=we 

[pftg4 102]

regex would be the way to go

[Brutal 164]

pftg4 - I hear ya man, but i still struggle with very low-scale regex... The regex it would take to do something like this is well above my skill level.

 

I appreciate your input man.

[pftg4 102]

hey they all have google in them

[Brutal 164]

oh - then I just didn't copy/paste in enough of the list - out of the 56 links, there are likely only around 15 that are non-google.

[pftg4 102]

ok then lets see um

[Brutal 164]

add list to list(%scrapegooglelinks, $scrape attribute(<outerhtml=w"*{#MyKeyword}*">, "fullhref"), "Delete", "Global")

 

Basically, because I couldn't find a better way, I just set up to scrape all urls in the google serps for a keyword, and then was/am trying to take out all of the google stuff so that I'm left with the 'regular' links left behind.
 

[pftg4 102]

http://ubotstudio.com/blog/2011/11/19/871/

 

 

read this will help with your selecting then you have no more problems

 

 

hope this helps (it will)

 

 

pftg4

[Brutal 164]

Thanks pftg4 - Very appreciated!