Rasho Han 1 Posted April 4, 2011 Report Share Posted April 4, 2011 I have been trying to find out how to scrape tweets to a list without all the hash tags and member names but can't seem to understand it all. I have a real hard time understanding what to do with regular expressions but I believe that is the only way to do it. I have tried wildcards with msgtext to no avail. Maybe it is the scrape chosen attribute that I have wrong? My goal is to scrape the tweets for just whatever the person wrote, and not anything else, then to order it in a readable list. Here I have included what I have made so far, I wish I had not taken a vacation from botting or maybe I would get it by now...Tweet Scrape A4.ubot Quote Link to post Share on other sites
Praney Behl 314 Posted April 4, 2011 Report Share Posted April 4, 2011 Ok here you go again .. This will work. I tested to scrape for the keyword "OMG" and here are some results: OMG OMG OMG I WANA WIN SOOO BADLY!! It's Friday Friday~ omg. The song is stuck in my head. And in other news - A man was paid $100,000 to get breast implants! Oh yes there is a pic ---> http://ow.ly/4mAeE Get Me A Napkin Jare! RT @kreamystifyd: OMG!! *Stunned* RT OluwaMuIYIwa: WipingMyShirt RT @kreamystifyd: *waitingforbloodtosplatter* RT O... RT @Marittjeex: RT @xDeeem97: OMG me rapport hea =| • Slecht ? • Gaat wel ;o @PaixAmourNicole omg lol danf i bet everyone thinking that... i literally have 0 friends on there. idk my account got hacked @xNereaKoolstra jij bent echt niet te geloven he hahaha omg Omg that fucken balloons @Marquee89 gave me aare scaring me right now. /: o.o Omg I can't do art for shit Omg more?? RT @YoNipplesHairy: Another V.I. person???? Yall are like roaches lol RT @Jermz_VI: @KdaWg_VI we talked earlier lol Photo: werdnahunt: hahahahahahahaah best http://tumblr.com/xpr1zp9s5i @thejoehoskins no lol. She does all these flips and omg it didn't last long but I was suprised at what Snooki could do haha Shes 12D:) omg I got to back up and regroup on this:O no way! thats pass my limit! omg… @FlipSpooren omg ik hou dr nu al van! En nog niks gezien! Omg this is horrible. Got to be the devil. SMDH Chooks-to-Go Liempo >>>> roasted chicken. OMG, drooling commences. omg!! my cookies came out great!! Want to taste them 8]? YUM! omg..im in love with this marc jacobs watch!....... whoa omg no way @FairlyAnon OMG 7ubail is...GOLLUM!! Pa suparto said: OMG student ! omg!! my cookies came out great!! Want to taste them 8]? YUM! Fuck youRT @NicTayPartyTime: OMG Mikey And @Brenda_Martini1 Never Stoped Tweeting Day Are Twitter Whores But Mikey Is A Twitter ManWhore LOL Omg, just found out, my prom dress is gonna be delievred @HeidiA19 OMG, why's everyone making me out to be som e brute?!:-O RT @Goditaditya: Dipeluk . OMG ! So hungry omg ;( @arizonaa omg I LOVE THE INTERNET I LOVE IT http://www.youtube.com/watch?v=PPkD_mIPDas omg ma cat wont stp starin @ meh wen i fukin eat!!!!1 "I'm so depressed I wore the same outfit twice this week!!" - Kurt, OMG I love him! <333333 @iJever OMG... @mallikasharmaaa What did he rage about??! OMG if it's Peter I'm sad I missed it. RT @UsGirls_: OMG..YES YES it'S FINAILY here ..checkout the new @Trillogy_SMG video on www.usgirlscanboogietoo.com...video sick &true meaning 2 the song.. omg Kate Winslet and Evan Rachel Wood are in a TV miniseries adaptation of Mildred Pierce?!?!? *Not* impressed. May watch just for the shigs wat is luuk arrogant vandaag, #omg Omg chi sin! This is too hard to sing *Thin Lizzy* has a new theme song!!! omg!!!! I hate it..... blehhh RT @faywarat: RT @slurpiez @faywarat OMG ! 55555 // OMG! à ¸—à ¸³à ¹„à ¸¡ à ¹€à ¸£à ¸·à ¹ˆà ¸Âà ¸‡à ¸ˆà ¸´à ¸‡à ¹€à ¸¥à ¸µà ¹‰à ¸¢à ¸‡à ¸ˆà ¸´à ¸‡ | à ¹€à ¸¥à ¸µà ¹‰à ¸¢à ¸‡à ¸žà ¸µà ¹ˆà ¹€à ¸«à ¸Âà ¸°à ¸™à ¹‰à ¸° >< OMG ik ben eindelijk klaar met me 1ste SE OMG are u fucking serious??????? Hurry the fuck up #Leggo Photo: nostalgicapplause: http://tumblr.com/x651zp9mpy @camisadoheartx omg yes! @TonightAlive omg! you guys are playing with Mötley Crüe and some other great bands! That's good for you guys. besok try out matematika OMG.. OMG! Kenny Rodgars <333 Oh here is the script: Sample_Tweet_Scraper_Praney.ubot Praney Quote Link to post Share on other sites
Rasho Han 1 Posted April 4, 2011 Author Report Share Posted April 4, 2011 Praney you are the man! Thanks much! I'm checking it out now Quote Link to post Share on other sites
Rasho Han 1 Posted April 4, 2011 Author Report Share Posted April 4, 2011 Wow it works like a charm! It still picks up the @ but I believe that can be fixed by using the advanced search to filter out such stuff, that is what I will work on now. When this is done I am going to make it free in the buy-borrow-steal section. I plan to make it grab stuff in more than 1 page too. This will be a bot to scrape the tweets so that people can use them for blog commenting. Pretty nice idea huh? Is there a way to order and organize the list in a better way? Like into a csv file and have 1 tweet per line? My inital idea was to make this a comment scraper for blog commenting and release it as a paid bot (something cheap) but since praney has been such a help that would be immoral and douchy so I will gladly release it to this and other communities freely Thanks Praney! And thanks to the rest of the great ubot community, we are small now but I have a feeling soon there will be a lot more people getting this badass software! Quote Link to post Share on other sites
Praney Behl 314 Posted April 5, 2011 Report Share Posted April 5, 2011 No worries mate, glad I could help. Praney Quote Link to post Share on other sites
Praney Behl 314 Posted April 5, 2011 Report Share Posted April 5, 2011 Wow it works like a charm! It still picks up the @ but I believe that can be fixed by using the advanced search to filter out such stuff, that is what I will work on now. When this is done I am going to make it free in the buy-borrow-steal section. That is a part of the TWEET message thus it gets scraped too. You can probably use RegX to remove that. John has good tutorials done about RegX you will be able to find. I plan to make it grab stuff in more than 1 page too. This will be a bot to scrape the tweets so that people can use them for blog commenting. Pretty nice idea huh? The idea has been used by many before, but I believe the effectiveness of idea depends upon how its implemented. Is there a way to order and organize the list in a better way? Like into a csv file and have 1 tweet per line?You can save the tweets list to a CSV file for sure. I see that you have a standard License. This can be accomplished using a loop. Will point you again to tutorials to learn more about table if you have doubts. My inital idea was to make this a comment scraper for blog commenting and release it as a paid bot (something cheap) but since praney has been such a help that would be immoral and douchy so I will gladly release it to this and other communities freely I appreciate that. Sharing is caring. I am sure new members will be benefited by your contributions. Thanks Praney! And thanks to the rest of the great ubot community, we are small now but I have a feeling soon there will be a lot more people getting this badass software! Keep up the good work. Praney Quote Link to post Share on other sites
UBotBuddy 331 Posted April 5, 2011 Report Share Posted April 5, 2011 I may have to revisit Tweeter. This has been an interesting thread to watch. Quote Link to post Share on other sites
Rasho Han 1 Posted April 26, 2011 Author Report Share Posted April 26, 2011 I have been messing with regex to try and get this to scrape only the text of a tweet. So far regex has been the hardest thing to learn and understand for me. I tried the following with no results, I have no idea why it does not work, it seems al together for me. I'm using regex magic to generate the regex needed. \b((<span id=))"msgtxt[0-9]+" class="msgtxt en">.{32,}</span>\b @"\b((<span id=))""msgtxt[0-9]+"" class=""msgtxt en"">.{32,}</span>\b" @"\b((<span id=))""msgtxt[0-9]+"" class=""msgtxt en"">.</span>\b" (\<((<span id=))"msgtxt[0-9]+" class="msgtxt en">[\s\S]</span>\>) Those were generated by using this as an example<span id="msgtxt62628851133591552" class="msgtxt en">only text and spaces here!! Yeah</span> The actual message had a bunch of url junk and I just replaced that with the message "only text and spaces here!! Yeah" so that I could read it better. Here is the bot, for further reference. I really believe it is something simple I am missing.Tweet Scrape A5b.ubot Quote Link to post Share on other sites
Guest leahDyou Posted June 5, 2011 Report Share Posted June 5, 2011 Cool. I will have to visit tweeter after reading this thread! Quote Link to post Share on other sites
Jon12345 0 Posted July 27, 2011 Report Share Posted July 27, 2011 I've tried this bot but I get a whole load of html junk in there. Anyone got an update? Quote Link to post Share on other sites
JackCrow 4 Posted April 5, 2013 Report Share Posted April 5, 2013 Anyway you can post the code for this? Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.