KneeOnLite 0 Posted July 18, 2010 Report Share Posted July 18, 2010 I am trying to scrape a webpage of its links, that bit works OK I then have a svae to file and that bit works. Next my bot clicks a next link and will loop again (once I add the Loop) through the same thing. So heres my problem when I check the file it is only saveing the latest page that has been scrapped all the other stuff has been overwritten. It will have about 4,200 pages to scrape. How can I get my bot to add each page it scrapes to a list, Ive tried most options such as add to list but Im getting nowhere fast. Im have trouble saving stuff to lists. here is what Im tring to do... http://www.pagezz.com/Capture.jpg Quote Link to post Share on other sites
pftg4 102 Posted July 18, 2010 Report Share Posted July 18, 2010 hi Welcome to the forum what you need to do is add everything to a list then save to file at the end outside of your loop,what is happening now is that you are overwriting what you just saved because your save to file is inside the loop that should sort out your problem,but that amount of data all in one lisy this might be problem i think you should split it up but this is just my opinion hopethis helps some what Pftg4 Quote Link to post Share on other sites
KneeOnLite 0 Posted July 18, 2010 Author Report Share Posted July 18, 2010 hi Welcome to the forum what you need to do is add everything to a list then save to file at the end outside of your loop,what is happening now is that you are overwriting what you just saved because your save to file is inside the loop that should sort out your problem,but that amount of data all in one lisy this might be problem i think you should split it up but this is just my opinion hopethis helps some what Pftg4Hi thanks for the fast reply, but Im confused, I dont have a loop running, and when I tried Add to list I couldnt get it to work. I think I must have tried a hundred different ways and commands, but the good news is I managed to learn some other stuff along the way by my trial and error. Quote Link to post Share on other sites
pftg4 102 Posted July 18, 2010 Report Share Posted July 18, 2010 what are you trying to scrape????give me the url and i will take a lookto me it looks like you are just scraping the url Pftg4 Quote Link to post Share on other sites
KneeOnLite 0 Posted July 19, 2010 Author Report Share Posted July 19, 2010 what are you trying to scrape????give me the url and i will take a lookto me it looks like you are just scraping the url Pftg4 Thanks but I really dont want to give out the URL but it is a page similar to Google and the bot does actually scrape what I need it just wont save page 1 then add page 2 then 3 and so on. The problem is data saved is getting over written and it saves only the last page scrapped. Quote Link to post Share on other sites
JohnB 255 Posted July 19, 2010 Report Share Posted July 19, 2010 Hi thanks for the fast reply, but Im confused, I dont have a loop running, and when I tried Add to list I couldnt get it to work. I think I must have tried a hundred different ways and commands, but the good news is I managed to learn some other stuff along the way by my trial and error. If you are not running a loop how are you overwriting the data? Are you running the script from the beginning each time? Because that WILL overwrite the data. As stated above, you need to run it in a loop and add it to a list prior to saving it to the file. I hope that helps. John Quote Link to post Share on other sites
KneeOnLite 0 Posted July 19, 2010 Author Report Share Posted July 19, 2010 If you are not running a loop how are you overwriting the data? Are you running the script from the beginning each time? Because that WILL overwrite the data. As stated above, you need to run it in a loop and add it to a list prior to saving it to the file. I hope that helps. JohnThanks John I tried to run it in a loop before but will try it again and add to list before I save to file. Quote Link to post Share on other sites
KneeOnLite 0 Posted July 19, 2010 Author Report Share Posted July 19, 2010 OK I ran the bot in a loop it still saves only one page at a time instead of adding the data to a file or a list, I also tried add to list instead of save to file and the save to file and add to list togeather. I think Im missing something simple here. Quite frankly Ive had trouble saving stuff and working with lists from the begining. Ubot is great but it really needs a better help system, I have been all over the net downloading Tutorials/videos and reading, it would be great if there was one place where a new user like me could go and find a structured help system. Quote Link to post Share on other sites
KneeOnLite 0 Posted July 20, 2010 Author Report Share Posted July 20, 2010 Can someone please give me a clue here Im completely stuck Quote Link to post Share on other sites
MiriamMB 63 Posted July 20, 2010 Report Share Posted July 20, 2010 KneeOnLite, If you need structured help there is the support.ubotstudio.com page. As for the last thing you said, do you have the save to file INSIDE the loop? Try putting it OUTSIDE of the loop, so that it scrapes everything, adds everything to a list, and then finally, outside of the loop, saves everything to file. Quote Link to post Share on other sites
KneeOnLite 0 Posted July 20, 2010 Author Report Share Posted July 20, 2010 OK I got it. What really confused me is the add to list, I was expecting to open the list and see data, now I moved the save to file outside the loop and saved the data from the list to file. It works just great, and the stupid thing is pftg4 gave me the answer in the first post. Thanks to all. Live and learn. Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.