Save to File Problem

[KneeOnLite 0]

I am trying to scrape a webpage of its links, that bit works OK I then have a svae to file and that bit works. Next my bot clicks a next link and will loop again (once I add the Loop) through the same thing. So heres my problem when I check the file it is only saveing the latest page that has been scrapped all the other stuff has been overwritten. It will have about 4,200 pages to scrape.

 

How can I get my bot to add each page it scrapes to a list, Ive tried most options such as add to list but Im getting nowhere fast. Im have trouble saving stuff to lists. here is what Im tring to do...

 

http://www.pagezz.com/Capture.jpg

[pftg4 102]

hi Welcome to the forum

 

what you need to do is add everything to a list then save to file at the end outside of your loop,

what is happening now is that you are overwriting what you just saved because your save to file is inside the loop that should sort out your problem,but that amount of data all in one lisy this might be problem i think you should split it up but this is just my opinion hopethis helps some what

 

 

Pftg4

[KneeOnLite 0]

hi Welcome to the forum

 

what you need to do is add everything to a list then save to file at the end outside of your loop,

what is happening now is that you are overwriting what you just saved because your save to file is inside the loop that should sort out your problem,but that amount of data all in one lisy this might be problem i think you should split it up but this is just my opinion hopethis helps some what

 

 

Pftg4

Hi thanks for the fast reply, but Im confused, I dont have a loop running, and when I tried Add to list I couldnt get it to work. I think I must have tried a hundred different ways and commands, but the good news is I managed to learn some other stuff along the way by my trial and error.

[pftg4 102]

what are you trying to scrape????give me the url and i will take a look

to me it looks like you are just scraping the url

 

 

Pftg4

[KneeOnLite 0]

what are you trying to scrape????give me the url and i will take a look

to me it looks like you are just scraping the url

 

 

Pftg4

 

Thanks but I really dont want to give out the URL but it is a page similar to Google and the bot does actually scrape what I need it just wont save page 1 then add page 2 then 3 and so on. The problem is data saved is getting over written and it saves only the last page scrapped.

[JohnB 255]

Hi thanks for the fast reply, but Im confused, I dont have a loop running, and when I tried Add to list I couldnt get it to work. I think I must have tried a hundred different ways and commands, but the good news is I managed to learn some other stuff along the way by my trial and error.

 

If you are not running a loop how are you overwriting the data? Are you running the script from the beginning each time? Because that WILL overwrite the data.

 

As stated above, you need to run it in a loop and add it to a list prior to saving it to the file. I hope that helps.

 

John

 

 

 

[KneeOnLite 0]

If you are not running a loop how are you overwriting the data? Are you running the script from the beginning each time? Because that WILL overwrite the data.

 

As stated above, you need to run it in a loop and add it to a list prior to saving it to the file. I hope that helps.

 

John

Thanks John I tried to run it in a loop before but will try it again and add to list before I save to file.

[KneeOnLite 0]

OK I ran the bot in a loop it still saves only one page at a time instead of adding the data to a file or a list, I also tried add to list instead of save to file and the save to file and add to list togeather. I think Im missing something simple here. Quite frankly Ive had trouble saving stuff and working with lists from the begining.

 

Ubot is great but it really needs a better help system, I have been all over the net downloading Tutorials/videos and reading, it would be great if there was one place where a new user like me could go and find a structured help system.

[KneeOnLite 0]

Can someone please give me a clue here Im completely stuck

[MiriamMB 63]

KneeOnLite,

 

If you need structured help there is the support.ubotstudio.com page.

 

As for the last thing you said, do you have the save to file INSIDE the loop? Try putting it OUTSIDE of the loop, so that it scrapes everything, adds everything to a list, and then finally, outside of the loop, saves everything to file.

[KneeOnLite 0]

OK I got it. What really confused me is the add to list, I was expecting to open the list and see data, now I moved the save to file outside the loop and saved the data from the list to file. It works just great, and the stupid thing is pftg4 gave me the answer in the first post. Thanks to all. Live and learn.