stanf
-
Content Count
571 -
Joined
-
Last visited
-
Days Won
11
Posts posted by stanf
-
-
i am a standard ubot user
all i am doing is scraping a few sites for my (hopefully) soon to open ecom site, ( although this christmas season will probably be a wash).
I try to keep my loops under 1000 reps on a laptop with xp and 2 gigs of memmory (still crashes 6 out of 10 trys).
i have clearmem set to wipe the memmory every 3 minutes.
i run the javascripts clear my tracks after every rep ( so you know scraping aprox 15000 pages takes forever).
what else can i do to make my time productive,while the dev crew works out the bugs in the next release?
1-shorten the loops?
2- improve my code logic?
3 anything you can think of?
thanx
stanf
-
Can I ask a ?....what's the difference between the above mentioned site and hma?
-
i want to thank you guys for taking the time to answer this post.
all three sites look interesting ,I looked all three over last night,
I am going to have to dig in each a little deeper over the weekend ,
but again thanx for the response and your opinions on each.
-
I know this is not ubot related, the this community seems to be about the buisness of makeing money so I feel that you guys will steer me in the rite direction.
what are some good forums to read about what works and what to avoid.
In the internet marketing and traffic generation
game.
I don't want to be spoon fed a meal, I want to learn how to hunt the game.
Thanx
stanf
-
thanx for shareing
-
1
-
-
it seems that these folks have decided to prevent hotlinking,
i need to download this image.
it is the resizable product image
-
thanks that what i thought check, check for a common element and if it is not there reload the page,
thanx again
-
I have the problem of crash between 300-1600 lines. I was hopeing to get an answer from the forum before going to support
-
I was scraping a site , every thing was going fine when the site had a hiccup and left me with nothing to scrape. how is this handled by you pros?
I know now that i have to put in a a test to make sure that the target of the scrape is actually there?
But if it isn't how do I make the loop start over?
or
do I put in a javascript to reload the page and hope that it reloads the proper page?
-
hi did you ever get an answer to the crash while looping problem?
-
ok ( i -c said the blind man) i saw the class tag but did not understand it ( my knowlege of html is acient) but what is the difernce between href and full href?
-
thanx kevin,
that did the trick,,but can you please explain why you chose what you chose?
-
this seems so simple, but still its failing.
this is a simple url scrape. I chose the page scrape because it seemed the simplest way to isolate the data i need.
-
thanx john that was quick and to the point.
kevin i appreciate your help ,but that was to complicated for me at the moment ( i aint got that far yet)
-
thanx for your input but thats not where i was going.
I could just give the loop count a a number of 100 and pretty much be sure to get all the links.
But the object was to be able to figure out how many loops if the item count were to change.(this bot will go thru
all amazon items i wish to sell and add the links to a link list script. and then i can rss blast all catagories and build
indivisual pages to search engine market.
I have made progress in isolating the number of total items.
since the first page will always show " Showing 1-10 of ### Results"
I just did a replace (Showinf 1 -10 of) = " "
replace (Results) = " "
but I still cant do math funtions (* \ + -) on ###
p.s. I understand that sooner or later I will have to learn regex, it will make future projects easier
-
this is the problem .
amazon search shows 250 items "1 - 10 of 250 Results" , I page scrape the left side(of) ---right side (Results)
= %item count or #item count
that gives me 250 to add as a line in a list or make a variable, in this case , the math would be
****************************************
**************this fails "cannot divide 2 strings****************
set loop count= (($eval (%item count or #item count / 10))
**************************************************************************
I want to use loop times (loop count)
-
thanx kevin
your code showed me that it did not need to be so complicated,( just give the loop a set number of times to loop) but i have another problem. I understand, I just find the total number of items and divide by 10, but how do you convert a string to a intiger to perform math on it.
scraping the total number of items and performing ($eval (%totalitems /10)) gives the error cannot divide two strings
-
can someone tell me why the the condition to end the first loop is not being met?
-
can someone tell me why the conditions are not being met to end the first loop
-
thanx, merkaba
but how do advance the rite to file
to the next line?
-
I know this is probably a simple issue, but i'm a rookie and this is my first attempt at this.
The 1st ? is why when I reed back the data ("ad item to list" postal coade,and address) it reads all the variables as one piece, I need them as seperate parts, I could just enter the imput data from above, but what would I learn that way?
the 2nd ? " how do I write it to a csv file and not over write the data?
-
here is the deal
how would you pros get the first colum of this page,
http://names.mongabay.com/data/19000.html
its the name section, I am a rookie to ubot and have been kicking
my self for three nights trying to get it
p.s i have the rookie version of ubot4
if you can do it please give a breif explenation of how and more importantly why
you chose the method that you did.
thanx
stanf
crash victem needs help
in General Discussion
Posted
a2mateit ----i took your advice and split the number of bots from 3 to 6 (less pages to work before closing the browser and opening a new one)
the list only grows to a max of 700 lines before being written to a file and a new list is begun
zap----thanks for the reload (force get) tip
a lot of the waits are because
1) i did not know what the problem was
2) if i bang that site too fast i get a lot more page load errors but some of that i can correct because now i have a better understanding of how to check for and handle page load errors.
thanx for the feedback fellas
stanf