How on earth do you use the ulib_strings library???

[tooltrainer 12]

This is making me nuts... can anyone shed some light on this??

 

http://screencast.com/t/NzE3MDMwYz

 

Thanks all!

 

Jonathan

[TommyTx 5]

Hey Jonathan,

 

Long time no hear... this is tommytx you may remember we played with a few projects a while back...(before you became world famous.. heee... heee.. back in the days of Cloaking... you may not remember...

Any way these guys here at UBOT are great at tackling most any problem but they can't do a lot without the actual ubot or at least the section giving the problem..

I am sure that is a subscription site so I certainly understand that you cannot let anyone in, but here is a thought that works some but not always. Save that page in question in two different ways...

1. View source and save as a simple html page..

2. Use the IE page save command and select html and try to save the page as active as possible..

Bottom line is to try to get that dropdown functional.. for test scraping.

 

Another possibility is to go in the source and save the div with the droppdown..

 

Then reassemble as much of the page as you can and if the dropdown works, then we can all try to capture it live..

3. A last resort would be to allow one of the most knowledgable techs here go into the site and just change the password immediatley upon leaving..

 

I have only recently started using Ubot, but it is just tooooo basic for me... I have now learned most of the commands and most of what I do is just not possible.. it does have a powerful web grabber source, but knowing Autoit as well as I do, this is like dark ages... no interface commands to speak of.. hell ubot looks like it was actually written using autoit and then compiled. Problem is if it was build using autoit, it only has about 1% of the commands in Autoit.... I can do everything ubot can do with autoit and 100,000 times more... its just little easier to grab the web stuff using autoit..I will give it that much.... I do hand it that but its just too limited to most of what I need.... I tried to use it for the last week, but keep diverting to autoit.. as it has so much more capability and its free... for gods sake.

 

Any way I will be happy to give the scrape a try for you, but the faulty section is going to have to be made available for use to do so... anytime I am doing a major project, I dupe as much of the site as I can so that I can train the program first and then run it on the live site.. that way a savy webmasster does not catch me hitting the site a thousand times when trying to program an unruly part of the program...

 

If you want to chat, my Skype is still jackrabbitsec

[tooltrainer 12]

Tommy!! Hey man great to run into you again!! Of course I remember ya.

 

This isn't really about the page in question, it's more that I can never get the strings library to give any useful results at all. Hoping someone can shed some light on how to use it even for the simplest of stuff!

 

I'll have to check out autoit, I hadn't heard of that one before. I love how drop-dead simple UBot makes it to select data from web elements, but with that ease of use of course comes limitations of power. The eternal tradeoff.

 

Thanks for offering the help, and very cool to see you hang around these parts!

 

Jonathan

[TommyTx 5]

Hey this is neater than shit.. I just found the button that allows you to receive the full post in your email.. its so nice to read the full post without stopping and going to the web... I am surprised they allow this as most webmaster want us to keep comong back so thier traffice looks good to alexa.. anyway.. its really neat..

 

If you can capture the entire outer html and save it as one data element, its very simple to save it on the run. and have another program parse it to grab and format the data... on the fly... or did you say that even if you saved the entire innerHTML it would not include the data you need... how about outerHTML...

 

Worse to worse I might be able to write you a vbs or or autoit.. that can be shelled at the right spot which might get the data and even place the formatted data back into hte page under a new value= spot that you could scrape immediate following the Shell('myexternal.exe') ... who knows.. where there is a will there is a way..

 

So did you say that the data is not even in the innerHTML or yes..

 

You know.. UBOT even fires the javascript in the html page... that is a really neat one..

 

By the way.. the most common windows stuff is..

VBS visual basic script.

WSH windows script hosting.

AU3 Autoit.. and it is free and has one of the hottest Interface generators I have ever seen.. and it looks exactly like the interface on UBOT.. if UBOT ain't autoit indisguise with 95% of the commands removed... it missed a good chance...

Even "Web Data Parser" at $77.00 is a spitting image of autoit.. under a new hood and radio cap...

The commands on autoit.. are massive.. and its all open source....

But you can't beat UBOT for basic web scraping... its really easy to do that...

<quote>

and it looks exactly like the interface on UBOT

</quote>

Not what I meant... autoit does not have an interface, what I meant is when I build an interface it will look just like ubot and Web Data Parser....

[TommyTx 5]

Here is the bottom line... rip out as much source code as you can and post it on a domain... it will look like hell without the css. and all the other shit.. but scrapers don't care... they are reading the dom structure and css usually is not part of that unless css is acutally building the tables...

Most of the time you can grab the source and post it without the css and all the graphics and it looks like hell but if most of the DOM is still there it will scrape like a big dog...

 

There are a lot of experts here, but without the DOM to experiment on.. they usually cannot just tell you what to do out of their head, but with a little experimentaion.. they will send you a working bot in short order... they hve helped me a lot and i have only been here a week...

 

however I pretty much can handle any job ubot can do in only a week.... fast learning curve here..

 

 

Look how simple it is to make VBS visual basic script act like a UBOT in the wild card mode..

See how it checks each entry one by one and at the end determines if it is the correct field..

Here is an important point, Craig on their form that you fill out will change the number of input fields on each refresh to keep you from filling them automatically... since they give you no name and id or the name and id is som 30 character bullshit number that changes each time... we divert to filling with position.. guess what the bastards do.. change the number of input fields on each refresh.. so you cannot fill by position... so what the hell to do now.. can't do it by position, can't use id, can't use name.. that really sucks...

 

So here is what i do....

1. For each entry that I need to fill I set up a loop that loops thru ever input field..

2. We look for all the things that do not change..

3. out of say 24 fields when we find a field that matches all our critera we dump the variable then.

4. So we must spin a 24 field loop for each and every field we need to fill.

5. Each time one of the criteria is met.. it kicks the counter... if at the end we have tripped the counter

the same number of times we know it should we dump the veriable... works great..

 

; $title = "POSTING TITLE"

$cnt=0

For $dog In $toby

$man = $dog.outerHTML

$cnt=0

if StringInStr($man, "class=req") then $cnt=$cnt+1

if StringInStr($man, "tabIndex=1") then $cnt=$cnt+1

if StringInStr($man, "maxLength=70") then $cnt=$cnt+1

if StringInStr($man, "size=30") then $cnt=$cnt+1

if $cnt=4 then $dog.value=$title

Next

 

; $locat = "SPECIFIC LOCATION"

For $dog In $toby

$man = $dog.outerHTML

$cnt=0

if StringInStr($man, "class=s") then $cnt=$cnt+1

if StringInStr($man, "tabIndex=1") then $cnt=$cnt+1

if StringInStr($man, "maxLength=40") then $cnt=$cnt+1

if $cnt=3 then $dog.value=$locat

Next

 

No matter what Craigs List cooks up we will find a workaround...

If anyone is having problems filling the Craigs List.. I can help...

The UBOT is great for this using the Wildcards....

But I need way more commands than ubot can give so I have to make my VBS work like upbot... hee.. hee..

Cause UBOT is great for web scraping..

 

Someone just asked... what the hell is the DOM you keep talking about.. that is what the html structure is called.. Document Object Model.

[tooltrainer 12]

Yeah thanks for the ideas... I've managed to confuse posts though, I first responded to this one thinking it was about my other post where I absolutely can't get the value of a particular dropdown. Then I realized this was a different post and so I changed my reply to reflect this actual post... but since you got the original response via email it seems, you responded to that one! LOL Talk about confusin.

 

Anyway, this post was about the ubot string library that was posted here to use. I just plain can't get it to work at all. I'd really appreciate it if someone could create simple bot that looks for a needle in a supplied haystack and actually returns something different depending on if it's found or not. Should be braindead simple to do but I can't make it work.

 

Now as to my other scraping problem posted at http://ubotstudio.com/forum/index.php?/topic/3799-cant-get-the-value-of-a-particular-dropdown/ - that's a whole other beast. I know where you're coming from with the various "brute force" ways to make it work, but this is primarily an exercise to see if UBot can do this... ultimately if I move forward on this project I'm going to hire a real trading system developer to build the system. For right now I just want to see a rough proof of concept and wanted it to be quick & dirty with UBot. I'm almost there too, just stumped by this one little weird issue.

 

Thanks again!!

 

Jonathan