Any ideas on this

[UBotBuddy 331]

I am compiling a report here at work from our main website and UBot typically does really well but this problem I am having a bit of a problem.

 

The content was apparently created by Word and it has those stupid characters embedded throughout the content. You know, the reverse quotes. Those kind of characters.

 

Anyway, when I am building my list the $search function shows them in the box ok but when I apply it to the Add List the characters appear. I have tried using Choose Attribute as well and then I use innertext, value, or even outertext but the effect is still the same.

 

I have also tried using javascript to do a search & replace but for some reason it is not working. Not sure if that is due to the Beta version.

 

I have done Search & Replace before so I am pretty sure its not working using these high characters.

 

Very strange indeed.

[Pete 121]

Really depends on how many different characters you need to remove before you page scrap

 

Choose attribute

Attribute

Innertext

SearchString

Bad characters

Method

exact match

 

change choose attribute

Attribute

Innertext

 

Newvalue

$nothing

[UBotBuddy 331]

Yep, that's what I did as well but it no workie.

[MiriamMB 63]

One thing you can do, since the characters are strange and kind of hard to pin down, is to use wildcards in the choose by attribute command. Wild cards are asterisks which represents any number of characters and a question mark represents just one character.

 

For example,

if you are trying to scrape this sentence:

 

-the dog bit the "sausage"

 

You could find it by having your search text as:

 

-the dog bit the ?sausage?

 

which would then scrape the sentence with the word "sausage", with the two question marks taking the place of the quotation marks.

 

It is best to use question marks because they are cleaner than using an asterisk.

[UBotBuddy 331]

Wildcards are not the problem. It is the result of the scrape. I see Quotes in the content but after the scrape I see odd characters (which come from MS Word). I cannot predict where those characters will appear to even use the wildcards since it is Content related.

 

I wish it would work.

 

Buddy

[Guest Jim]

Attached is a little example I put together that uses a javascript regular expression to strip all characters that aren't on a 'whitelist'

 

and here's the javascript for easy copypasting (goes in an $eval)

"{1}".replace(/[^a-zA-Z 0-9\-_\`\~\!\@\#\$\%\^\&\*\(\)\.\,\:\;\?\']+/g,"");

{1} being your scraped data variable

strip_crazy_characters.ubot

[alcr 135]

Is it UTF8 characters or is the symbols messed up in word? Because 1. UBot does not support UTF8 and 2. Use open office instead

[Frank 177]

Word does a lot of very screwy things. I've had issues in many other places too. Maybe saving out of word in 'open document format' may help. I believe that this is the same format that openoffice uses.

 

BUT, that's just a big guess.

 

Frank

[UBotBuddy 331]

I wish I could use OpenOffice for this but I am scraping from the site not building it. My goal is to take this data and autoload it on another site. Its a joke site so these are small paragraphs that I am grabbing.

 

http://www.ajokeaday.com/ChisteDelDia.asp

 

Buddy

[Pete 121]

What are you saving it to?

I am thinking excel as when i save it to a txt file them bad characters turn out to be , " ! and so on

so maybe saving to a .txt file then reading it back into ubot to save in your fomat

[Net66 54]

Attached is a little example I put together that uses a javascript regular expression to strip all characters that aren't on a 'whitelist'

 

and here's the javascript for easy copypasting (goes in an $eval)

"{1}".replace(/[^a-zA-Z 0-9\-_\`\~\!\@\#\$\%\^\&\*\(\)\.\,\:\;\?\']+/g,"");

{1} being your scraped data variable

 

Thanks for this Jim! Another gem from the Javascript expert :-)

[UBotBuddy 331]

Jim. You da'Man!

 

Thanks that did the trick PERFECTLY!

[UBotBuddy 331]

Hmmm..

 

It works BUT it totally strips out the first two jokes. I mean gone. But five are left perfectly.