Regex for matching url

[Abs* 12]

HI Guys -

 

just wondering if anyone could help with matching a url when checking the search engines - so it wouldnt cause an issue if it was a .com or .co.uk etc

 

thanks

 

abs

[LoWrIdErTJ - BotGuru 904]

have this one a go..

 

^(http(?:s)?\:\/\/[a-zA-Z0-9\-]+(?:\.[a-zA-Z0-9\-]+)*\.[a-zA-Z]{2,6}(?:\/?|(?:\/[\w\-]+)*)(?:\/?|\/\w+\.[a-zA-Z]{2,4}(?:\?[\w]+\=[\w\-]+)?)?(?:\&[\w]+\=[\w\-]+)*)$

[Abs* 12]

have this one a go..

 

^(http(?:s)?\:\/\/[a-zA-Z0-9\-]+(?:\.[a-zA-Z0-9\-]+)*\.[a-zA-Z]{2,6}(?:\/?|(?:\/[\w\-]+)*)(?:\/?|\/\w+\.[a-zA-Z]{2,4}(?:\?[\w]+\=[\w\-]+)?)?(?:\&[\w]+\=[\w\-]+)*)$

 

Hi - Thanks alot - will give it a shot -

 

Just wondering how on earth your writing such complex regex

 

I dont have any experience with regex and find it a nightmare to try and understand

 

I understand parts of it like it could have a-z lowercase or A-Z uppercase 0-9 http or could have the s

 

But for the life of me im lost with all this complex code man

 

A few pro tips would go a long way

 

thanks

 

abs

[UBotBuddy 331]

Just look in the pinned thread in this sub-forum and you will see a link to Rubular.

 

http://www.rubular.com/r/nidQpOizwC

 

Aww heck there it is

[JohnB 255]

abs, it can be very intimidating at first, but if you play with it for a few days it becomes second nature. It is actually nowhere near as difficult as it looks. Let us know if we can help out!

 

John

[UBotBuddy 331]

Oh I hate regex. That's why I have friends like John. I'll whine about it and then he'll do it for me. LOL

 

Or make some fatal attempt at it and then someone with provide a Winner of a regex formula.

 

Mu ah ha ha {evil laugh}

[Pete 121]

I can see these regex posts will keep popping up again and again would it not help user’s if the expression generated where to be kept in a library?

[LoWrIdErTJ - BotGuru 904]

I've used php for web development for around 14 years now. Doing so I have had to use a lot of regex on things ranging from sign up to filtering. So it kinda comes second nature once you get the hang of it.

 

There are programs out there though to make it a lot easier for building regex code as well though

Here is a tester for regex code

http://regexlib.com/RETester.aspx

Everything you want to know about regex (cheat sheet)

http://regexlib.com/CheatSheet.aspx

 

Regex Resources

http://regexlib.com/Resources.aspx

[Abs* 12]

Hay thanks alot guys

 

Yes will certainly look into all the regex guides - Im sure it will only help so better to get stuck in asap rather then waiting

 

Thanks for all your help guys

 

Will be trying to work on the regex provided today

 

Abs

[Abs* 12]

hi guys

 

Ok I guess I aint got a clue how to use these regex codes with evals

 

Ive looked at the tutorials and also the tutorial on ubotjunkie by frank

 

The issue that im having is checking to see if the url is on the page - So I have a UI field for the keyword and also the url field -

 

The url field may not be the full url - Just want to see if the actual url ranks regardless of it being a home page or a inner page

 

SO am a little confused where to place the reg expression to evaluate if it is noted

[Abs* 12]

How about a $50 donation to anyone that can PM me a working google rank checker using the regex noted.

 

It should do a google search based on a keyword entered in the UI field - and search for a url noted regardless of it being a home or inner page.

 

It should then bring back the google position it ranks in - Based on Organic searches -

 

You can do this based on a while command so that it indefinatly loops till found-

 

I think there is a free google rank checker .ubotfile which already does this but not based on regex and it doesnt work with .co.uk or similar domains

 

I will be working with the .ubot file to enable it to work with proxies and from lists maybe even sockets. If you want to include this in the bot then feel free to do so -Would help me loads - If you feel $50 is too less then let me know how much you feel is enough and we can work somehting out .

 

Would love to do it myself but just aint got the hours in the day at the moment-

 

Thanks

 

Abs

[LoWrIdErTJ - BotGuru 904]

Here is the specific Regex for looking up all sub domains, http, https, ftp, imap etc for UK domains

 

example: .co.uk or .me.uk or .plc.uk or .gov.uk etc etc etc

 

 

Uk based URLs

^(ht|f)tp((?<=http)s)?://((?<=http://)www|(?<=https://)www|(?<=ftp://)ftp)\.(([a-z][0-9])|([0-9][a-z])|([a-z0-9][a-z0-9\-]{1,2}[a-z0-9])|([a-z0-9][a-z0-9\-](([a-z0-9\-][a-z0-9])|([a-z0-9][a-z0-9\-]))[a-z0-9\-]*[a-z0-9]))\.(co|me|org|ltd|plc|net|sch|ac|mod|nhs|police|gov)\.uk$

 

US based URLs

^(http(?:s)?\:\/\/[a-zA-Z0-9\-]+(?:\.[a-zA-Z0-9\-]+)*\.[a-zA-Z]{2,6}(?:\/?|(?:\/[\w\-]+)*)(?:\/?|\/\w+\.[a-zA-Z]{2,4}(?:\?[\w]+\=[\w\-]+)?)?(?:\&[\w]+\=[\w\-]+)*)$

 

 

Using both in a scan will produce all possible Urls for your search

I will try to put together a sample bot for you.

 

TJ

[Abs* 12]

Oh thanks man

[LoWrIdErTJ - BotGuru 904]

Let me ask this. Are you searching google for a specific url (UI inserted) and trying to find what page its on?

 

I guess let me know a bit more functionality and the information that is going to be supplied to the bot for it to search.

[LoWrIdErTJ - BotGuru 904]

I have made the attached for ya. Have a look and see if this works for you.

 

• 
   
  
• Set domain your looking for on pages
  
• Set keyword to search by
  
• Set number of pages back in Engine you want to look for it
  
• Set which engine you want to use
  

 

 

However regex was not a matter in this case. I was able to set it up without having to use regex for the search.

Let me know if im doing something incorrectly for what your wanting from the engines?

 

 

Bot will return where you are found on what pages of the rankings for the keyword searched

if no results within the pages back it will tell you so at the end of the bot.

 

TJ

find site in serps.ubot

[LoWrIdErTJ - BotGuru 904]

Did the above work for you?

[Abs* 12]

Did the above work for you?

 

HI - Thanks alot

 

I have been speaking to John and it looks like I got myself very confused - Regex cant be used to in the way I wanted it to - No need for regex

 

ive looked at the google code you have noted - Its kind of cool the way your using the regex to scrape the result stats -

 

Thanks

[UBotBuddy 331]

I only use regex as a last resort.

[LoWrIdErTJ - BotGuru 904]

yeah regex is unforgiving but when absolutely needed to be very specific on the data your trying to find it is good to use.

 

TJ

[James J 3]

@Abs I sent you a pm.

 

James