Jump to content
UBot Underground

Scraping a 3 cells from every row of a table and inserting into mysql directly

Recommended Posts

I am new to uBot, but not new to JS, PHP and scraping with dom html.  What I'm trying to do seems simple in uBot, yet I'm having a tough time with it.  The built-in table scrape works fine (entire table dumps to a *.csv super easily and quickly) - but that doesn't scale well for what I need.   Here is the basic outline; in my script I connect to a mysql database and retrieve a url and a target_id.  so lets say I'm scraping target_id #15, - uBot will store "15" in a "id" variable and navigate to www.webaddress.com/details.php?item=123456789. On that page is a small table - about 10 rows total, and the two columns I am interested in are "description (varchar) " and  "cost (decimal with a $ and ,) - there are other columns but I do not need those. Assume column 3 for "description" and column 6 for "cost".  Sometimes there are more than 10 rows, sometimes less like 6 or 8.  One value outside the table is a div the does not have an div id, but always has the words "Grand Total:" before a decimal value with again a $ and a ,  -- What I would like to insert is all 10 rows (or however many there are) as id:15, description: wonderful red item, cost: 10.99 and then a final insert into another mysql table -> id:15, total:89.65


Thanks in advance for any assistance!!

Link to post
Share on other sites

if you know the ending point of the data you want to scrape, then page scrape the page and then use heopas get before to grab everything before that Grand Total. Parse that result and add it to a list, then add that list to a row in a table. Parsing that data will obviously be the biggest challenge. Good luck! You already know your target id's so it should be easy.

Link to post
Share on other sites

I guess my first challenge is scraping the table without using the built in table scrape function- that function only outputs to a file, which isn't scalable.  I'm pretty sure I can figure out the grand total thing, but the table scrape directly into mysql is imperative.

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...