Jump to content



Photo

Reading Big Csv File To Iterate


  • Please log in to reply
5 replies to this topic

#1 flamerz

flamerz

    Member

  • Members
  • PipPip
  • 12 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 13 February 2020 - 11:32 AM

Hello, I'm just creating a blog post about reading some "big" csv file. Lets say, 100.000 lines, 2 columns, name and age.
 
I wish to load the data and iterate/sum the age variable, just for documenting it.
 
set(#total,0,"Global")
set(#row,0,"Global")
create table from file("C:\\Users\\Usuario\\Desktop\\names.csv",&dataTable)
loop($table total rows(&dataTable)) {
    set(#total,$add(#total,$table cell(&dataTable,#row,1)),"Global")
    increment(#row)
}

 
This is my UBot code. My issue is, with 100k lines this goes a bit slow, maybe its not good idea to load it on a table.
 
this is my mockup post:
 
any guidance? my post is not a competition for speed, just a guide to read/iterate a csv file in a good way. I wish to have the data loaded and do not use a driver or a database query to do a SUM. Just a simple loop.
 
thanks in advance.
 
PD: names file attached.

 

Attached Files


Edited by flamerz, 13 February 2020 - 11:35 AM.


#2 Bot-Factory (Dan)

Bot-Factory (Dan)

    Advanced Member

  • Fellow UBotter
  • PipPipPip
  • 3242 posts
  • LocationEurope/Germany
  • OS:Windows 8
  • Total Memory:8Gb
  • Framework:v4.0
  • License:Developer Edition

Posted 13 February 2020 - 01:01 PM

Ubot will have lot of memory issues with that. And performance is awful. 

 

Take a look at this:

http://network.ubots...gin-large-data/

 

Dan



#3 ATuringtest

ATuringtest

    Advanced Member

  • Members
  • PipPipPip
  • 76 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 13 February 2020 - 08:54 PM

Ubot will have lot of memory issues with that. And performance is awful. 

 

Take a look at this:

http://network.ubots...gin-large-data/

 

Dan

 

That plugin link is dead



#4 flamerz

flamerz

    Member

  • Members
  • PipPip
  • 12 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 14 February 2020 - 05:05 AM

not sure why the plugin its not working on my ubot

 

 

works:  create table from file("H:\\FileCenter\\test.csv",&test)
does nothing:  plugin command("Bigtable.dll", "Large Table From file", "H:\\FileCenter\\test.csv", &test2)



#5 flamerz

flamerz

    Member

  • Members
  • PipPip
  • 12 posts
  • OS:Windows 10
  • Total Memory:8Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 14 February 2020 - 05:06 AM

That plugin link is dead

 

yes.. seems gone. I still have the version 1.7.0.0, not sure that one was the latest.



#6 ATuringtest

ATuringtest

    Advanced Member

  • Members
  • PipPipPip
  • 76 posts
  • OS:Windows 10
  • Total Memory:More Than 9Gb
  • Framework:v4.5+, unsure
  • License:Developer Edition

Posted 14 February 2020 - 08:36 AM

Personally I moved to SQLlite far easier in the long run. There's a plugin for it and it integrates really well with C#.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users