Reading Big Csv File To Iterate

[flamerz 2]

Hello, I'm just creating a blog post about reading some "big" csv file. Lets say, 100.000 lines, 2 columns, name and age.

 

I wish to load the data and iterate/sum the age variable, just for documenting it.

 

set(#total,0,"Global")
set(#row,0,"Global")
create table from file("C:\\Users\\Usuario\\Desktop\\names.csv",&dataTable)
loop($table total rows(&dataTable)) {
    set(#total,$add(#total,$table cell(&dataTable,#row,1)),"Global")
    increment(#row)
}

 

This is my UBot code. My issue is, with 100k lines this goes a bit slow, maybe its not good idea to load it on a table.

 

this is my mockup post:

https://translate.google.com/translate?hl=en&sl=es&tl=en&u=https%3A%2F%2Fscraper.es%2Fabrir-archivos-csv%2F

 

any guidance? my post is not a competition for speed, just a guide to read/iterate a csv file in a good way. I wish to have the data loaded and do not use a driver or a database query to do a SUM. Just a simple loop.

 

thanks in advance.

 

PD: names file attached.

 

names.csv

Edited February 13, 2020 by flamerz

[Bot-Factory 602]

Ubot will have lot of memory issues with that. And performance is awful. 

 

Take a look at this:

http://network.ubotstudio.com/forum/index.php/topic/16308-free-plugin-large-data/

 

Dan

[ATuringtest 14]

Ubot will have lot of memory issues with that. And performance is awful. 

 

Take a look at this:

http://network.ubotstudio.com/forum/index.php/topic/16308-free-plugin-large-data/

 

Dan

 

That plugin link is dead

[flamerz 2]

not sure why the plugin its not working on my ubot

 

 

works:  create table from file("H:\\FileCenter\\test.csv",&test)
does nothing:  plugin command("Bigtable.dll", "Large Table From file", "H:\\FileCenter\\test.csv", &test2)

[flamerz 2]

That plugin link is dead

 

yes.. seems gone. I still have the version 1.7.0.0, not sure that one was the latest.

[ATuringtest 14]

Personally I moved to SQLlite far easier in the long run. There's a plugin for it and it integrates really well with C#.