Super Dave 26 Posted August 13, 2010 Report Share Posted August 13, 2010 Needed a solution to removing duplicate lines in massive text files (+15mb) before processing in uBot. uBot's own ignore-list-duplicate function worked but with the file sizes I was dealing with it wouldn't scale. I've used 'sort' to do this through a terminal session but now I needed a way to use sort with ubot. After a little research I found a developer's site with this description:Here are some ports of common GNU utilities to native Win32. In this context, native means the executables do only depend on the Microsoft C-runtime (msvcrt.dll) and not an emulation layer like that provided by Cygwin tools. For a little perspective, I killed a bot after an hour into processing a 600k line file... with the "sort.exe -u" utility processed the file in less than 15 seconds. Enjoy: http://sourceforge.net/projects/unxutils/ Quote Link to post Share on other sites
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.