Go Back   FlashFXP Forums > > > >

NeoXed's scripts nxAutoNuke, nxTools, etc.

Reply
 
Thread Tools Rate Thread Display Modes
Old 03-14-2005, 10:32 PM   #1
CoLt-[45]
Member
FlashFXP Registered User
ioFTPD Foundation User
 
Join Date: May 2003
Posts: 32
Question NXTools Memory Bug?

Not sure exactly what's happening here, but when i edit this in ioftpd.ini
Code:
[FTP_Pre-Command_Events]
stor = TCL ..\scripts\nxTools\nxDupe.itcl prestor
The memory has started to increased by approx 1mb or 2mb everytime a file is uploaded till ioftpd.exe has crashed.

When I disable it just like this
Code:
[FTP_Pre-Command_Events]
;stor = TCL ..\scripts\nxTools\nxDupe.itcl prestor
Works very stable, but I need this nice script for my site!

Any idea? Suggestion?

Cheers in advance
CoLt-[45] is offline   Reply With Quote
Old 03-14-2005, 10:53 PM   #2
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

nxTools doesn't "directly" allocate any memory, since it's a Tcl script. However, it may be caused an extension used by nxTools, such as SQLite. I've never encounted this problem nor heard of anything like it.

I'll look into it when I have time later this week.
neoxed is offline   Reply With Quote
Old 04-06-2005, 04:38 AM   #3
PaRaNoiD_JaCK
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Join Date: Mar 2005
Posts: 101
Default

I have this problem aswell. At the moment im just rebooting the ftpd every 24 hours to purge the memory allocated by ioftpd.

n00b question here:

By disabling

mkd = TCL ..\scripts\nxTools\nxDupe.tcl premkd
stor = TCL ..\scripts\nxTools\nxDupe.itcl prestor

what features are lost with the script?
PaRaNoiD_JaCK is offline   Reply With Quote
Old 04-06-2005, 04:16 PM   #4
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

Which version of ioFTPD and Windows (service pack as well) are you two using?

The prestor and premkd events are used by the file and directory dupe checker, by disabling them dupe checking will not be enforced.
neoxed is offline   Reply With Quote
Old 04-06-2005, 05:14 PM   #5
PaRaNoiD_JaCK
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Join Date: Mar 2005
Posts: 101
Default

Running ioFTPD version 5-8-5r

Windows 2003 server standard SP1
PaRaNoiD_JaCK is offline   Reply With Quote
Old 04-06-2005, 05:29 PM   #6
fobban
Senior Member
FlashFXP Registered User
ioFTPD Foundation User
 
Join Date: Jan 2003
Posts: 186
Default

Quote:
Originally posted by NeoXed
nxTools doesn't "directly" allocate any memory, since it's a Tcl script. However, it may be caused an extension used by nxTools, such as SQLite. I've never encounted this problem nor heard of anything like it.

I'll look into it when I have time later this week.
Actually there is a similar problem with bounty's newdir script.
If you have a huuuuge dupedir file and make a site dupe you see that the size of dupedir.log is added ioFTPD's used memory.
I think one have to unset variables when you've finished using them.
fobban is offline   Reply With Quote
Old 04-06-2005, 06:34 PM   #7
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

fobban: the file and directory names are stored in a SQLite database and accessed using the TclSQLite extension, not a read from text file.

On a side note: The variable doesn't have to be unset if it's a procedural/local variable, since Tcl's reference counting will destroy it once the function exits. If the variable is scoped in a namespace (i.e. the global namespace), it would have to be unset. This doesn't apply to nxTools, since databases are not accessed in the same manner that newdir accesses log files.
neoxed is offline   Reply With Quote
Old 04-16-2005, 03:04 PM   #8
Freestyler
Junior Member
ioFTPD Registered User
 
Join Date: Nov 2003
Posts: 19
Default

I see my memory usage grow slightly when i upload a release, since i started with a empty db.

I tested with the db of PaRaNoiD_JaCK and saw the memory usage grow from 11MB to 20MB after 15 uploads.

NXTools seems to be pretty tight, but the memory problem is a showstopper for me.

I also run on Windows 2003 SP1. Can we somehow help debugging?
Freestyler is offline   Reply With Quote
Old 04-16-2005, 06:23 PM   #9
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

It seems this bug is specific to Windows 2003 SP1; I've only tested nxTools on Windows XP SP2 and Windows 2003 (no SP1).

Do any of you have any entries in logs\nxError.log or logs\SystemError.log?
neoxed is offline   Reply With Quote
Old 04-16-2005, 06:35 PM   #10
Freestyler
Junior Member
ioFTPD Registered User
 
Join Date: Nov 2003
Posts: 19
Default

Clearly im not ready with the setup. I get this line a lot in NxError.log:

04-17-2005 01:32:20 - LoadScript : unable to source "../scripts/nxAutoNuke.cfg": couldn't read file "../scripts/nxAutoNuke.cfg": no such file or directory

Two errors in systemlog:

04-16-2005 21:30:16 "..\scripts\nxTools\nxDupe.tcl" terminated abnormally
--------------------------- ErrorInfo ----------------------------
database is locked
while executing
"FileDb eval {INSERT INTO DupeFiles (TimeStamp,UserName,GroupName,FileName) VALUES($TimeStamp,$user,$group,$FileName)}"
(procedure "UpdateFiles" line 11)
invoked from within
"UpdateFiles $Action $VirtualPath"
(procedure "UpdateLog" line 9)
invoked from within
"UpdateLog "UPLD" $VirtualPath"
("upload" arm line 4)
in04-16-2005 21:39:16 "..\scripts\nxTools\nxDupe.tcl" terminated abnormally
--------------------------- ErrorInfo ----------------------------
database is locked
while executing
"FileDb eval {INSERT INTO DupeFiles (TimeStamp,UserName,GroupName,FileName) VALUES($TimeStamp,$user,$group,$FileName)}"
(procedure "UpdateFiles" line 11)
invoked from within
"UpdateFiles $Action $VirtualPath"
(procedure "UpdateLog" line 9)
invoked from within
"UpdateLog "UPLD" $VirtualPath"
("upload" arm line 4)
in
Freestyler is offline   Reply With Quote
Old 04-17-2005, 07:20 AM   #11
CoLt-[45]
Member
FlashFXP Registered User
ioFTPD Foundation User
 
Join Date: May 2003
Posts: 32
Default

Quote:
Originally posted by NeoXed
It seems this bug is specific to Windows 2003 SP1; I've only tested nxTools on Windows XP SP2 and Windows 2003 (no SP1).

Do any of you have any entries in logs\nxError.log or logs\SystemError.log?
I get that problems even if i'm using Windows 2000 SP4 or Windows XP SP2

I've tried the new version, for some reason - i can't get it to work! :/ (of course i haven't finished my ioftpd setup )

There's one thing i'd like to know HOW do I add all the filenames into the database? - is there's a way round to it rather than having to add via ftp? - like a batch file or something?

Thanks
CoLt-[45] is offline   Reply With Quote
Old 04-17-2005, 07:23 AM   #12
PaRaNoiD_JaCK
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Join Date: Mar 2005
Posts: 101
Default

I get similar error as freestyler:

04-17-2005 07:21:44 "..\scripts\nxTools\nxDupe.tcl" terminated abnormally
--------------------------- ErrorInfo ----------------------------
database is locked
while executing
"FileDb eval {INSERT INTO DupeFiles (TimeStamp,UserName,GroupName,FileName) VALUES($TimeStamp,$user,$group,$FileName)}"
(procedure "UpdateFiles" line 11)
invoked from within
"UpdateFiles $Action $VirtualPath"
(procedure "UpdateLog" line 9)
invoked from within
"UpdateLog "UPLD" $VirtualPath"
("upload" arm line 4)
in

This error doesnt happen with every upload. The memory increase does tho.

PaRaNoiD_JaCK is offline   Reply With Quote
Old 04-17-2005, 01:24 PM   #13
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

Quote:
Originally posted by CoLt-[45]
There's one thing i'd like to know HOW do I add all the filenames into the database? - is there's a way round to it rather than having to add via ftp? - like a batch file or something?
Check the manual for the "SITE REBUILDLOGS" command.

Quote:
Originally posted by CoLt-[45]
I get that problems even if i'm using Windows 2000 SP4 or Windows XP SP2
ioFTPD recycles Tcl interpreters so you will notice an increase in memory each time a new interpreter is created. As far as I know, none of the interpreters are destroyed - so all of the Tcl code stays in memory. My guess is the number of Tcl interpreters is related to the number of worker threads.

Do any of you have Worker_Threads set to some extreme or unreasonable value?

http://www.ioftpd.com/board/showthre...&threadid=3970

Quote:
Originally posted by darkone
Optimal amount for io threads, is 2x number of logical cpus.
Optimal amount for worker threads on site that runs lots of scripts, is ~half of max users online.
Optimal amount for encryption threads, is number of logical cpus.
neoxed is offline   Reply With Quote
Old 04-17-2005, 02:25 PM   #14
Freestyler
Junior Member
ioFTPD Registered User
 
Join Date: Nov 2003
Posts: 19
Default

This is what I use:

Io_Threads = 2 # Number of io threads
Worker_Threads = 5 # Number of worker threads
Encryption_Threads = 2 # Number of encryption threads
Freestyler is offline   Reply With Quote
Old 05-09-2005, 12:53 AM   #15
neoxed
Too much time...
FlashFXP Beta Tester
ioFTPD Scripter
 
Join Date: May 2003
Posts: 1,326
Default

Alright, turns out SQLite was infact the source of the memory leaks. More specifically, the Tcl extension to SQLite.

I built the latest CVS version of TclSQLite, which includes a fix for a memory leak discovered shortly after the release of SQLite v3.2.1. http://www.sqlite.org/cvstrac/chngview?cn=2435

You may download the updated libraries from here:
http://www.ioftpd.com/scripts/script.php?id=167

Edit: I posted the wrong ones, please redownload them.
neoxed is offline   Reply With Quote
Reply

Tags
memory, prestor, scriptsnxtoolsnxdupe.itcl, tcl, [ftp_pre-command_events]

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 02:31 AM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)