Go Back   FlashFXP Forums > > > >

ioFTPD General New releases, comments, questions regarding the latest version of ioFTPD.

Reply
 
Thread Tools Rate Thread Display Modes
Old 02-20-2010, 11:01 AM   #31
pion
Senior Member
 
Join Date: Feb 2006
Posts: 138
Default

How can it be faster, to wait for all acks to return, shut down the tcp connection, write file to disc, process it with zipscript, and THEN allow read access to it?

The problem with routing has always been there, but I suppose that slow home connection used to be to slow to notice. I don't know what you mean by 'alot of incomplete uploads', but in my experience, incomplete uploads are fairly rare. As for the case of A -> B routing causing slow transfers both in/out as it happends with people putting drftpd slaves on boxes in both Korea and Netherlands and what not, it should be easy enough to set a lower speed limit, and block the incoming source for a few.

I fail to see how 'lose most races' is positive for any party involved
pion is offline   Reply With Quote
Old 02-20-2010, 11:22 AM   #32
o_dog
Senior Member
 
Join Date: May 2007
Posts: 692
Default

it is since incomplete uploads are not that rare, you just don't see them since most people shut off the announce.

Now we turn to logic. One person uploads a slow file in a chain which in turn means all people uploads the file slow that use that site as a source, this then multiplies from the sites that are raced. Racer gets one file in race and release takes longer to complete on the site. When that user notices the upload speed he most likely aborts the transfer. now it gets uploaded again, but prolly from one of the other sites that used first as a source, thus making it fail crc, how long this goes on for differs. It's not complete so no one can finish the race and now you're left with an inomplete release that someone has to fill.

On the other hand if you only use fully uploaded files that never happens and most of the time you can max out the bw in all directions. It might take a few secs longer to get started but the result is better speeds. The only problem with this approch is that glftpd and drftpd actually allows download of partially uploaded files. It is a problem since it multiplies to other sites, over and over and over again.
__________________
ioNiNJA
o_dog is offline   Reply With Quote
Old 02-20-2010, 01:44 PM   #33
pion
Senior Member
 
Join Date: Feb 2006
Posts: 138
Default

Your logic is only valid for uploads consisting of a large ammount of files, like 30-40 equally sized files, with multiple sources for each file at any time, to simply transfer an already complete file at all times. In the matter of zip files, often only 1-2, music videos, usually 1 file, music, often ~5 files. This would result in all clients sitting and waiting for a fair ammount of time, and spread surely won't be faster.

Just imagine the 100-200mb music video files that has to be all the way uploaded before anyone can even touch it.. even with 1% of files ending up incomplete, it would have to replicate to a whole lot of servers before the spread would be comparable. In all likelyhood the source would get bombarded with requests, making an exponential ammount of servers just sit on their asses until someone else after a good ammount of time complete the file for further uploading.

Lets say 1 out of 10 x 100mb files turned incomplete at 99.99% complete, and error replicate to 6 servers. All theese servers would then max utilise 90% bw for payload without read locking. With read locking you would get a replication lag of 6x 100mb data for server 6. So from server 6 point of view, that's half the transfer! If transfer had started right away to server 6, even with the broken file, it would still have half the transfer.

To transfer one file from a source server1 to server 2 which fail, and retransfer that 1 file. Server 3 also needs to wait for that file on server 2, unless it take a shortcut and get it directly from server1. But then server1 is sending the same file to 2 servers, which would result in largely reduced bandwidth for both servers. Given that the retransfer of the incomplete file succeed, the spread of the upload is nearly twice as fast without write lock!


Feel free to correct me if I'm wrong, but I'm having a hard time understanding how blocking the file until it's fully processed can speed up the spread.
pion is offline   Reply With Quote
Old 02-20-2010, 05:02 PM   #34
o_dog
Senior Member
 
Join Date: May 2007
Posts: 692
Default

100-200mb? thats like 4s and yes if you want make an exception to the rule it's a good one. It also happens that a car ends up in the water,does that mean we should design them to be used as boats too?

as far as logic goes you kinda lost it with all that 1 2 3 4 5 back and fourth. Incomplete files are often transfered, taking up bw and making sure releases are not completed as fast as possible. Preventing incomplete dls speeds up the completion since no file is transfered under the minimum amount of bw avalible. While a slow incomplete transfer from one server ends up slowing down completion of races all over the place. Not to mention it costs alot of credits for users to transfer incomplete files since it gets taken from their credits but not added on target site. If a file gets transfered multiple times, let's say a hdd runs out of space, file gets deleted by zipscript and transfered over and over again, escalating both credit and bw consumption. How many racers abort a slow transfer to race something else? The point with the incomplete download was to speed up transfers when connections were slow and the upload time was long. Today it isn't nessecery, but sure, implent it and make it optional since both drftpd and glftpd support it, might be beneficial for racers to get access to files quickly. Just don't try to convince me that it's a good idea, cause it' really not, everyone loses on the crap, it might be needed, but that doesn't make it a good idea.
__________________
ioNiNJA
o_dog is offline   Reply With Quote
Old 02-22-2010, 05:52 AM   #35
pion
Senior Member
 
Join Date: Feb 2006
Posts: 138
Default

My back and fourth was an attempt to quantify speeds of upload completion when you have x sources, wanting to spread to xx destinations.

Slows transfers is a problem for both scenarios, however, a slow transfer on the very last file replicating down the chain is annoying to watch, and I'm all for making reading a file in use optional. However, slow transfers can be avoided by actively denying them at the server side after xx seconds, using script, then block that source for a few minutes at the server side. In addition, uploaders do change their source when hitting slow files. And not doing so make them unpopular fast. A common workaround is 'always upload largest file first' logic at the client side.

If you do the math, you'll see that an example of 5 equally large files in the upload, with 3 sources, spreading to 10 new destinations it would take 8 transfer cycles with read lock to complete it everywhere, and 5 transfers without, under optimal conditions for both scenarios (only 1 file per source uploaded at the time so server2 starts sending at the fastest possible point in time, and no incompletes). That's a 60% (!) increase in time needed for filling 10 servers.

In sub optimal conditions, it is not uncommon that all 3 sources start sending to all 10 destinations at once, making 10 servers idle for the duration. And incompletes do happend, but 1 incomplete/crc fault transfer, would result in 1 additional transfer cycle. For my example, with 2 broken transfers (with x replicas down the chain) the overall spread time would still be faster than the optimal case for read lock!

As for the credits argument, a 'racer' needs to transfer 3 incompletes for every 1 complete file to loose credits overall..

And it's Yil I want to convince!
pion is offline   Reply With Quote
Old 02-25-2010, 04:51 PM   #36
monk-
Member
 
Join Date: Oct 2007
Posts: 32
Default

its is indeed a great disadvantage compaired to glftpd that incomplete files cant be used for transfers.
speed and crc problems are only worst case scenarios which only occure 0,01% of the time, prolly even lesser. with the readlock an ioftpd site gets much less attractive to use as source than with a gl site, even if the gl site has less speed/routing.
being able to start the first rar/mkv file right after sfv upload has a speed advantage, and also increase the chance of winning a race majorly
i rather have a bit slower ioftpd that can do this, than a lightning fast process, that has wait times in the actual usage

Last edited by monk-; 02-25-2010 at 04:58 PM.
monk- is offline   Reply With Quote
Old 02-28-2010, 06:28 AM   #37
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

Yil, whare do i put these 2 help.ini files? Can i just make a empty files and stick them somewhare to make log nag go away?
Flow is offline   Reply With Quote
Old 02-28-2010, 09:02 AM   #38
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Flow: Ooops! I've completely failed to get the help files out the door! I really did think I'd finish them up quickly, but while documenting every single command and argument available to users I kept running across things I wanted to fix and, well, it's quite boring work...

To remove the error, just create help.ini and sitehelp.ini in the System dir. That should probably do it. I have finished help.ini, but still have all the 'site change' commands to do as well as a few new commands for the next release... If I don't finish them up in the next week kick me

Couple of teasers. There is a 'site perms' command now that will show you exactly what you (or any other user you indicate) can do to a file or directory. For directories it also pulls out matching actions for files/subdirs that match the path and shows you the perms for things like 'RemoveDir', 'DeleteOwn', etc. Basically I reproduced permission checking in the resolver and all the standard FTP commands and put the results in a table or two. I also made it runnable by regular users (they can only view themselves) if you want to let them see some of that info (not all fields are shown). I really think that will be quite useful to new admins and people setting up the server for the first time.

Site uptime for the FTP and the OS as well as cookies for displaying it in message files and a TCL function for it are added.

Oh, and fine grained access to the MDTM command. You can now allow access to reading the timestamp, but control who can set it via the TimeStamp and TimeStampOwn access checks. There is even an option to let only the last file uploaded by the user be modified if you want to disable modification in general, but allow that limited case which I think might be useful to some people. I can't believe people haven't complained about this...

And finally, I've been playing a lot with virtual dirs. It works for what I envisioned it to do, but now I want it even better. I'm trying to support items that don't exist as real files or dirs. Basically I want you to be able to rename, add, delete, etc items that exist only because they got pulled out of a file or database. Right now ioFTPD expects real paths to everything...

If anyone using virtual dirs has suggestions or comments now would be a good time I believe I've got a simple fix for things like SIZE, MDTM, XCRC, and downloading files from virtual dirs for which you provided a linked real dir. Those changes will just start working without you doing anything. Uploading, directory creation, and renaming will require some new code from you though and I can't promise my half of that for the next release.
Yil is offline   Reply With Quote
Old 02-28-2010, 09:23 AM   #39
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

Quote:
To remove the error, just create help.ini and sitehelp.ini in the System dir. That should probably do it.
No sir, that didn solve it. Still getting: Missing help file 'SiteHelp.ini'
Flow is offline   Reply With Quote
Old 02-28-2010, 11:32 PM   #40
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

Hmm... Yea, that didn't work. Just comment out the reference to SiteHelp.ini under [Help] in the config file. It won't know to try to load it then
Yil is offline   Reply With Quote
Old 03-01-2010, 02:28 PM   #41
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

Code:
.-------------------------------------------------------------.
|                          SITE HELP                          |
'-------------------------------------------------------------'
  |     RULES - Show site rules.                            |
  |   REQUEST - Add a request.                              |
  | REQFILLED - Mark a request as filled.                   |
  |    RESCAN - Rescan a release.                           |
  |       PRE - PRE a release.                              |
  |    INVITE - Invite yourself to the IRC channel.         |
  |      ONEL - Add or view onliners.                       |
  |       MSG - Send a message to a user or group.          |
  |  TRANSFER - Transfer credits from a section.            |
  |      DUPE - Search dupe database.                       |
  |    SEARCH - Search for a directory on the site.         |
  |       NEW - Show recent directories.                    |
  |     NUKES - Show recent nukes.                          |
  |   UNNUKES - Show recent unnukes.                        |
  |    PASSWD - Change your password.                       |
  |     STATS - Show user stats.                            |
  |   TAGLINE - Change your tagline.                        |
  |       WHO - Show who is online.                         |
  |-[User]--------------------------------------------------|
  |   ADDUSER - Add a user.                                 |
  |   DELUSER - Delete a user.                              |
  |   RENUSER - Rename a user.                              |
  |  GADDUSER - Add a user to a specific group.             |
  |     ADDIP - Add an IP to a user.                        |
  |     DELIP - Delete an IP from a user.                   |
  |     GINFO - Detailed info of a group.                   |
  |     UINFO - Detailed info of a user.                    |
  |    CHANGE - Change settings on users or groups.         |
  |     USERS - List users on site.                         |
  |    GROUPS - List groups on site.                        |
  |-[SiteOps]-----------------------------------------------|
  |    GRPADD - Add a group.                                |
  |    GRPDEL - Delete a group.                             |
  |    GRPREN - Rename a group.                             |
  |     CHGRP - Change a user's group.                      |
  |      KICK - Kick a user.                                |
  |      KILL - Kill a connection ID.                       |
  |      SWHO - Detailed who.                               |
  |      BANS - Modify banned users.                        |
  |    CONFIG - Modify the ioFTPD config.                   |
  |  SHUTDOWN - Shutdown ioFTPD.                            |
  |    REQDEL - Delete a request.                           |
  |      GIVE - Give credits to a user.                     |
  |      TAKE - Take credits from a user.                   |
  |   NEWDATE - Manually create today's dated directories.  |
  |RESETSTATS - Reset stats for a section.                  |
  | RESETUSER - Reset a user's credits and/or stats.        |
  |    WEEKLY - Add a user to the weekly allotment.         |
  |    CMDLOG - Show lines from the command log file.       |
  |    ERRLOG - Show lines from the error log file.         |
  |    SYSLOG - Show lines from the sysop log file.         |
  |-[VfsAdmin]----------------------------------------------|
  |    CHATTR - Modify symlinks and/or private dirs.        |
  |     CHMOD - Change permission of a directory or file.   |
  |     CHOWN - Change ownership of a directory or file.    |
  |      SIZE - Shows the size of a directory.              |
  |      WIPE - Wipes a directory.                          |
  |-[Nukers]------------------------------------------------|
  |      NUKE - Nuke a release.                             |
  |    UNNUKE - Unnuke a release.                           |
  |    UNDUPE - Remove a file or directory from dupe.       |
.-------------------------------------------------------------.
|        For extended help, use 'SITE HELP [command]'.        |
'-------------------------------------------------------------'
Flow is offline   Reply With Quote
Old 03-02-2010, 03:53 AM   #42
newguy
Junior Member
 
Join Date: Jul 2009
Posts: 16
Default

Quote:
Originally Posted by Yil View Post
Hmm... Yea, that didn't work. Just comment out the reference to SiteHelp.ini under [Help] in the config file. It won't know to try to load it then
I pasted the file posted by Flow and added this in system dir as SiteHelp.ini, after performing site help command ioFTPD generated a crash log.

I also noticed that i wasn't beable to perform help command with normal user although permision was set correctly.

Last edited by newguy; 03-02-2010 at 03:58 AM.
newguy is offline   Reply With Quote
Old 03-02-2010, 06:43 AM   #43
Yil
Too much time...
FlashFXP Beta Tester
ioFTPD Administrator
 
Join Date: May 2005
Posts: 1,194
Default

newguy: What Flow posted was not a valid ioFTPD .ini help file but rather what I presume is his manually updated/modified nxHelp output. Just remove the file and you will be fine.

As an aside I decided to tackle the challenge of writing an nxTools helpfile using my new format since a lot of it was already done via nxHelp. It will probably take a day or so to do, but is a good example for me on how script writers may choose to integrate their help commands into the default output in a non-trivial way. It's easy enough to create a new nxTools section, but if you want things like 'site search' to show up with directory related commands it takes a bit of planning on my part...
Yil is offline   Reply With Quote
Old 03-02-2010, 07:42 AM   #44
newguy
Junior Member
 
Join Date: Jul 2009
Posts: 16
Default

Quote:
Originally Posted by Yil View Post
newguy: What Flow posted was not a valid ioFTPD .ini help file but rather what I presume is his manually updated/modified nxHelp output. Just remove the file and you will be fine.

As an aside I decided to tackle the challenge of writing an nxTools helpfile using my new format since a lot of it was already done via nxHelp. It will probably take a day or so to do, but is a good example for me on how script writers may choose to integrate their help commands into the default output in a non-trivial way. It's easy enough to create a new nxTools section, but if you want things like 'site search' to show up with directory related commands it takes a bit of planning on my part...
Can you give an example on how a good ini file should look like?

I thought something like:
Code:
[command]
Description

[command2]
Description

[command3]
Description
newguy is offline   Reply With Quote
Old 03-02-2010, 09:30 AM   #45
Flow
Senior Member
FlashFXP Beta Tester
ioFTPD Foundation User
 
Flow's Avatar
 
Join Date: Dec 2001
Posts: 306
Default

so. Sorry for the mess. Lets all wait couple days for Yil help file.
Flow is offline   Reply With Quote
Reply

Tags
commands, enabled, path, site, symbolic

Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 09:50 PM.

Parts of this site powered by vBulletin Mods & Addons from DragonByte Technologies Ltd. (Details)