FlashFXP on XP SP2 - MP3 uploads choke

[UglyBunz]

OS: Windows XP SP2, from streamlined fresh install, no further patches (none available yet), registered
Firewall: Windows XP firewall with FlashFXP set in exceptions
FlashFXP: Version 3, build 1015, registered
Antivirus: AVG Free Edition V6.0.745, database 497
NAT: No, on static registered IP address
Router: Allied Telesyn AT-AR250E configured as router (no NAT, firewall, filters etc.) /29 registered IP pool
LAN: No, router configured for WAN only
Internet feed: 2048/256 ADSL

Note: I have previously used my router/net feed with FlashFXP 2.x with no problems to the remote servers I have since had problems with, so the problem appears to be a FlashFXP 3/XP SP2 interaction. Unfortunately I upgraded to FFXP 3 at the same time that I installed XP SP2!

Symptoms: When uploading MP3 files to different remote FTP servers, FlashFXP upload falls to 0bps on some files and eventually times out. On resume, after the rollback data has been uploaded, the upload chokes in exactly the same position as before. The upload choke ALWAYS occurs on the same files in the same position, regardless of which remote server I am sending to. The remote servers are mainly Serv-U, BSD FTPd, and NCFTPd. This is simple FTP, not FXP. Setting passive or port mode has no effect, nor does setting keep-alives. Disabling my AV software also has no effect. I cannot really disable my firewall, for obvious reasons!

This is completely consistent - it always occurs on the same files in the same places. I have not been able to try a different firewall yet, as my previous firewall (Kerio Personal Firewall) didn't seem to be very happy with SP2 (SSH sessions constantly timed out on me - the Windows firewall was disabled at the time. I have not experienced this SSH problem with the Windows firewall).

I have tried uploading the same MP3 files that I cannot upload with FlashFXP using the Windows command-line FTP in binary mode, and the transfers work successfully. So FTP transfers appear to be possible: the Windows firewall alone is not breaking these.

Workaround: This problem occurs with maybe 1 in 5 MP3 files: there are almost always one or two tracks in my queue that exhibit the problem and mess up the upload. So I have taken to compressing the MP3s into multi-segment RARs (10MB segments) with WinRAR 3. The RAR files seem to upload with no problems. I know there's no benefit to compressing MP3s, but I do it to transform the files, on the assumption that some data sequence within the MP3s (and probably common in MP3 files, judging from my experiences) is causing the transfers to break.

I can supply specimen files that always fail to upload via FlashFXP in my environment if required.

Here is a sample log file with some specific info edited out (the remote server is really Serv-U 4.1 with spoofed info):

WinSock 2.0 -- OpenSSL 0.9.7d 17 Mar 2004
[R] Connecting to x.x.x -> IP=y.y.y.y PORT=21
[R] Connected to x.x.x
[R] 220 BunnySoft FTPd V0.93c - © BunnySoft Inc.
[R] USER w
[R] 331 User name okay, need password.
[R] PASS (hidden)
[R] 230 User logged in, proceed.
[R] SYST
[R] 215 UNIX Type: L8
[R] FEAT
[R] 211-Extension supported
[R] MDTM
[R] MDTM YYYYMMDDHHMMSS[+-TZ] filename
[R] SIZE
[R] SITE PSWD;EXEC;SET;INDEX;ZONE;CHMOD;MSG
[R] REST STREAM
[R] 211 End
[R] PWD
[R] 257 "/d:" is current directory.
[R] TYPE A
[R] 200 Type set to A.
[R] PORT z,z,z,z,7,207
[R] 200 PORT Command successful.
[R] LIST -al
[R] 150 Opening ASCII mode data connection for /bin/ls.
[R] 226 Transfer complete.
[R] List Complete: 1 KB in 0.42 seconds (3.0 KB/s)
[R] CWD upload
[R] 250 Directory changed to /d:/upload
[R] PWD
[R] 257 "/d:/upload" is current directory.
[R] PORT z,z,z,z,7,209
[R] 200 PORT Command successful.
[R] LIST -al
[R] 150 Opening ASCII mode data connection for /bin/ls.
[R] 226 Transfer complete.
[R] List Complete: 906 bytes in 0.44 seconds (2.0 KB/s)
[R] CWD w
[R] 250 Directory changed to /d:/upload/w
[R] PWD
[R] 257 "/d:/upload/w" is current directory.
[R] PORT z,z,z,z,7,211
[R] 200 PORT Command successful.
[R] LIST -al
[R] 150 Opening ASCII mode data connection for /bin/ls.
[R] 226 Transfer complete.
[R] List Complete: 429 bytes in 0.45 seconds (0.9 KB/s)

NOTE NOTE - the next entries are me renaming the same file that I successfully uploaded with Windows FTP (see later log)

[R] RNFR Freeform - 03 - Saigon.mp3
[R] 350 File or directory exists, ready for destination name
[R] RNTO xFreeform - 03 - Saigon.mp3
[R] 250 RNTO command successful.
[R] PORT z,z,z,z,7,213
[R] 200 PORT Command successful.
[R] LIST -al
[R] 150 Opening ASCII mode data connection for /bin/ls.
[R] 226 Transfer complete.
[R] List Complete: 430 bytes in 0.38 seconds (1.1 KB/s)
[R] TYPE I
[R] 200 Type set to I.
[R] PORT z,z,z,z,7,215
[R] 200 PORT Command successful.
[R] STOR Freeform - 03 - Saigon.mp3
[R] 150 Opening BINARY mode data connection for Freeform - 03 - Saigon.mp3.
[R] 426 Data connection closed, receive file Freeform - 03 - Saigon.mp3 aborted.
[R] Transfer Failed!
[R] PORT z,z,z,z,7,217
[R] 200 PORT Command successful.
[R] REST 2228224
[R] 350 Restarting at 2228224. Send STORE or RETRIEVE.
[R] STOR Freeform - 03 - Saigon.mp3
[R] 150 Opening BINARY mode data connection for Freeform - 03 - Saigon.mp3.
[R] Connection lost: x.x.x
[R] Attempting to Reconnect.
[R] Connecting to x.x.x -> IP=y.y.y.y PORT=21 (attempt # 1)
[R] Connected to x.x.x
[R] 220 BunnySoft FTPd V0.93c - © BunnySoft Inc.
[R] USER w
[R] 331 User name okay, need password.
[R] PASS (hidden)
[R] 230 User logged in, proceed.
[R] SYST
[R] 215 UNIX Type: L8
[R] FEAT
[R] 211-Extension supported
[R] MDTM
[R] MDTM YYYYMMDDHHMMSS[+-TZ] filename
[R] SIZE
[R] SITE PSWD;EXEC;SET;INDEX;ZONE;CHMOD;MSG
[R] REST STREAM
[R] 211 End
[R] PWD
[R] 257 "/d:" is current directory.
[R] CWD /d:/upload/w/
[R] 250 Directory changed to /d:/upload/w
[R] PWD
[R] 257 "/d:/upload/w" is current directory.
[R] TYPE A
[R] 200 Type set to A.
[R] PORT z,z,z,z,7,222
[R] 200 PORT Command successful.
[R] LIST -al
[R] 150 Opening ASCII mode data connection for /bin/ls.
[R] 226 Transfer complete.
[R] List Complete: 511 bytes in 0.44 seconds (1.1 KB/s)
[R] TYPE I
[R] 200 Type set to I.
[R] PORT z,z,z,z,7,224
[R] 200 PORT Command successful.
[R] REST 2228224
[R] 350 Restarting at 2228224. Send STORE or RETRIEVE.
[R] STOR Freeform - 03 - Saigon.mp3
[R] 150 Opening BINARY mode data connection for Freeform - 03 - Saigon.mp3.
[R] Connection lost: x.x.x
[R] Attempting to Reconnect.
[R] Connecting to x.x.x -> IP=y.y.y.y PORT=21 (attempt # 1)
[R] Connected to x.x.x
[R] 220 BunnySoft FTPd V0.93c - © BunnySoft Inc.
[R] USER w
[R] 331 User name okay, need password.
[R] PASS (hidden)
[R] 230 User logged in, proceed.
[R] SYST
[R] 215 UNIX Type: L8
[R] FEAT
[R] 211-Extension supported
[R] MDTM
[R] MDTM YYYYMMDDHHMMSS[+-TZ] filename
[R] SIZE
[R] SITE PSWD;EXEC;SET;INDEX;ZONE;CHMOD;MSG
[R] REST STREAM
[R] 211 End
[R] PWD
[R] 257 "/d:" is current directory.
[R] CWD /d:/upload/w/
[R] 250 Directory changed to /d:/upload/w
[R] PWD
[R] 257 "/d:/upload/w" is current directory.
[R] List (cached)
[R] List Complete: 511 bytes in 0.08 seconds (6.4 KB/s)
[R] TYPE I
[R] 200 Type set to I.
[R] PORT z,z,z,z,7,229
[R] 200 PORT Command successful.
[R] REST 2228224
[R] 350 Restarting at 2228224. Send STORE or RETRIEVE.
[R] STOR Freeform - 03 - Saigon.mp3
[R] 150 Opening BINARY mode data connection for Freeform - 03 - Saigon.mp3.
[R] Connection lost: x.x.x
[R] Attempting to Reconnect.
[R] Connecting to x.x.x -> IP=y.y.y.y PORT=21 (attempt # 1)
[R] Connected to x.x.x
[R] 220 BunnySoft FTPd V0.93c - © BunnySoft Inc.
[R] USER w
[R] 331 User name okay, need password.
[R] PASS (hidden)
[R] 230 User logged in, proceed.
[R] SYST
[R] 215 UNIX Type: L8
[R] FEAT
[R] 211-Extension supported
[R] MDTM
[R] MDTM YYYYMMDDHHMMSS[+-TZ] filename
[R] SIZE
[R] SITE PSWD;EXEC;SET;INDEX;ZONE;CHMOD;MSG
[R] REST STREAM
[R] 211 End
[R] PWD
[R] 257 "/d:" is current directory.
[R] CWD /d:/upload/w/
[R] 250 Directory changed to /d:/upload/w
[R] PWD
[R] 257 "/d:/upload/w" is current directory.
[R] List (cached)
[R] List Complete: 511 bytes in 0.08 seconds (6.4 KB/s)
[R] TYPE I
[R] 200 Type set to I.
[R] PORT z,z,z,z,7,234
[R] 200 PORT Command successful.
[R] REST 2228224
[R] 350 Restarting at 2228224. Send STORE or RETRIEVE.
[R] STOR Freeform - 03 - Saigon.mp3
[R] 150 Opening BINARY mode data connection for Freeform - 03 - Saigon.mp3.

Got fed up here! The timeouts means it takes quite some time to generate this amount of logfile.

Here's the same file being uploaded with Windows FTP (command line heaven!). I uploaded this file prior to the FFXP transfer logged above, hence the file rename in the above log:

ftp: 358 bytes received in 0.00Seconds 358000.00Kbytes/sec.
ftp> binary
200 Type set to I.
ftp> put "Freeform - 03 - Saigon.mp3"
200 PORT Command successful.
150 Opening BINARY mode data connection for Freeform - 03 - Saigon.mp3.
226 Transfer complete.
ftp: 10897119 bytes sent in 454.47Seconds 23.98Kbytes/sec.
ftp> quit
221 Goodbye!

Note that I had to add Windows FTP to my Windows firewall exceptions in order to use it.

I intend to experiment with other firewalls shortly, as the Windows one is mainly outwards-looking and really offers next to no tweakability, so I'll follow up here with my experiences - whether things improve or not.

Thanks,
UglyBunz

[blubb-]

Hi,

did you try enabling the 'Send noops during transfer'-option? maybe it helps to keep the connection.

cu

[UglyBunz]

Originally posted by blubb-
Hi,

did you try enabling the 'Send noops during transfer'-option? maybe it helps to keep the connection.

cu

Hi,

Yeah, as I said in my post, setting keep-alives (send noops during transfer) makes no difference.

If the problem was due to a simple control channel timeout from the server then surely I would see the same problem with Windows FTP?

Thanks for your input :)

UglyBunz

[blubb-]

Hi,

sorry, i didn't read that before ;)
Maybe you should try an older version of flashfxp, this bug would have been noticed if it would exist in previous versions of flashfxp.

cu

[bigstar]

Have you tried uploading to another ftp servers?

I've never heard of "BunnySoft FTPd V0.93c - © BunnySoft Inc." do you know where one could download this ftp server? I'd be interested in installing it to see if I could reproduce the problem.

FlashFXP does not do anything with the file data prior to uploading, the file data is sent directly to the socket. If there is a problem sending files that contain certain data then this issue is outside the scope of FlashFXP.

[DYN_DaTa]

Bigstar:

Originally posted by UglyBunz
... (the remote server is really Serv-U 4.1 with spoofed info) ...

[bigstar]

Ahhh, Thanks for the info.. Perhaps the ftp server should be upgraded to the latest v5.1..

[UglyBunz]

Originally posted by bigstar
Ahhh, Thanks for the info.. Perhaps the ftp server should be upgraded to the latest v5.1..

Hi,

I have verified that the problem exists sending files to ftpds on 3 different platforms (Windows, FreeBSD & SuSE Linux) and 4 ftpds (serv-u, bsd ftpd, ncftpd and pureftpd). I just happened to pick the serv-u log as the example, but logs to other ftpds are the same, so I don't think upgrading serv-u will make the problem go away!

Thanks,

UglyBunz :)

[UglyBunz]

Well, I managed to transfer one of the problem files to an ftpd on a non-standard port number, but not so far to any server on port 21.

I tried a load of different ftp clients also (there's some scary stuff lurking on some "free" or "shareware" sites!), and they all worked fine. I uninstalled FlashFXP 3 1015 (also removed all of its ancillary files) and installed it again, fresh install, and the problem persisted :(

So now I've reverted to 2.1, build 924, which I suspect is a very old build. It's running in shareware mode at the moment and works perfectly. It would be nice if it was possible to download the most recent 2.1 build from somewhere, as I don't know what the auto-upgrade in 2.1 did with all the installers it downloaded!

I think I'll hold off on 3 until these more obscure issues have been resolved on XP SP2 systems!

Thanks to all who followed up! :)

UglyBunz

[bigstar]

FlashFXP v2.1 build 924 was the last release build of v2.1.

Can you please post the ftp status log from FlashFXP v2.1

[UglyBunz]

Originally posted by bigstar
FlashFXP v2.1 build 924 was the last release build of v2.1.


So that explains it... it's smarter than I thought :)


Can you please post the ftp status log from FlashFXP v2.1

Sure, same ftpd as before (really serv-u 4.1), same MP3 file. I have other files that similarily failed to transfer with 3 that I can now transfer with 2.1. The log has had identifying info replaced.

WinSock 2.0
Connecting to x.x.x
Connected to x.x.x -> IP=y.y.y.y PORT=21
220 BunnySoft FTPd V0.93c - © BunnySoft Inc.
USER w
331 User name okay, need password.
PASS (hidden)
230 User logged in, proceed.
SYST
215 UNIX Type: L8
REST 100
350 Restarting at 100. Send STORE or RETRIEVE.
REST 0
350 Restarting at 0. Send STORE or RETRIEVE.
PWD
257 "/d:" is current directory.
TYPE A
200 Type set to A.
PORT z,z,z,z,4,60
200 PORT Command successful.
LIST
150 Opening ASCII mode data connection for /bin/ls.
226 Transfer complete.
List Complete: 1 KB in 1.34 (0.96 KBps)
CWD upload
250 Directory changed to /d:/upload
PWD
257 "/d:/upload" is current directory.
PORT z,z,z,z,4,62
200 PORT Command successful.
LIST
150 Opening ASCII mode data connection for /bin/ls.
226 Transfer complete.
List Complete: 930 bytes in 0.48 (0.91 KBps)
CWD w
250 Directory changed to /d:/upload/w
PWD
257 "/d:/upload/w" is current directory.
PORT z,z,z,z,4,64
200 PORT Command successful.
LIST
150 Opening ASCII mode data connection for /bin/ls.
226 Transfer complete.
List Complete: 254 bytes in 0.45 (0.25 KBps)
TYPE I
200 Type set to I.
PORT z,z,z,z,4,66
200 PORT Command successful.
STOR Freeform - 03 - Saigon.mp3
150 Opening BINARY mode data connection for Freeform - 03 - Saigon.mp3.
226 Transfer complete.
Transferred: Freeform - 03 - Saigon.mp3 10.39 MB in 08:52 (19.99 KBps)
TYPE A
200 Type set to A.
PORT z,z,z,z,4,68
200 PORT Command successful.
LIST
150 Opening ASCII mode data connection for /bin/ls.
226 Transfer complete.
List Complete: 337 bytes in 0.47 (0.33 KBps)
Transfer queue completed
Transferred 1 file totaling 10.39 MB in 08:53 (19.99 KBps)
QUIT
221 Goodbye!
Logged off: x.x.x

Thanks,

UglyBunz

[OngL]

Bigstar,

After upgrading to XP SP2 (official), I had problem too downloading with proxies i.e. disconnected even I enabled both noops in control channel and during transfer.

It didn't happen before upgrade to SP2. Every 3 minutes reconnections.


Compatibility issue?
[08:04:00] [R] Opening data connection via Proxy
[08:04:00] [R] REST 38619326
[08:04:00] [R] SOCKS: Connecting to x.x.x.1x:1112
[08:04:01] [R] SOCKS: Connected to x.x.x.1x:1112
[08:04:01] [R] 350 Restarting at 38619326. Send STORE or RETRIEVE.
[08:04:02] [R] RETR sosset04.r53
[08:04:02] [R] 150 Opening BINARY mode data connection for sosset04.r53 (50000000 Bytes).
[08:04:24] [R] 200 Command okay.
[08:05:25] [R] Connection lost: Server x

[bigstar]

UglyBunz: I was hoping there might be something slightly different but it looks exactly the same :/

OngL: I don't see how it could be WinXP SP2 but I suppose anything is possible. However I'm not having any problems with WinXP SP2 on either of my two machines. I even tested HTTP and SOCKS5 proxies.

[UglyBunz]

Originally posted by bigstar
[b]UglyBunz: I was hoping there might be something slightly different but it looks exactly the same :/


I was careful to do things in exactly the same way, so that the only variable was the ftp client. Have you tried sending that file I sent you when testing to an ftpd on port 21, from an XP SP2 system with the Windows firewall enabled? If you see the same problem as me it will choke at about 2.15MB.

Unlike OngL, my problem does not seem to be time-related.

UglyBunz

[bigstar]

No actually I did not try to upload the file. Unfortunately I didn't keep it either. Could you please upload the file to me again and I will try that.

Thank you

[UglyBunz]

Originally posted by bigstar
No actually I did not try to upload the file. Unfortunately I didn't keep it either. Could you please upload the file to me again and I will try that.

Thank you

OK, I just sent you that file again to see if you can reproduce the problem.

Thanks,

UglyBunz

[Mouton]

I had a similar problem once.
When transfering files via FTP (I was downloading), some files would always stop at the exact same point. Even retrying would not work.

The problem was my ISP.
They finally fixed something on their side and all the problematic files downloaded fine after that.

I think I remember I couldn't download those files from any server: ftp, http, etc.

They must have been playing with a packet filter of some sort.
Could be something like that...

Have you tried playing with the rollback size on resume..? Like tried setting it to 0 and >0 see if it makes any diff. on resumes...

[bigstar]

I took some time to perform several tests on two different computers.

I was not able to reproduce the problem :(

I have never seen anything like this and I am unable to explain why v2.1 and other ftp clients work but v3.0 doesn't. It's quite a mystery to me.

I can't think of anything else to try or to suggest.

[UglyBunz]

Hi Mouton,

Yeah, the type of problem you describe can be caused by MTU settings on routers, but in this case I'm confident that the network is OK at both ends, as the problem only occurs with FFXP3.

Your suggestion about the rollback warrants a test though, so I'll try that later when I get home. FFXP3 seems to be fine on my Win2k SP4 here at work!

Thanks for your input :p

UglyBunz

[UglyBunz]

I tried a few more experiments with FFXP3 & XP SP2:

- I disabled the Windows firewall and installed Tiny Professional Firewall 6.110 in shareware mode. The problem was still present, so it wasn't the XP firewall causing the problem.

- I verified that my NIC drivers were the latest versions. They were, and were certified by MS.

- I installed the FFXP 3 beta build 1022, but the problem was still there.

- I decided to tweak lots of settings that I don't normally mess with (on the basis that if it ain't broke...). I found one that fixes the problem: reducing the Upload Packet Size (in advanced options) to 2048 made the problem vanish. I then tested it again at 3072 and the uploads worked fine also. Resetting it to the default 4096 broke the upload of the problem files I had identified.

- Weirder still : I then tried setting it to 5120, on the assumption that it was some type of MTU problem and that this would make the problem worse, but no, the uploads worked fine also!

- So the problem is only apparent at the default upload packet size of 4096 bytes! And then it only happens on maybe 1 in 5 MP3 files. I haven't experienced the problem with any other file types, though I do tend to transfer more MP3s than anything else.

Just for badness, I even tried uploading problem files with the packet size set to the obscenely large 16384 setting, and it worked also!

Finally I reverted back to build 1015, and checked that its behaviour was the same as 1022 - it is.

So if anyone else is having weird time-out type problems with uploads from FFXP3 on XP SP2, then try tweaking the upload packet size and see if that helps. I've left mine set to 5012 and so far have experienced no problems.

UglyBunz :cool:

[Mouton]

If you got some time on your hand, a packet capture (using Ethereal or similar) would probably help to find the cause of this.

Thanks for your time!

[UglyBunz]

Originally posted by Mouton
If you got some time on your hand, a packet capture (using Ethereal or similar) would probably help to find the cause of this.

Thanks for your time!

I'm getting ready to move house next weekend, so now is not a good time for me to begin messing about with packet loggers. But I know how to switch the problem on and off now, so maybe in a week or so I can have a better look!

Thanks to all,

UglyBunz ;)

[bigstar]

I'm glad you solved your problem..

However I am rather surprised that changing the Upload Packet Size solved the problem.

The Upload Packet Size changes the size of the internal memory buffer used when performing a Winsock.Send().. The amount of data that is sent per Winsock.Send()

If the Upload Packet Size is larger than 8kb then the send buffer size is increased using setsockopt (SO_SNDBUF)

I will continue to look into this matter.

[Thog]

Sounds like the same problem I was having... And still am having ;)... I keep it at 1024 now and it works on most files but no matter what the packet size is, there will be an error on some files... I just change the packetsize and upload all the failed files...

[WinEggDrop]

Originally posted by bigstar
I took some time to perform several tests on two different computers.

I was not able to reproduce the problem :(

I have never seen anything like this and I am unable to explain why v2.1 and other ftp clients work but v3.0 doesn't. It's quite a mystery to me.

I can't think of anything else to try or to suggest.

V2.1 is quiet different from V3.0.
V3.0 uses asynchronous function with non-blocking socket(I believe) and V2.1 used synchronous with blocking socket.

[WinEggDrop]

By the way,what BunnySoft FTPd V0.93c is? I have never seen this ftpd or even heard it?

[MxxCon]

Originally posted by WinEggDrop
By the way,what BunnySoft FTPd V0.93c is? I have never seen this ftpd or even heard it? 5th and 6th posts in this thread:rolleyes: