Indy FTP Server Error Question (Could not bind socket)

[snorkel]

Hi,
I have a windows service I wrote with the Indy 10 FTP server (which works great by the way)
Anyways, I recently had the need for the server to listen on two ports (21 and a non standard port of lets say 30021)
So I added the bindings and it indeed is listening on both ports now.

the problem seems to be that after I did this once in awhile I get errors like this:

FTP server error:(10.0.8.1:60963) Could not bind socket. Address and port are already in use.

I did set both bindings to listen on 0.0.0.0 and the server is set to reuse sockets  (property ReuseSocket set to rsTrue)

Anyone have any ideas?  should I use the actual IP of the server instead of 0.0.0.0 for the two bindings?

[Remy Lebeau]

the problem seems to be that after I did this once in awhile I get errors like this:

FTP server error:(10.0.8.1:60963) Could not bind socket. Address and port are already in use.

My guess would be that a client has connected to the server and is performing a passive mode file transfer and the server is trying to bind a new transfer socket to a listening port that is not available.  What do you have the server's PASVBoundPortMin, PASVBoundPortMax, and DefaultDataPort properties set to?  It is important that for Passive transfers, the server has a pool of multiple ports available so multiple transfers can run simultaneously on separate ports.

the server is set to reuse sockets  (property ReuseSocket set to rsTrue)

That property is only used for the main command sockets.  Secondary transfer sockets do not use that property.

should I use the actual IP of the server instead of 0.0.0.0 for the two bindings?

That is not necessary.

[snorkel]

Hi Remy,

PASVBoundPortMin = 60000
PASVBoundPortMax = 65000

DefaultDataport = 20

Do you think the passive port range is maybe too large?  Is it possible some other script or whatever is once in awhile grabbing from that range?

got another one today:  FTP server error:(10.0.8.1:40913) Could not bind socket. Address and port are already in use.

[snorkel]

I did notice that in the FTPSecurityOptions there is a option called:
NoReservedRangePort and it's set to true, would that have anything to do with this?

[Remy Lebeau]

What you describe does not make sense.  60963 is in your PASV port range, but TIdFTPServer would not raise a bind error on a Passive-mode transfer unless the full range of PASV ports is exhausted (and even then, the error message would be different).  But where is port 40913 coming from?  For that matter, where is your error message actually coming from and what does the call stack look like when the error is raised?  Indy does not add IP/Port information to that error message, so were are you getting it from?

The specific error message you have shown is only raised under one condition - failure to bind a socket to a specific port, not to a range of ports.  And the only times that TIdFTPServer binds a socket to a specific port are:

• each listening Binding.
• all Active-mode transfers, which are bound to DefaultDataPort.  These are outbound connections, so multiple transfers can be bound to the same port at the same time as long they are connecting to different IP/Port targets.
• Passive-mode transfers, which are bound to DefaultDataPort when PASVBoundPortMin/Max are 0.  These are inbound connections, so multiple transfers can be bound to the same port at the same time if different IP/Port sources are connecting.

Whether or not the PASV range of ports is too large depends on how many simultaneous transfers you want to support, and how many other socket apps are running along your server.  You are using a range of 5001 ports.  When a Passive-mode transfer is requested, TIdFTPServer is going to try opening each port in the range until it finds one that is available.

As for NoReservedRangePort, if it is enabled then TIdFTPServer will reject an Active-mode transfer that requests a connection to a port between 1-1024, inclusive.  On some platforms, particular *Nix systems, those ports require admin rights to use.

[snorkel]

Remy,
I should have mentioned that error msg is for my logging purposes, so port 40913 is the peer port in this case.
i'm gettting the error message from the OnException event and in that event I do this for the log:

 if assigned(AContext) then
          begin
               ip:=acontext.Binding.PeerIP;
               port:=acontext.Binding.PeerPort;
          end;   

Then I push the error message into a queue:

push_log_msg(format('FTP server error:(%s:%d) %s',[ip,port,AException.Message]),logonid,Error);

I looked back in the log and that port already in use error was happening before I added the server bindings for port 10021 and 21, was just coincidence that I noticed it because more users are hitting the server now.

The connections coming in are coming from a mainframe and I don't know if the data connections are passive or not.

I can't reproduce it when I run the server in the IDE, so no call stack.

[Remy Lebeau]

i'm gettting the error message from the OnException event

That is odd, because the OnException event is fired only for uncaught exceptions that are fatal to clients (ie, the exception will disconnect the client).  The only socket bindings that a client should be performing is when opening transfer sockets, and any exceptions raised during that process are caught (so errors can be sent back to the client), they shouldn't be firing the OnException event.  I would really need to see the call stack of the exceptions to know what is going on.

[snorkel]

I thought the onexception event was the global exception handler for the server?

Anyways, what I think is happening is the mainframe is sending each file in a separate connection, i.e. they logon, upload a file and then disconnect and do that for each file they have to send.  Why they can't just logon once and do multiple files under a single logon is beyond me :-)
But because of the way they are sending them and the way the Indy FTP server starts at the max passive port for each upload the separate uploads going concurrently must be stepping on each other occasionally, it must be getting a port that is in a timed wait state or something and then when it tries to use it raises that error.
I don't know for sure but that seems like what is happening.

I wonder if it would be better to grab a random starting point from the passive port range, or maybe from the max to the mid or something like that instead of doing a down to loop starting at max for each upload and connection.

[snorkel]

Remy,
I got this back from the mainframe folks:

Code: Pascal  [Select][+][-]

1. 150 File status okay; about to open data connection.                            
2. EZA2589E Connection to server interrupted or timed out. Waiting for data connect
3. EZA1636I *** I can't open a data-transfer connection:                          
4. 426 Data connection closed abnormally.
5. EZA1735I Std Return Code = 27426, Error Code = 00009  

     

This is the error they saw when the binding error occurred on the server.                                 
         

[Cyrax]

Sounds like problem with FTP passive ports and their ranges. They should check their firewall and allow FTP passive ports.

[Remy Lebeau]

I thought the onexception event was the global exception handler for the server?

It is a server event for uncaught exceptions that terminate client threads on a per-client basis.  The event provides the TIdContext belonging to the client thread that is raising the reported exception.

But because of the way they are sending them and the way the Indy FTP server starts at the max passive port for each upload the separate uploads going concurrently must be stepping on each other occasionally, it must be getting a port that is in a timed wait state or something and then when it tries to use it raises that error.

I still don't see how that could cause the error you are seeing.  Even if the ENTIRE passive port range were in TIME_WAIT at the time of a new transfer, a bind error during a new transfer should be caught by the invoking command handler (PASV, etc) and not bubble up to the thread that is managing the client connection. 

In case the ENTIRE passive port rage is in TAIME_WAIT, I wonder if using the TIdFTPServer.OnDataPortBeforeBind event to enable the ReuseSocket property (ASender.DataChannel.FDataChannel.Socket.ReuseSocket := rsTrue) on each new data transfer socket would help.

I don't know for sure but that seems like what is happening.

I can't answer that without seeing the call stack of the exception on the server side.  Can't you please capture that?

I wonder if it would be better to grab a random starting point from the passive port range, or maybe from the max to the mid or something like that instead of doing a down to loop starting at max for each upload and connection.

Adding some element of randomness would probably help.  But it still needs to loop through the whole range in the end, and it is more overhead to loop through the entire range using random indexes while avoiding duplicates during the looping.  I'll have to figure out an algorithm to do it.

I got this back from the mainframe folks:

Code: Pascal  [Select][+][-]

1. 150 File status okay; about to open data connection.                            
2. EZA2589E Connection to server interrupted or timed out. Waiting for data connect
3. EZA1636I *** I can't open a data-transfer connection:                          
4. 426 Data connection closed abnormally.
5. EZA1735I Std Return Code = 27426, Error Code = 00009  

     

This is the error they saw when the binding error occurred on the server.                                         

That is not very helpful without seeing the exact FTP commands and replies leading up to the error.  TIdFTPServer uses 426 for any unexpected error that happens during a data transfer, so there is still a number of things that could be failing to cause that error.

[snorkel]

"In case the ENTIRE passive port rage is in TAIME_WAIT, I wonder if using the TIdFTPServer.OnDataPortBeforeBind event to enable the ReuseSocket property (ASender.DataChannel.FDataChannel.Socket.ReuseSocket := rsTrue) on each new data transfer socket would help."

That sounds like it might be worth trying.

FDatachannel is protected, so currently not accessible, I can add a property to test this out.

EDIT:  nope that made it much worse.  Tons of errors during file uploads.

[Remy Lebeau]

FDatachannel is protected, so currently not accessible, I can add a property to test this out.

Yes, I know it is protected.  A property is not needed, an accessor class will also work.

[snorkel]

Remy,
This is still any issue and I tracked it down to procedure TIdSocketHandle.Bind in IdSocketHandle.

I noticed there is a comment in there about socket in use errors and it has a IDEF for Linux64 and I am compiling for 64bit just not linux.

would it hurt to enable the REUSEPORT?

[Remy Lebeau]

would it hurt to enable the REUSEPORT?

Windows does not support SO_REUSEPORT.  If you enable that code on Windows, TIdSocketHandle.SetSockOpt() will end up raising a socket error exception.