slowdown tcp transfer under Windows 10 64bits

[Alain1533 2]

Hello,

 

I use your components (wsocket) in industrial applications, binary TPC transfer between applications and Siemens PLCs. (s1500s).

The exchanged data blocks have a size of 4096 Bytes ,.

 

The transfer time under Windows 7 32bits is 4Milliseconds which is perfect for

our applications.

 

On our new machines, we need to install Windows 10 64bits.

 

The transfer programs still work perfectly, however the transfer time has dropped to 15 milliseconds, which is absolutely not suitable

for our applications.

 

I tried several different configurations, compilation in 32bits, 64bits, adjustment of the buffer sizes, without much change.

 

WIndows 10, Window 7 32Bits: Transfer = 4 Milliseconds

Window 10 64Bits: Transfer = 13 to 15 Milliseondes

 

It therefore seems that as soon as the OS is in 64bits the communication is slower.

 

We absolutely have to install Windows 10 64bits versions, this is a request from our customers, so I absolutely have to find a solution.

 

IDE Rad Studio Achitect

RIO 10.3 compiler

Overbyte Release: V8.65

Delphi language.

 

I would like to thank your team, your components are excellent and have always given us complete satisfaction.

 

Alain Verdier

Edited January 23, 2021 by Alain1533

[FPiette 383]

Did you check the speed when compiling your application as 32 and 64 bit?

 

I would also check if connectivity is OK. Maybe the new machine has a bad Ethernet cable or switch or network card and experience a lot of retries.

 

Are "jumbo packets" enabled on the network card?

 

Is there any "security" product checking connections (Firewall, anti-virus,...)?

 

Is the new computer hardware superior to the old one?

 

[Alain1533 2]

Hello François,

 

Thank you for your reply.

 

Yes the connection is Ok

 

Jumbo, I tried with and without, no change

 

I did some tests with and without anti-virus no change

 

Yes the new computer hardware is superiror to the old one

 

merci

[Alain1533 2]

compilation 32 or 64 bits no change

[FPiette 383]

We'll try to determine if the slow down comes from the computer or from ICS.

 

If you still have the old computer, is it possible to test a simple file copy using Windows explorer from the old computer to a third computer and from the new computer and the same computer. Same file in both cases. Use a big file so that the duration is at least 30 seconds. The goal is to compare the throughput of Windows networking with a common reference (the 3rd computer). Measure several copies, usually the first is slower than the next because the file is not yet in disk cache.

 

Other test: setup a web server or a ftp server on the 3rd computer (You may use ICS demo application). Then measure the throughput to copy the same file as above using ICS at both ends. Again the goal is to compare is HTTP of FTP has changed speed between the old and new computer, everything else staying the same.

 

Using the command line, run "netstat -e" to show Ethernet statistics. You'll see the error and reject count. They should be both zero.

 

By the way, are you using IPv4 or IPv6?

 

[Guest]

My two cents :

 

Many things can cause that effect from IPv6 (with/out OS, router specific intervene...) conflict to software installed or OS service ( based on Windows edition like Home vs Pro..) to Network adapter and its driver....

In short the cause will take long time to track, while with one WireShark log, you (also we) can narrow the cause greatly, you will need to capture two logs preferably (if possible) of the same process of sending like the first few packets on both OS's.

 

If you are not familiar with WireShark, then its time to learn the most basic things about it, run it and filter the log based on target IP.

 

This will show where the traffic is taking time, and will help to see if the delay coming form the adapter or the application ( like what if your Delphi code is somehow missing something on x64 and casing the delay, this will eliminate the ICS involvement and might show some retransmission due to a bad checksum..)

[Alain1533 2]

all our machines are in IPV4

[Alain1533 2]

I'm gonna do these tests and tell what's going on

[Angus Robertson 574]

I don't believe ICS has any OS checks between Windows 7 and 10, only back to XP, nor does it check for 32-bit or 64-bit versions of Windows, and no conditional Win64 code for TWSocket. But the Windows winsock code is 12 years younger so maybe different. 

 

Are you sending packets using simple TWSocet functions, using IP addresses not domain names?

 

Angus

 

[Alain1533 2]

Hello Angus,

Yes I am using simple TWSocket, the data is binary data.

/// <summary>
/// Buffer de traitement des DB
/// </summary>
TS7Buffer = array[0..MAXWORD] of Byte;

...

/// <summary>

// Send Mapstream to Siemens DB100

/// </summary>

FS7BufferOut : TS7Buffer;
ActiveCliSocket.Send(@FS7BufferOut,iSize);

....

The reception buffer is large, the data exchanged between the S1500S PLC and the PC is only 4096 Bytes

I tried to reduce the size of the buffers, but it didn't change anything

No domain name

IP:

  S1500S 192.168.7.21

  PC         192.168.1.100

I will try to compile with the EMbarcadero beta test version (10.4) I received the new version yesterday

Thank you very much everyone for your help

Alain   

 

[Alain1533 2]

No domain name

IP:

  S1500S 192.168.7.21

  PC         192.168.7.100

[Alain1533 2]

I have just redone the following tests:

Server:
Simple S1500S simulator, Receive / transmit 4096Bytes binary frame
IP: 192.16.7.100

Client :
Basic program used on our machines
IP: 192.16.7.150

Loop Cycle = 69 Milliseconde

Wireshark:
There is a huge amount of error: Incorrect TCP checksum

I added an image of wireshark

 

Copy Paste of a '19 774 kb' file with windows explorer is instantaneous

 

Edited January 23, 2021 by Alain1533

[Guest]

@Alain1533 The image is unreadable, please repost another one with smaller size.

Two screenshots from the fast and the slow would be better to compare. the two tabs of the content is not important here.

[Alain1533 2]

I don't have screenshot of the fast version, i do it Monday at the office

 

 

Edited January 23, 2021 by Alain1533

[Guest]

7 minutes ago, Alain1533 said:

I don't have screenshot of the fast version, i do it Monday at the office

OK.

 

Reason number one for failed checksum is bad ethernet cable or buggy router or very weak wireless connection, start there to figure it out, the application and the OS are most likely are not responsible for this delay, in other words your network is malfunctioning.

[FPiette 383]

50 minutes ago, Alain1533 said:

There is a huge amount of error: Incorrect TCP checksum

One of the involved network card is failing or the Ethernet cable is defective, or the connections, or the junction in the patch panel (if any) or the switch (if any). If it is possible, change the way the PC and PLC are connected (direct cable is best, maybe cross-cable if old network card not able to seen how it is connected).

 

By the way, are error displayed with command 'netstat -e" I suggested to use?

Edited January 23, 2021 by FPiette

[Alain1533 2]

I had a probleme with my cable, i switched to WIFI and there i have no more checksum error

in WIFI the cycle time is 15 ms, i try again on Monday at the office with direct connection.

 

netstat -e -> Errors = 0

 

Thank you for your help

 

Alain

[Fr0sT.Brutal 900]

Btw, to check the local network speed I successfully used iperf tool. There's plenty of such apps but that one satisfied all my needs

[Der schöne Günther 316]

Maybe there is something wrong with my eyes or the image is too compressed, but...

 

I think "TCP Checksum incorrect" does not have to be an error. If it can, Windows will delegate the TCP checksum calculation to the hardware itself, therefore the checksum is just visible as zeroes to Wireshark. Is the checksum zeroes or something else?

Edited January 25, 2021 by Der schöne Günther

[Alain1533 2]

hello,

"TCP CheckSum incorrect'

I solved the problem, it was a faulty cable

 

[Guest]

28 minutes ago, Alain1533 said:

"TCP CheckSum incorrect'

I solved the problem, it was a faulty cable

What about the speed and the delay ? fixed or not ?

[Alain1533 2]

Hello,

I performed the tests under the following conditions:
    Same PC
    Same PLC (S1500S)
    Direct cable connection
    Same application:
        Delphi 10.3
        overbytes Release: V8.65
        32bits compilation
    PLC IP = 192.168.7.3, Port = 2000
    PC  IP  = 192.168.7.11

Test1: With windows 10 32bits
Test2: With windows 10 64bits

For test1 the cycle time = 3 Milliseconds
For test2 the cycle time = 15 Milliseconds

I made images of the WireShark for the two tests, I put in red the notable differences between the sequences:
   [ACK] and [PSH, ACK] which marks the beginning of the exchange.
With windows 10 32 bits delta time is between 1 and 3 Ms
With windows 10 64 bits delta time is between 12 and 13 Ms

I think the slowdown in communication is due to this, however I'm not sure what to do to solve this problem.
I have tried different configuration without noticeable change, for the moment we will therefore stay under windows10 in 32bits, but I would like to know where it can come from.

Thanks for your help.
Alain

 

[Der schöne Günther 316]

Do you have a minimal, reproducible example?

Also, you are still running a 32 bit executable on a 64 bit OS. What if you run the example as a 64 bit application?

[Guest]

@Alain1533 We are getting somewhere now.

 

Now without the failure/malfunctioning in the hardware we can look deeper into packets and their order and timing.

 

You already marked the delay in the small red rectangle and the first one is an ACK, it is a simple acknowledgment from the PC to the PLC of receiving, also means the network adapter received the data and forwarded it to the application !

next packet is new series or may be a response, while the content is irrelevant for us now, its delay is important.

 

So what you have to do is to log the ICS socket operation within your application with high precision timer, you must log them to a file, and each operation should have a time with 1ms precision at least.

After that you can find the code responsible for this delay, which i think it is not coming form ICS but from very inefficient code for 64 bit, but to pinpoint this you need to log these parts of code and compare them.

 

 

 

[Alain1533 2]

Unfortunately it is not easy to reproduce the problem without having a PLC,
I will try to give you the simulator that I use with a minimum of code.

Yes I tried with a 64-bit compilation without noticeable change