I've been mailing a bit on the python-users mailing list about STP this week, and there were some discussions there that were unfruitful - but I got some useful tips in the process:
https://mail.python.org/pipermail/python-list/2022-July/9070...
This TCP proxy needs to scale and perform reasonably well, so the last thing I worked on was making sure that downloading large files via the proxy works reasonably well, in these commits:
https://github.com/morphex/stp/commit/3328a45d5314040abd37fa...
https://github.com/morphex/stp/commit/5e477bf5756e2763db799c...
In was interesting, and relieving to see that I could speed up the download process 3-4x by multiplying the "transfer buffer" by 8. A local download of the test.tar file was around 1.3 GB/s via lighttpd, and 122 MB/s via STP. After the buffer change, the speed increased to 449 MB/s. Which is more than what a gigabit connection to the internet can handle.
So, so far so good, and good enough.
The previous commit:
https://github.com/morphex/stp/commit/5e477bf5756e2763db799c...
I got a tip for, and reduced the thread stack size to the minimum, which is from what I gather, necessary to ensure that it will run reliably on different platforms - as Linux has a rather big thread size and that could break on others if you're reliant on it.
I also implemented a "fall through", so that the thread sending data back and forth doesn't stop and wait for .N number of seconds, if data has just been sent back and forth.
And in the commit before that, I adjusted the sleep time for threads "polling" on whether or not they have become active:
https://github.com/morphex/stp/commit/9910ca8c80e9d150222b68...
which also increased the speed at which STP could deal with the connection queue.
There has been some questions on the Python list as to exactly what I'm building and why, but I think a fair description would be that it is a simple to understand, and run, TCP proxy which does some useful things out of the box.
Coming from the web development and hosting world, I'm tempted to add some HTTP protocol features as well, but that's something that belongs in another project, maybe using STP as the frame which it plugs into.
https://mail.python.org/pipermail/python-list/2022-July/9070...
This TCP proxy needs to scale and perform reasonably well, so the last thing I worked on was making sure that downloading large files via the proxy works reasonably well, in these commits:
https://github.com/morphex/stp/commit/3328a45d5314040abd37fa...
https://github.com/morphex/stp/commit/5e477bf5756e2763db799c...
In was interesting, and relieving to see that I could speed up the download process 3-4x by multiplying the "transfer buffer" by 8. A local download of the test.tar file was around 1.3 GB/s via lighttpd, and 122 MB/s via STP. After the buffer change, the speed increased to 449 MB/s. Which is more than what a gigabit connection to the internet can handle.
So, so far so good, and good enough.
The previous commit:
https://github.com/morphex/stp/commit/5e477bf5756e2763db799c...
I got a tip for, and reduced the thread stack size to the minimum, which is from what I gather, necessary to ensure that it will run reliably on different platforms - as Linux has a rather big thread size and that could break on others if you're reliant on it.
I also implemented a "fall through", so that the thread sending data back and forth doesn't stop and wait for .N number of seconds, if data has just been sent back and forth.
And in the commit before that, I adjusted the sleep time for threads "polling" on whether or not they have become active:
https://github.com/morphex/stp/commit/9910ca8c80e9d150222b68...
which also increased the speed at which STP could deal with the connection queue.
There has been some questions on the Python list as to exactly what I'm building and why, but I think a fair description would be that it is a simple to understand, and run, TCP proxy which does some useful things out of the box.
Coming from the web development and hosting world, I'm tempted to add some HTTP protocol features as well, but that's something that belongs in another project, maybe using STP as the frame which it plugs into.