[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"project-6601":3},{"id":4,"name":5,"fullName":6,"owner":7,"repo":5,"description":8,"homepage":9,"htmlUrl":10,"language":11,"languages":10,"totalLinesOfCode":10,"stars":12,"forks":13,"watchers":14,"openIssues":15,"contributorsCount":15,"subscribersCount":15,"size":15,"stars1d":15,"stars7d":15,"stars30d":16,"stars90d":15,"forks30d":15,"starsTrendScore":15,"compositeScore":17,"rankGlobal":10,"rankLanguage":10,"license":18,"archived":19,"fork":19,"defaultBranch":20,"hasWiki":21,"hasPages":21,"topics":22,"createdAt":10,"pushedAt":10,"updatedAt":43,"readmeContent":44,"aiSummary":45,"trendingCount":15,"starSnapshotCount":15,"syncStatus":16,"lastSyncTime":46,"discoverSource":47},6601,"tcpcopy","session-replay-tools\u002Ftcpcopy","session-replay-tools","An online request replication and TCP stream replay tool, ideal for real testing, performance testing, stability testing, stress testing, load testing, smoke testing, and more.","http:\u002F\u002Fsession-replay-tools.github.io\u002Ftcpcopy\u002F",null,"C",4676,1025,364,0,2,61.23,"Other",false,"master",true,[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"apache","load-testing","mysql","nginx","performance","performance-improvements","performance-testing","real-testing","redis","replay","replay-http-traffic","replay-mysql-traffic","simulation","smoke-testing","stability","stress-testing","tcp","tcp-server","testing","testing-tools","2026-06-12 04:00:29","# [TCPCopy](https:\u002F\u002Fgithub.com\u002Fsession-replay-tools\u002Ftcpcopy) - A TCP Stream Replay Tool\r\n\r\nTCPCopy is a TCP stream replay tool for realistic testing of Internet server applications. \r\n\r\n## Getting to Know TCPCopy\r\n\r\n[An Overview of TCPCopy for Beginners](doc\u002FBeginners.md)\r\n\r\n[A General Overview of TCPCopy Architecture](doc\u002FArchitecture.md)\r\n\r\n[TCPCopy Testing Use Cases](doc\u002FUsage.md)\r\n\r\n[TCPCopy Pre-Warming Examples](doc\u002FPre-Warming.md)\r\n\r\n## Description\r\n\r\nAlthough real live traffic is crucial for testing Internet server applications, accurately simulating it is challenging due to the complexity of online environments. To enable more realistic testing, TCPCopy was developed as a live flow reproduction tool that generates test workloads closely resembling production workloads. TCPCopy is widely used by companies in China.\r\n\r\nTCPCopy minimally impacts the production system, consuming only additional CPU, memory, and bandwidth. The reproduced workload mirrors the production environment in terms of request diversity, network latency, and resource usage.\r\n\r\n## Use Cases\r\n\r\n* Distributed Stress Testing\r\n  - Use TCPCopy to replicate real-world traffic for stress testing your server software, uncovering bugs that only appear under high-stress conditions.\r\n* Live Testing\r\n  - Validate the stability of new systems and identify bugs that only manifest in real-world scenarios\r\n* Regression testing\r\n  - Ensure that recent changes have not introduced new issues.\r\n* Performance comparison\r\n  - Compare system performance across different versions or configurations.\r\n\r\n## Architecture\r\n\r\n![tcpcopy](images\u002Ftcpcopy.png)\r\n\r\nFigure 1. Overview of the TCPCopy Architecture.\r\n\r\nAs shown in Figure 1, TCPCopy is composed of two components: `tcpcopy` and `intercept`. The `tcpcopy` component runs on the online server, capturing live requests, while `intercept` operates on the assistant server, performing tasks such as passing response information to `tcpcopy`. The test application itself runs on the target server.\r\n\r\nBy default, `tcpcopy` uses raw sockets to capture packets at the network layer (depicted by the orange arrows in the figure). It handles processes such as TCP interaction simulation, network latency control, and upper-layer interaction simulation. It then sends packets to the target server using raw sockets for output (shown by the light red arrows in the figure).\r\n\r\nThe only required task on the target server is configuring route rules to direct response packets (shown by light green arrows in the figure) to the assistant server.\r\n\r\nThe `intercept` component's role is to forward the response header (by default) to `tcpcopy`. It captures the response packets, extracts the response header information, and sends this information to `tcpcopy` via a dedicated channel (represented by light blue arrows in the figure). Upon receiving the response header, `tcpcopy` uses the information to modify the attributes of online packets and proceeds to send subsequent packets.\r\n\r\nIt is important to note that responses from the target server are routed to the assistant server, which functions as a black hole.\r\n\r\n## Quick Start\r\n\r\nFor `intercept`, you have two options:\r\n\r\n* [Download the latest intercept release](https:\u002F\u002Fgithub.com\u002Fsession-replay-tools\u002Fintercept\u002Freleases).\r\n* Clone the repository:\r\n  `git clone git:\u002F\u002Fgithub.com\u002Fsession-replay-tools\u002Fintercept.git`.\r\n\r\nFor `tcpcopy`, you also have two options\r\n\r\n* [Download the latest tcpcopy release](https:\u002F\u002Fgithub.com\u002Fsession-replay-tools\u002Ftcpcopy\u002Freleases).\r\n* Clone the repository:\r\n  `git clone git:\u002F\u002Fgithub.com\u002Fsession-replay-tools\u002Ftcpcopy.git`.\r\n\r\n## Installing intercept on the Assistant Server\r\n\r\n1. Navigate to the `intercept` directory:\u003Cbr>\r\n   `cd intercept`\r\n2. Run the configuration script:\u003Cbr>\r\n   `.\u002Fconfigure` \u003Cbr>\r\n   Optionally, specify any necessary configuration options.\r\n3. Compile the source code:\u003Cbr>\r\n   `make`\r\n4. Install the `intercept` tool:\u003Cbr>\r\n   `make install`\r\n\r\n### Configure Options for `intercept`\r\n\r\n- `--single`  \r\n  Run `intercept` in non-distributed mode.\r\n\r\n- `--with-pfring=PATH`  \r\n  Specify the path to the PF_RING library sources.\r\n\r\n- `--with-debug`  \r\n  Compile `intercept` with debug support, with logs saved to a file.\r\n\r\n## Installing `tcpcopy` on the Online Server\r\n\r\n1. Navigate to the `tcpcopy` directory: \u003Cbr>\r\n   `cd tcpcopy`\r\n2. Run the configuration script: \u003Cbr>\r\n   `.\u002Fconfigure` \u003Cbr>\r\n    Include any necessary configuration options as needed.\r\n3. Compile the source code: \u003Cbr>\r\n   `make`\r\n4. Install the `tcpcopy` tool: \u003Cbr>\r\n   `make install`\r\n\r\n### Configure Options for `tcpcopy`\r\n\r\n- `--offline`  \r\n  Replay TCP streams from a pcap file.\r\n\r\n- `--pcap-capture`  \r\n  Capture packets at the data link layer.\r\n\r\n- `--pcap-send`  \r\n  Send packets at the data link layer instead of the IP layer.\r\n\r\n- `--with-pfring=PATH`  \r\n  Specify the path to the PF_RING library sources.\r\n\r\n- `--set-protocol-module=PATH`  \r\n  Set `tcpcopy` to work with an external protocol module.\r\n\r\n- `--single`  \r\n  If both `intercept` and `tcpcopy` are configured with the `--single` option, only one `tcpcopy` instance will work with `intercept`, leading to better performance.\r\n\r\n- `--with-tcmalloc`  \r\n  Use tcmalloc instead of malloc.\r\n\r\n- `--with-debug`  \r\n  Compile `tcpcopy` with debug support, with logs saved to a file.\r\n\r\n## Running TCPCopy\r\n\r\nAssume that both `tcpcopy` and `intercept` are configured using `.\u002Fconfigure`.\r\n\r\n1. **On the Target Server Running Server Applications:**\r\n   \r\n   Configure the route rules to direct response packets to the assistant server. For example, if `61.135.233.161` is the IP address of the assistant server, use the following route command to direct all responses from clients in the `62.135.200.x` range to the assistant server:\r\n   \r\n   `route add -net 62.135.200.0 netmask 255.255.255.0 gw 61.135.233.161`\r\n\r\n2. **On the Assistant Server Running `intercept` (Root Privilege or CAP_NET_RAW Capability Required):**\r\n   \r\n    `.\u002Fintercept -F \u003Cfilter> -i \u003Cdevice>`\r\n   \r\n   Note that the filter format is the same as the pcap filter. For example:\r\n   \r\n   `.\u002Fintercept -i eth0 -F 'tcp and src port 8080' -d`\r\n   \r\n   In this example, `intercept` will capture response packets from a TCP-based application listening on port 8080, using the eth0 network device.\r\n   \r\n   Please note that `ip_forward` is not enabled on the assistant server.\r\n\r\n3. **On the Online Source Server (Root Privilege or CAP_NET_RAW Capability Required):**\r\n   \r\n   `.\u002Ftcpcopy -x localServerPort-targetServerIP:targetServerPort -s \u003Cintercept server> [-c \u003Cip range>]`\r\n   \r\n   For example (assuming 61.135.233.160 is the IP address of the target server):\r\n   \r\n   `.\u002Ftcpcopy -x 80-61.135.233.160:8080 -s 61.135.233.161 -c 62.135.200.x`\r\n   \r\n   In this example, `tcpcopy` captures packets on port 80 from the current server, changes the client IP address to one from the 62.135.200.x range, and sends these packets to port 8080 on the target server (61.135.233.160). It also connects to 61.135.233.161 to request `intercept` to forward response packets. While the `-c parameter` is optional, it is used here to simplify route rules.\r\n\r\n## Note\r\n\r\n1. Platform: Tested only on Linux (kernel 2.6 or above).\r\n2. Packet Loss: TCPCopy may lose packets, which could result in lost requests.\r\n3. Permissions: Requires root privilege or the `CAP_NET_RAW` capability (e.g., setcap CAP_NET_RAW=ep tcpcopy).\r\n4. Connection Type: Currently supports only client-initiated connections.\r\n5. SSL\u002FTLS: Does not support replay for applications using SSL\u002FTLS.\r\n6. Due to the additional layer of forwarding in tcpcopy, the throughput of a single application connection cannot be too high; otherwise, it won't match the native connection throughput, especially in performance tests like sysbench or ab.\r\n7. If the volume of replicated requests is too large, tcpcopy may become unstable, with the single thread overwhelmed by packet capture, significantly reducing replication effectiveness. In such cases, other auxiliary methods can be used, such as leveraging switch mirroring with a divide-and-conquer packet capture strategy or using offline replay.\r\n8. MySQL Session Replay: For details, visit [mysql-replay-module](https:\u002F\u002Fgithub.com\u002Fsession-replay-tools\u002Fmysql-replay-module) or [mysql-sgt-replay-module](https:\u002F\u002Fgithub.com\u002Fsession-replay-tools\u002Fmysql-sgt-replay-module).\r\n9. The `.\u002Fconfigure --with-resp-payload` option for `intercept` cannot be used together with the `.\u002Fconfigure` option for `tcpcopy`.\r\n10. IP Forwarding: Ensure `ip_forward` is not enabled on the assistant server.\r\n11. Help: For more information, run `.\u002Ftcpcopy -h` or `.\u002Fintercept -h`.\r\n\r\n## Influential Factors\r\n\r\nSeveral factors can impact TCPCopy, as detailed in the following sections.\r\n\r\n### 1. Capture Interface\r\n\r\nBy default, `tcpcopy` uses a raw socket input interface to capture packets at the network layer on the online server. Under high load, the system kernel may drop some packets.\r\nIf configured with `--pcap-capture`, `tcpcopy` captures packets at the data link layer and can filter packets in the kernel. Using `PF_RING` with pcap capturing can reduce packet loss.\r\nFor optimal capture, consider mirroring ingress packets via a switch and distributing the traffic across multiple machines with a load balancer.\r\n\r\n### 2. Sending Interface\r\n\r\n`tcpcopy` defaults to using a raw socket output interface to send packets at the network layer to the target server. To avoid `ip_conntrack` issues or improve performance, use `--pcap-send` to send packets at the data link layer instead.\r\n\r\n### 3. On the Way to the Target Server\r\n\r\nPackets sent by `tcpcopy` may face challenges before reaching the target server. If the source IP address is the end-user's IP (by default), security devices may drop the packet as invalid or forged. To test this, use `tcpdump` on the target server. If packets are successfully sent within the same network segment but not across segments, packets may be dropped midway.\r\n\r\nTo address this, deploy `tcpcopy`, target applications, and `intercept` within the same network segment. Alternatively, use a proxy in the same segment to forward packets to the target server in another segment.\r\n\r\nDeploying the target server’s application on a virtual machine within the same segment may still encounter these issues.\r\n\r\n### 4. OS of the Target Server\r\n\r\nThe target server might use `rpfilter` to verify the legitimacy of source IP addresses, dropping packets deemed forged. If packets are captured by `tcpdump` but not processed, check `rpfilter` settings and adjust or remove them as needed. Other issues like `iptables` settings may also affect `tcpcopy`.\r\n\r\n### 5. Applications on the Target Server\r\n\r\nApplications on the target server may not process all requests promptly. Bugs or limitations in the application can lead to delayed responses or unprocessed requests in the socket buffer.\r\n\r\n### 6. OS of the Assistant Server\r\n\r\nEnsure that `ip_forward` is set to false on the assistant server to prevent it from routing packets and ensure it functions as a black hole.\r\n\r\n## Logical Analysis of the Issue Where the Test Server Fails to Receive Data\r\n\r\nFirst, use `telnet` on the online server to connect to the test server's port. This will check if the network path is accessible. If the connection fails, resolve this issue before proceeding with the following diagnostics.\r\n\r\nAssume that during the `tcpcopy` test, the application on the test server does not receive any requests. Determine if the initial handshake packet (i.e., the SYN packet) reaches the test server.\r\n\r\n### 1. **If the SYN packet reaches the test server, the following scenarios are possible:**\r\n\r\n   **1.1 Only SYN Packets Captured:**\r\n   If you use `tcpdump` on the test server and see that the replicated SYN packets are arriving, it indicates that they have reached the data link layer of the test server. If `netstat` shows no connections for the application, it means the packets were dropped at the IP layer. Check if `rpfilter` is configured—if so, remove this setting, and the issue should generally be resolved. If `rpfilter` is not set, confirm that there are no conflicts in the `iptables` settings and adjust the relevant rules if necessary.\r\n\r\n   **1.2 SYN Followed by RST Packet:**\r\n   If the SYN packet is immediately followed by a reset (RST) packet (with less than 1 second between them in the same session), it indicates a routing issue or conflict, causing the response packet to be sent directly back to the real client.\r\n\r\n  **1.3 Test Server Responds with the Second Handshake Packet:**\r\n   Capture packets on the assistant server to check if the second handshake packet has reached it.\r\n\r\n- **If the packet hasn't reached the assistant server,** it suggests that the routing setup is not effective, and therefore `intercept` cannot capture the second handshake packet, preventing further replay. A potential solution is to run `intercept` directly on the test server (note: keep the routing setup unchanged, and ensure that the `-c` parameter in `tcpcopy` is not set to the IP address used by `tcpcopy` to connect to `intercept`, or else `tcpcopy` won’t connect to `intercept`).\r\n\r\n- **If the second handshake packet is captured,** check if `ip_forward` is enabled. If it is, disable this setting, as it may cause the response packets to be sent directly back to the client, interfering with the test.\r\n\r\n### 2. **If the SYN packet does not reach the test server, there are two possible scenarios:**\r\n\r\n   **2.1 `tcpcopy` Packets Captured on the Online Server:**\r\n   If you capture `tcpcopy`'s forwarded packets using `tcpdump` on the online server, but the packets do not reach the test server, it indicates that they were dropped along the way. You can try using the `-c` parameter in `tcpcopy` to modify the client IP address to a valid one. In extreme cases, set the client IP to the IP address of the machine running `tcpcopy` (note: NAT issues may arise, and if `intercept` is running on the test server, ensure the `-c` parameter in `tcpcopy` is not set to the IP address used by `tcpcopy` to connect to `intercept`, or else `tcpcopy` won’t connect to `intercept`).\r\n\r\n   **2.2 `tcpcopy` Packets Not Captured on the Online Server:**\r\n\r\n- **If no `all clt:xx` information is found in `tcpcopy`'s log,** it indicates that `tcpcopy` is unable to capture packets at the IP layer. In this case, use the `--pcap-capture` option to capture packets at the data link layer. Set the `-F` parameter (e.g., 'tcp and dst port 80 and dst host 10.100.1.2') and the `-i` parameter (network interface) to bypass IP layer capturing.\r\n\r\n- **If `all clt:xx`, where `xx > 0`, is seen in `tcpcopy`'s log,** it means `tcpcopy` successfully captured the packet, but it was filtered out by the IP layer on the online server. Check `iptables` restrictions on the output chain, among other settings. If `iptables` is the problem and cannot be modified on the online server, use the `--pcap-send` option to send packets from the data link layer.\r\n\r\n## Release History\r\n\r\n+ 2014.09  v1.0    TCPCopy released\r\n+ 2024.09  v1.0    Open source fully uses English\r\n\r\n## Bugs and Feature Requests\r\n\r\nHave a bug or a feature request? [Please open a new issue](https:\u002F\u002Fgithub.com\u002Fsession-replay-tools\u002Ftcpcopy\u002Fissues). Before opening any issue, please search for existing issues.\r\n\r\n## Support\r\nIf you find this project helpful, consider donating:\r\n[![Donate](https:\u002F\u002Fimg.shields.io\u002Fbadge\u002FDonate-PayPal-blue.svg)](https:\u002F\u002Fpaypal.me\u002Fwangbin579)\r\n\r\n## Copyright and License\r\n\r\nCopyright 2025 under [the BSD license](LICENSE).\r\n\r\n## Acknowledgments\r\n\r\nSeveral individuals have been crucial in the writing of this document by reviewing drafts and offering feedback. I am especially grateful for the contributions of Hongshen Wang. \r\n","TCPCopy 是一个用于互联网服务器应用程序真实测试的TCP流重放工具。其核心功能包括在线请求复制和TCP流重放，能够模拟生产环境中的请求多样性、网络延迟和资源使用情况。技术上，TCPCopy 通过原始套接字在网路层捕获数据包，并利用专门的组件进行响应信息传递，从而实现对目标服务器的测试。该工具适用于分布式压力测试、实时测试、回归测试以及性能对比等多种场景，尤其适合需要高度仿真生产环境负载的情况。使用 TCPCopy 可以帮助开发者发现仅在高压力或特定条件下才会出现的问题，确保系统稳定性和性能。","2026-06-11 03:07:51","top_language"]