There are a great deal of factors that can affect upload and download speeds from a dedicated server.

Resource usage: This is typically the most common culprit. Heavy CPU, RAM, or I/O usage can severely limit the transfer speeds that the server is capable of. The easiest way to diagnose this issue is to test within a clean environment, such as a Linux Live CD.

Peering and Congestion: Specific routes may have points of congestion, where the full throughput speed is not available at a specific point along the route between the server and client. Our direct connections with our upstream are guaranteed never to be more than 80% utilized, to allow for traffic bursts.

Distance: The further the distance between the client and the server the points of congestion and the higher the latency. Both of these can have a major part in transfer speeds. If there is no congestion along the route, the server OS may not have TCP Windows Scaling enabled.

Enabling TCP Windows Scaling in Linux and Windows will usually drastically improve transfer speeds.

Additional Resources:

https://systembash.com/tweaking-tcp-for-fast-100mbps-connections-and-transfers-on-linux/