This is more of a question for someone like @Jad.
I personally am unaware of the fact of if our server currently utilizes Fast DL at the current moment or not, but what you mention is most certainly common practice for any gmod server. Typically content from any server will get sent to your steamapps\common\GarrysMod\garrysmod\download folder, as well as your steamapps\common\GarrysMod\garrysmod\cache. This is more helpful than harmful, as the content it's placing into your folders helps you load faster because you have the content, rather than it being deleted every time like you say.
There's a couple of key controls server owners have at their disposal to handle how this feature is dealt with, but I'm personally clueless. In my eyes, what @Maverickmentioned is more likely the case, where this is seemingly a client related issue. Maybe your specs aren't the best, or as you said, your EU ping doesn't benefit you. There's lots of factors that come in to play.