While working for a broadcasting/networking integration gig, we designed, assembled and field-terminated thousands of copper and fibre runs from 1/2 ft to football field lengths.
The amount of time required to terminate a copper cable in the field is seconds, and felt a bit like art. Something about the way it reliably reacted was magical and felt "strong."
Terminating or splicing a fibre cable felt like wrestling a snake covered in melted crayons, and the failure rate was significantly higher across the board. And it wasn't just workmanship, but quality of product, terminating environment, available equipment, misuse by future operators etc.
That said, at a certain point, we as a firm learned that most purchasers would rather the low latency/small footprint of optical/fibre versus copper, maintenece/failure be damned. Though, maybe part of our willingness to push fibre came from knowing that most purchasers would in 1-2 years call us back in to replace the rack terminations with copper :)
zamadatix 7 hours ago [-]
There are 2 very different main types of "copper" in this context. Each can break down into more specific subcategories but they have a relatively common general behavior with their primary type.
First there is the BASE-T RJ45 stuff, which it sounds like you might have been working with. At 10G or higher speeds this get relatively power hungry and is not really an advantage over fiber unless you are also delivering PoE or are trying to reuse existing cabling.
This type (DAC) is a special type of pre-made cable assembly which eschews much of the advanced signalling/conversion logic. The upside is the power usage is low (often even lower than fiber) and the cost is dirt cheap. The downside is the lengths are much more limited and it's intended to be preterminated SFP-to-SFP (or the like) cable assemblies instead of modular patching/custom built.
There is indeed a latency difference, usually DAC < fiber < BASE-T, but they are all within a few microseconds (not milliseconds) of each other so you really have to be pressed to care about it (to the point you're looking at specialized low latency switches and paying extra to lay things out in a way which minimizes the number of L2 hops rather than the cost).
myself248 5 hours ago [-]
> it's intended to be preterminated SFP-to-SFP
I wonder why nobody's making field-terminatable DACs for custom lengths. If you've torn one down, they're not exactly complicated inside.
If they existed, would people buy them?
Aurornis 3 hours ago [-]
DAC cables use twinax. Properly terminating twinax is not easy. It's not as forgiving as twisted pair going into an RJ45.
DAC cables have a limited length range. It's not like 10GBASE-T where you can run the cable hundreds of feet and then put a connector on exactly where you need it. The cables only work at short distances so it's easy to stock the cable sizes close enough to everyone's needs.
zamadatix 4 hours ago [-]
I think the only reason RJ45 termination is an (occasional) thing for short patching is you're also in spec to go 100 meters with it so the tools and materials to do so are already commonly in use.
For passive DACs the range of lengths is so low you can just get away with having 2 or 3 different lengths on hand and never need to worry about it. Active DACs start to be too much to bother with again.
Fiber it's possible, but again really only because you can go kilometers with it rather than because people want to make short patch cables themselves.
Filligree 5 hours ago [-]
Doesn't that reduce to RJ45?
mwpmaybe 5 hours ago [-]
No. It's twin-axial. Think coaxial, but more axial.
jeffrallen 5 hours ago [-]
Nope. For production, you want to reduce risk and variation. DACs are already available in about 5 sizes up to the max 7m length, why would you terminate any other size in the field?
iso1631 4 hours ago [-]
I did spec a couple of 7m dacs a few months ago to run between two adjacent bays, but normally for more than 2m I'll just drop a SM sfp and run a preterminated fibre cable.
In the field its the armoured fibre on a reel, 100m, 200m, 500m etc, with opticon connectors, or some normal cat5 typically for APs
bobdvb 3 hours ago [-]
Some network guys I know prefer fibre even within the rack, just because they don't want the weight and the obstructions in the rack. Apparently more than once the weight of the DACs and the bulk of the cable bundle has caused a problem with NICs.
Personally that surprised me, but I can see where they're coming from.
mwpmaybe 1 hours ago [-]
If you start doing bonded links with DACs or if you have a bunch of servers, the cable management situation gets ugly in a hurry, and the usual solutions like patch panels and keystones aren't applicable. Source: my basement
buildbot 2 hours ago [-]
Yep, I did this for my little DGx spark cluster because 100-200gb copper cables are very thick, heavy, and annoying.
Aurornis 3 hours ago [-]
> The amount of time required to terminate a copper cable in the field is seconds,
This article is about Direct Attach Copper (DAC) cables which are not something you can field terminate. They use twinax copper and have special modules on the end.
You are thinking about standard RJ45 terminated cabling for 1GBASE-T or 10GBASE-T, which is a different.
> That said, at a certain point, we as a firm learned that most purchasers would rather the low latency/small footprint of optical/fibre versus copper, maintenece/failure be damned.
Direct Attach Copper has slightly lower latency than fiber, but the difference is negligible. Both have significantly better latency than 10GBASE-T through twisted pair cabling.
SV_BubbleTime 2 hours ago [-]
Yes, thanks. It was confusing me for a second because no, no almost one is terminating twinax in the server room.
I had IT company recommend to me a bunch of fiber cables for a cleanup I was doing. They had about 20 or 30 laser modules we would need, and however much fiber.
When they asked why I planned on doing copper, I told him because each run is three fucking feet.
I’m not sure if they just didn’t realize that’s not what fiber is for or they didn’t know that DAC existed.
Aurornis 2 hours ago [-]
Many people don’t know about DAC. Like the comment above I think they see the word copper and assume it means the twisted pair they’re familiar with.
runjake 2 hours ago [-]
Couple points:
1. The copper cables discussed in the article are not field terminable. And if they were, they'd be a pain in the ass.
2. Terminating fiber used to be a pain, but is now pretty easy with the right tools, fuser, and someone with basic training. Even cheap fusers do the job with very low failure rates. They now have so-called "knuckle draggers" terminating fiber.
throw0101c 6 hours ago [-]
> That said, at a certain point, we as a firm learned that most purchasers would rather the low latency/small footprint of optical/fibre versus copper, maintenece/failure be damned.
Copper bundles get real thick, real fast: I ran an OneFS cluster for many years, and we had >50 nodes, and all the cables (each node dual-connected) ran to two central switches for backend replications. Rat's nest.
I was very happy when Isilon started officially supporting active optical cable (AOC) on the backend. Really helped with airflow and keeping things tidy.
kohlschuetter 6 hours ago [-]
Many 10GbE DACs can be hacked to run 25GbE, which is something I had to figure out after realizing that it’s not just plug and play as it was with twisted pair cables coming from 1Gb to 2.5Gb…
I'm waiting for 3 DACs and a few other bits to arrive today to move closer to 10G networking at home. Moving house soon and the new place will have 2.5Gbps FTTP (both up and down) so I wanted to be prepared for that. Given my existing broadband is only 500/75Mbps FTTP I was fine with a 1GbE internal network and Wifi-6 meshing. I could have planned to move to 2.5GbE but it may have been a bottleneck at some point, so may as well push straight on to 10G.
I have a USW-Aggregation with 8 SFP+ ports arriving today too. Just have to install Intel X520-DA2 cards in two of my servers (Proxmox host and a general Linux server), and the NAS also has a 10G SFP+ port, and then connect it all up.
Most of it second hand from eBay for half the usual retail price.
lostlogin 6 hours ago [-]
Nice work, that agg switch is excelllwnt.
I went with some cheap eBay cards and slotted them into a synology and PC.
The card is obviously 16-lane, but it also has two ports; 40Gb total. In a server that’s fine, but if you want 10G in a desktop you’ll have a problem.
I’m probably not telling you anything new. NICs using newer PCI generations are rare as hen’s teeth. It should be possible to do this with four lanes, but isn’t…
Unless you find a 25G dual-port card, in which case the single lane my secondary slots hand out does at least suffice for 10G one way.
zamadatix 5 hours ago [-]
PCIe is also a full duplex connection so 2x10G is still just 20G instead of 40G. For PCIe 2.0 an x8 connection should get you full bandwidth on both ports simultaneously while x4 will fall just short for simultaneously usage (but still higher than 1 port). Unless you're really hankering for that full 20G, in which case a 25G NIC is definitely the better pick, that means you can just slot it in an x4 slot off the chipset on even a standard desktop PC.
Funnily enough, if you want a dirt cheap PCIe 3.0 based card the MCX353A-QCBT and MCX354A-QCBT give 1/2 ports of 40G QSFP+. They support QSFP+ to SFP+ adapters, so you can plug a 10G SFP+ into the QSFP+ port, but they don't support 4x10G breakout unfortunately. I ended up using the 2 port variant in both of my NASes - one port is 40G between the 2 for dirt cheap fast backups and the other is adapted to 10G to connect to the rest of the home network.
mwpmaybe 4 hours ago [-]
I've started buying Intel E810s for most purposes, even for 10G links. (SFP28 ports are generally backward-compatible with SFP+ DACs and transceivers.) The ones you can get on eBay for cheap typically run Dell firmware but it's serviceable. An E810-XXVDA2 is Gen4 x8; as long as the host slot can physically accept the card connector you only need Gen4 x1 electrical for a single 10G link or Gen4 x2 for dual 10G or single 25G.
alexfoo 5 hours ago [-]
I'm only planning on using one of the SFP+ ports on each of the cards, the dual port cards were just more common and cheaper on eBay.
The specs say they require PCIe v2.1 x8 lane.
My Proxmox server is quite old and has a Gigabyte GA-X79-UP4 mobo and has loads of spare PCI slots. One slot is taken up by a generic graphics card as the Mobo has no on-board graphics. (I think I went for this mobo because of the number of SATA ports, but it was over 10 years ago so not entirely sure.)
My general Linux server is newer and has an ASUS Prime H610M-A D4 mobo. Only two PCI slots (not used at the moment) and so the Intel X540-DA2 will use up the PCIe 4.0 x16 slot leaving just a PCIe 3.0 x1 slot. But that's fine as this machine is just a CPU (i7-13700), 64GB RAM and a 2TB NVMe. Sticking a good graphics card in it for GPU related fun had been on my list for years but I never got around to it, now the prices are just insane so I'll ignore that for now or something second hand falls into my lap.
vermilingua 5 hours ago [-]
I really have to wonder what can you use 10G for? I have 500M down from my ISP, and it is faster than I can imagine ever needing, unless I get into data-hoarding 8k movies.
mwpmaybe 4 hours ago [-]
My homelab has a 10G fabric (switched) for NFS, iSCSI, NVMe-OF, etc. and a 25G fabric (a mix of back-to-back and switched) for clustering (Ceph, DRDB, ZFS replication, migrating VMs).
I spun up some iSCSI-backed SQL Server a few months ago and 10G couldn't keep up with the workload, so I dropped in a pair of 100G ConnectX-4 cards with iSER (iSCSI Extensions for RDMA) support for that particular use-case.
Just because your uplink is less than 10G doesn't mean the rest of your network can't be a bit more capable. :)
alexfoo 5 hours ago [-]
True, I don't really feel limited by my existing 500Mbps down, but knowing I'll be having 2500Mbps up/down soon means I want to have the infra to handle it.
Basing things on 2.5GbE would certainly have been cheaper but some things don't support it (they either do 1GbE or 10G SFP+) so settling on 10G where possible made more sense to me. My future ISP also has a 5Gbps up/down option, but even I can't justify that right now.
My wife and kid just want their phones/laptops to work, and to be able to stream stuff to watch, they don't care about the underlying speed.
Having a faster network may make some of my work related things run a bit quicker. A few times a day I'll need to pull something big down (either an ISO or a bunch of docker images) and that can take up to 2 minutes with 500Mbps down. Having those take a fifth of that time will make it seem less of a roadblock to doing work. 2 minutes meant I went and got a cup of coffee and often got more distracted, 30 seconds should keep me at my desk and focused on what I was doing. That's not a big enough reason to justify it on its own obviously.
I also want to do offsite backups with/for various family members, so something better than 75Mbps up is going to be a huge boost. Getting 1Gbps+ out will be huge (assuming whatever is at the other end can support that).
I don't do any kind of data hoarding, I think I've got something under 4TB of data that I actually care about, and most of that are family photos/videos.
Deep down it's mostly because I'm a networking geek so it's fun to play with some new kit and make blinkenlights.
hylaride 4 hours ago [-]
For most people, 500M is probably fine. But once you have a few family members, each streaming 4K movies to their devices, and a parent that needs a video call to work seamlessly, you start to see the benefits.
10G is probably overkill, but it's also future proofing. The way things are going, loading the NYtimes will require 10G just for the advertising alone...
zamadatix 5 hours ago [-]
It's less "what new thing can you do" and more "what things involve noticeably waiting, how long is the waiting, and what else is impacted". E.g. updating a game on Steam practically takes slightly under half the time for me (1.2 Gbps actual rate) and has absolutely 0 impact to any other traffic in the house. If it was 10x the price to get 10x the bandwidth I wouldn't bother but it was actually about the same as my old cable modem plan.
whalesalad 3 hours ago [-]
What if you want to access your NAS at 10g+ speed? You're focusing on WAN when there is also the LAN side.
toast0 2 hours ago [-]
[dead]
bombcar 6 hours ago [-]
I did similar with the Mikrotik CRS305-1G-4S+IN and some surplus eBay gear. The nice thing is the NAS and my MacBook dock both have 10G and are connected - and it’s noticeable.
alexfoo 5 hours ago [-]
I had a big debate with myself whether to go Mikrotik or Unifi. Being EU based I really wanted to go Mikrotik but ended up with Unifi as I'd had more experience of it when helping out friends/neighbours.
Maybe my "last house" (i.e. the one we'll get to see us through to retirement and beyond) will be Mikrotik based. By then I'll probably want as little computing stuff as possible and will just sit in a comfy chair doing crosswords and sudoku with a pencil.
bombcar 3 hours ago [-]
Mikrotik is "quite low level" if you want it to be (it reminds me somewhat of old Cisco IOS) but it works great.
And even if you're a bit scared of manual configurations, the web GUI and Claude understand it pretty well.
mwpmaybe 4 hours ago [-]
I use UniFi for most of my home network so It Just Works™, but I've thought about mixing in Mikrotik for e.g. the compute rack so I can play around with 100G+ links and more esoteric stuff like VXLAN.
justsomehnguy 5 hours ago [-]
> Most of it second hand from eBay for half the usual retail price.
You were scammed. X520 is old enough to drive a car, the shop should pay you to get it off their hands.
alexfoo 5 hours ago [-]
Ha. I meant the rest of the equipment (USW-Aggregation, Unifi Pro Max 16, UNAS Pro, Unifi Express 7) was somewhere around half retail price.
I think I paid ~$15 for each X520-DA2 including postage.
mwpmaybe 4 hours ago [-]
I've started buying E810s even for 10G links. PCIe Gen4, lower power draw, RDMA support, generally backward-compatible with SFP+ DACs and transceivers, and relatively inexpensive. Not nearly as dirt-cheap as the X520s but not crazy expensive (last I looked, at least). As I gradually replace switches over the next few years I can start taking advantage of 25G.
theatrus2 3 hours ago [-]
This is the way. The 10G cards are ancient and hog all the PCIe lanes.
I’ve had the best luck with Mellanox ConnectX 4 or 5 cards. The 5 can happily run 25G on a modern lane constrained system.
whalesalad 6 hours ago [-]
I just went through the same process over the last few months. I had a USW agg and ran out of ports so now I have the big dog 24-port version. Mainly wanted L3 routing capability but it’s nice having more ports to lagg connections.
alexfoo 5 hours ago [-]
The limiting factor for me is that I'm renting so I can't put my own cabling in to the property. And with the new place there's no existing cabling, nor any conduits to run anything in, and chasing things into the walls/etc is going to be prohibited by the landlord or just too expensive if I'm only in this place for a year or two.
The spools of bend insensitive fibre are pretty cheap and very discreet so I'll probably have a couple of those running along skirting boards/etc in order to connect disparate areas of the house. (The ONT is ~15m away from where the majority of the equipment will live, that's the main bit I have to bridge.)
whalesalad 3 hours ago [-]
I used to sneak cabling around the basement and pop it out of AC register vents when I was renting. I had one cable coming out the access panel to a bathtub which was conveniently in my office as I was next to the bathroom.
Edwardian houses in the UK rarely have that level of access. No basement at all and I can't lift the carpets and floorboards to get to where I might be able to pass things through/around. No AC ducts. No coax to be able to use MoCA either.
But, yes, that video is exactly the kind of thing I had in mind for the bend insensitive fibre.
It all depends how I set things up (and I can't tell that until I've had more access to the property). The ONT and the rack with the USW-Aggregation switch are 10 yards apart, in terms of absolute distance, but probably 20 yards if you follow the walls/skirting-boards/etc.
The FTTP is presented as 2.5GbE Ethernet (apparently) so I can either:
However this will be sub-optimal in terms of Wifi and I'll probably need extra APs to cover all three floors and out into the back yard.
b) put my Unifi Express 7 in the hallway in the middle of the house (which should give me full Wifi coverage with no extra APs). This would mean a short (2m) DAC to connect it to the USW-Aggregation nearby, and I can use a 20m long flat/flexible Cat-6 Ethernet cable to go between the ONT and the Unifi Express 7.
Shank 6 hours ago [-]
I’m a huge fan of fiber optic cables instead of DAC. For one thing, the distance you can run is far greater. For another, they don’t get as hot as metal cables, particularly at high speeds. OM3 and OM4 is actually quite affordable and SFP+ is nice.
Yes on an absolute sense, DAC can be cheaper, but for a homelab or something, it’s not that much different. Also, it’s much cooler to run fiber optic to things (imho).
GoofGarage 5 hours ago [-]
In a homelab scenario when asking “DAC vs. fiber” the answer is usually, “yes.” :) Basically, it’s a tradeoff as with everything else.
DACs will usually be even (slightly) lower power per port, and slightly lower latency[1] (we’re fighting over microseconds here!), with excellent durability. The tradeoff is for passive DACs you’re limited on range, cost is often higher, and they may need to be encoded for your interfaces. Moreover, the range is very limited.
Fiber (the cable) is immune to electrical noise, can run long distances, advances in wave division multiplexing extends the life of the fiber by changing what’s the fiber connects to. The downside is you pay slightly in latency for media interface changes (the electrical-to-optical conversions), the limits of bend radius of the cable to not break the cable or reduce bandwidth, and the relative complexity of field terminations compared to twisted pair. I’ve 25+ years experience with fiber, and trust me, it’s great.
————
Outside of cost, both crush twisted pair like an ant. The power consumption per port is also far lower. However, this is only going to matter if you focus on limiting power consumption (not for cost, on principle), have very high-bandwidth applications where latency matters (I do!), and/or just want field experience with things other than twisted pair.
I use DAC and fiber for some things as I try to get every scrap of capability out of my hardware. For example, I have VERY low power (silent or near silent) hardware where I can push 5GB (so ~40Gbps) / sec storage. Not just sending it over the wire, but actually committing it to disk without buffering in RAM. So I have the capability of “PCIe 3.0/4.0 x4 NVMe” speeds across the network… from the (mostly silent) storage server, to anything else that can send or ingest the data that fast. Despite the storage server having very little flash (a few TB vs 100TB+ disk). That’s harder to do with twisted pair, or at least the power consumption of the network connectivity itself starts to add up for a few virtualization cluster nodes.
———-
Generally, “DAC in the rack, fiber to out back” is a reasonable approach. Though “fiber-only” works if you want to limit complexity!
[1] Fiber and DAC tend to trade places on latency every generation or so. It’s a very close race, but they crush twisted pair.
HappMacDonald 3 hours ago [-]
I think DAC has to win out on price not because of the price of the fiber patch but those optical modules can cost some money.
I'm getting a really bad taste in my mouth for 10GBASE-T RJ45 SFP+ modules mostly due to the god awful heat they pump out (which scales linearly with power draw): So when we are forced to use them (for example to connect to a far end that only accepts 10GBASE-T) we often cannot use any SFP cage directly adjacent.
I had the Twisted Pair/DAC/Fiber discussion with one of my an engineer that we peer with recently and they made the pitch for Fiber that "being able to see the light levels takes a lot of the guesswork out of their troubleshooting". So I do have to admit that that is one utility lost when choosing a DAC over Fiber even in a rack, but my counter is that DAC has fewer moving parts and thus a lower surface area for failure in the first place.
We have had some gear refuse to work with some DACs (just as some gear refuses to work with some SFP+ modules of various stripes) but aside from that I have yet to see a DAC that starts working stop working, while I get that with fiber runs (though usually the longer ones running outdoors) quite frequently.
mwpmaybe 1 hours ago [-]
> I'm getting a really bad taste in my mouth for 10GBASE-T RJ45 SFP+ modules mostly due to the god awful heat they pump out
It's truly bananas especially in a homelab environment. Nearly every time I've thought to myself "oh I can just do a copper gbic for this" it's been the wrong thing to do.
mwpmaybe 5 hours ago [-]
Couldn't have said it better myself.
sedimannapoleon 5 hours ago [-]
[dead]
KaiserPro 2 hours ago [-]
When I did server room buildouts for VFX, I used to use DAC for short distance and fibre for anything longer than a rack.
However that was mainly because I always thought fibre was much more fragile. I mean in a sense it is, it really doesn't like being pull through a thatch of other wires. The solution of course is to do nice cabling.
Nowadays I'm not actually sure if DACs are cheaper than fibre now
mwpmaybe 1 hours ago [-]
> Nowadays I'm not actually sure if DACs are cheaper than fibre now
Yep still are, although fiber transceivers are surprisingly affordable, and of course you can usually re-use jumpers when you upgrade transceivers so TCO over a couple generations might be comparable.
throw0101c 6 hours ago [-]
> This is a bit dependent on the type of DAC, and the vendor. Generally speaking, we are moving into an era as we move above 100GbE speeds where DACs will be limited to 5m runs and shorter runs.
This is generally good enough for intra-rack runs, especially if you put your top-of-rack (ToR) switch in the middle: you basically only have to go about 1m/3' in either direction (up/down).
Havoc 6 hours ago [-]
Surprised it doesn't mention heat - one of the big advantages.
There is also a new generation of "invisible" fiber that is interesting for home use. Basically looks like fishing line so you can run it somewhere that you just don't see unless you're really looking. Fragile tho
anonymousiam 3 hours ago [-]
I picked up one of these just a few days ago for use with some SFP28 ports. They're cheap, fast, and don't use any power.
happyPersonR 2 days ago [-]
Dacs are great … they let you avoid having to convert to optical or having to splice your own cable.
Downside the cables are kind of heavy and sometimes folks dont like that.
bombcar 6 hours ago [-]
Main advantage for me was saving the cost of the SPF (surprisingly pricey f^#%rs) on each end.
namibj 5 hours ago [-]
Huh, what were you doing, buying Cisco/HPE, or trying to use fancy single mode optics for distances that DACs could do?
bombcar 5 hours ago [-]
I just wanted to use my CRS305-1G-4S+IN without spending more on SPF modules than I did on the cards/switch.
protocolture 7 hours ago [-]
Dacs are awesome until you encounter a full rack with nothing but 2 and 4 meter dacs. They appeal very much to the lazy engineer, and a lazy engineer can turn them into a rats nest.
HappMacDonald 2 hours ago [-]
You.. but you put a ladder on the side.
Rat's nest goes in the ladder, cover goes on top of the ladder. That way cables aren't blocking access to add/remove any U's.
You make sure all of the cabling has ends that can be pulled safely through the rat nests in the ladders (EG especially RJ45 cabling needs competent clip guards) and Bob's your uncle
kjs3 6 hours ago [-]
To be fair, the dedicated lazy engineer can make a rats nest out of any interconnect technology.
mercurialuser 5 hours ago [-]
And they can block heat exhaustion
amelius 3 hours ago [-]
Is this connection galvanically isolated?
jeffrallen 5 hours ago [-]
Too bad they didn't mention that while the cable is direct attach, the ends each have an EEPROM in them saying what the cable is. This EEPROM can be a source of immense frustration, if it's got the wrong data in it, or if both ends do not agree.
83 4 hours ago [-]
This should be higher up for anyone considering DACs for homelab use. I had a lot of issues due to this when mixing Aruba switches with Intel nics in my rack. Ended up going to fiber and SFPs.
Rendered at 18:31:29 GMT+0000 (Coordinated Universal Time) with Vercel.
The amount of time required to terminate a copper cable in the field is seconds, and felt a bit like art. Something about the way it reliably reacted was magical and felt "strong."
Terminating or splicing a fibre cable felt like wrestling a snake covered in melted crayons, and the failure rate was significantly higher across the board. And it wasn't just workmanship, but quality of product, terminating environment, available equipment, misuse by future operators etc.
That said, at a certain point, we as a firm learned that most purchasers would rather the low latency/small footprint of optical/fibre versus copper, maintenece/failure be damned. Though, maybe part of our willingness to push fibre came from knowing that most purchasers would in 1-2 years call us back in to replace the rack terminations with copper :)
First there is the BASE-T RJ45 stuff, which it sounds like you might have been working with. At 10G or higher speeds this get relatively power hungry and is not really an advantage over fiber unless you are also delivering PoE or are trying to reuse existing cabling.
This type (DAC) is a special type of pre-made cable assembly which eschews much of the advanced signalling/conversion logic. The upside is the power usage is low (often even lower than fiber) and the cost is dirt cheap. The downside is the lengths are much more limited and it's intended to be preterminated SFP-to-SFP (or the like) cable assemblies instead of modular patching/custom built.
There is indeed a latency difference, usually DAC < fiber < BASE-T, but they are all within a few microseconds (not milliseconds) of each other so you really have to be pressed to care about it (to the point you're looking at specialized low latency switches and paying extra to lay things out in a way which minimizes the number of L2 hops rather than the cost).
I wonder why nobody's making field-terminatable DACs for custom lengths. If you've torn one down, they're not exactly complicated inside.
If they existed, would people buy them?
DAC cables have a limited length range. It's not like 10GBASE-T where you can run the cable hundreds of feet and then put a connector on exactly where you need it. The cables only work at short distances so it's easy to stock the cable sizes close enough to everyone's needs.
For passive DACs the range of lengths is so low you can just get away with having 2 or 3 different lengths on hand and never need to worry about it. Active DACs start to be too much to bother with again.
Fiber it's possible, but again really only because you can go kilometers with it rather than because people want to make short patch cables themselves.
In the field its the armoured fibre on a reel, 100m, 200m, 500m etc, with opticon connectors, or some normal cat5 typically for APs
Personally that surprised me, but I can see where they're coming from.
This article is about Direct Attach Copper (DAC) cables which are not something you can field terminate. They use twinax copper and have special modules on the end.
You are thinking about standard RJ45 terminated cabling for 1GBASE-T or 10GBASE-T, which is a different.
> That said, at a certain point, we as a firm learned that most purchasers would rather the low latency/small footprint of optical/fibre versus copper, maintenece/failure be damned.
Direct Attach Copper has slightly lower latency than fiber, but the difference is negligible. Both have significantly better latency than 10GBASE-T through twisted pair cabling.
I had IT company recommend to me a bunch of fiber cables for a cleanup I was doing. They had about 20 or 30 laser modules we would need, and however much fiber.
When they asked why I planned on doing copper, I told him because each run is three fucking feet.
I’m not sure if they just didn’t realize that’s not what fiber is for or they didn’t know that DAC existed.
1. The copper cables discussed in the article are not field terminable. And if they were, they'd be a pain in the ass.
2. Terminating fiber used to be a pain, but is now pretty easy with the right tools, fuser, and someone with basic training. Even cheap fusers do the job with very low failure rates. They now have so-called "knuckle draggers" terminating fiber.
Copper bundles get real thick, real fast: I ran an OneFS cluster for many years, and we had >50 nodes, and all the cables (each node dual-connected) ran to two central switches for backend replications. Rat's nest.
I was very happy when Isilon started officially supporting active optical cable (AOC) on the backend. Really helped with airflow and keeping things tidy.
https://kohlschuetter.github.io/blog/posts/2026/03/22/unlock...
I have a USW-Aggregation with 8 SFP+ ports arriving today too. Just have to install Intel X520-DA2 cards in two of my servers (Proxmox host and a general Linux server), and the NAS also has a 10G SFP+ port, and then connect it all up.
Most of it second hand from eBay for half the usual retail price.
I went with some cheap eBay cards and slotted them into a synology and PC.
They work great and have for years.
https://www.ebay.com/itm/384094168784?_skw=connectx+mellanox...
The card is obviously 16-lane, but it also has two ports; 40Gb total. In a server that’s fine, but if you want 10G in a desktop you’ll have a problem.
I’m probably not telling you anything new. NICs using newer PCI generations are rare as hen’s teeth. It should be possible to do this with four lanes, but isn’t…
Unless you find a 25G dual-port card, in which case the single lane my secondary slots hand out does at least suffice for 10G one way.
Funnily enough, if you want a dirt cheap PCIe 3.0 based card the MCX353A-QCBT and MCX354A-QCBT give 1/2 ports of 40G QSFP+. They support QSFP+ to SFP+ adapters, so you can plug a 10G SFP+ into the QSFP+ port, but they don't support 4x10G breakout unfortunately. I ended up using the 2 port variant in both of my NASes - one port is 40G between the 2 for dirt cheap fast backups and the other is adapted to 10G to connect to the rest of the home network.
The specs say they require PCIe v2.1 x8 lane.
My Proxmox server is quite old and has a Gigabyte GA-X79-UP4 mobo and has loads of spare PCI slots. One slot is taken up by a generic graphics card as the Mobo has no on-board graphics. (I think I went for this mobo because of the number of SATA ports, but it was over 10 years ago so not entirely sure.)
My general Linux server is newer and has an ASUS Prime H610M-A D4 mobo. Only two PCI slots (not used at the moment) and so the Intel X540-DA2 will use up the PCIe 4.0 x16 slot leaving just a PCIe 3.0 x1 slot. But that's fine as this machine is just a CPU (i7-13700), 64GB RAM and a 2TB NVMe. Sticking a good graphics card in it for GPU related fun had been on my list for years but I never got around to it, now the prices are just insane so I'll ignore that for now or something second hand falls into my lap.
I spun up some iSCSI-backed SQL Server a few months ago and 10G couldn't keep up with the workload, so I dropped in a pair of 100G ConnectX-4 cards with iSER (iSCSI Extensions for RDMA) support for that particular use-case.
Just because your uplink is less than 10G doesn't mean the rest of your network can't be a bit more capable. :)
Basing things on 2.5GbE would certainly have been cheaper but some things don't support it (they either do 1GbE or 10G SFP+) so settling on 10G where possible made more sense to me. My future ISP also has a 5Gbps up/down option, but even I can't justify that right now.
My wife and kid just want their phones/laptops to work, and to be able to stream stuff to watch, they don't care about the underlying speed.
Having a faster network may make some of my work related things run a bit quicker. A few times a day I'll need to pull something big down (either an ISO or a bunch of docker images) and that can take up to 2 minutes with 500Mbps down. Having those take a fifth of that time will make it seem less of a roadblock to doing work. 2 minutes meant I went and got a cup of coffee and often got more distracted, 30 seconds should keep me at my desk and focused on what I was doing. That's not a big enough reason to justify it on its own obviously.
I also want to do offsite backups with/for various family members, so something better than 75Mbps up is going to be a huge boost. Getting 1Gbps+ out will be huge (assuming whatever is at the other end can support that).
I don't do any kind of data hoarding, I think I've got something under 4TB of data that I actually care about, and most of that are family photos/videos.
Deep down it's mostly because I'm a networking geek so it's fun to play with some new kit and make blinkenlights.
10G is probably overkill, but it's also future proofing. The way things are going, loading the NYtimes will require 10G just for the advertising alone...
Maybe my "last house" (i.e. the one we'll get to see us through to retirement and beyond) will be Mikrotik based. By then I'll probably want as little computing stuff as possible and will just sit in a comfy chair doing crosswords and sudoku with a pencil.
And even if you're a bit scared of manual configurations, the web GUI and Claude understand it pretty well.
You were scammed. X520 is old enough to drive a car, the shop should pay you to get it off their hands.
I think I paid ~$15 for each X520-DA2 including postage.
I’ve had the best luck with Mellanox ConnectX 4 or 5 cards. The 5 can happily run 25G on a modern lane constrained system.
The spools of bend insensitive fibre are pretty cheap and very discreet so I'll probably have a couple of those running along skirting boards/etc in order to connect disparate areas of the house. (The ONT is ~15m away from where the majority of the equipment will live, that's the main bit I have to bridge.)
This vid comes to mind when you said bend insensitive fiber: https://www.youtube.com/watch?v=Z2FbzCyiNr4
But, yes, that video is exactly the kind of thing I had in mind for the bend insensitive fibre.
It all depends how I set things up (and I can't tell that until I've had more access to the property). The ONT and the rack with the USW-Aggregation switch are 10 yards apart, in terms of absolute distance, but probably 20 yards if you follow the walls/skirting-boards/etc.
The FTTP is presented as 2.5GbE Ethernet (apparently) so I can either:
a) put my Unifi Express 7 next to the ONT and then need a fibre run (something like https://uk.store.ui.com/uk/en/category/accessories-modules-f...) from the SFP+ port on the Express 7 to the USW-Aggregation in the rack.
However this will be sub-optimal in terms of Wifi and I'll probably need extra APs to cover all three floors and out into the back yard.
b) put my Unifi Express 7 in the hallway in the middle of the house (which should give me full Wifi coverage with no extra APs). This would mean a short (2m) DAC to connect it to the USW-Aggregation nearby, and I can use a 20m long flat/flexible Cat-6 Ethernet cable to go between the ONT and the Unifi Express 7.
Yes on an absolute sense, DAC can be cheaper, but for a homelab or something, it’s not that much different. Also, it’s much cooler to run fiber optic to things (imho).
DACs will usually be even (slightly) lower power per port, and slightly lower latency[1] (we’re fighting over microseconds here!), with excellent durability. The tradeoff is for passive DACs you’re limited on range, cost is often higher, and they may need to be encoded for your interfaces. Moreover, the range is very limited.
Fiber (the cable) is immune to electrical noise, can run long distances, advances in wave division multiplexing extends the life of the fiber by changing what’s the fiber connects to. The downside is you pay slightly in latency for media interface changes (the electrical-to-optical conversions), the limits of bend radius of the cable to not break the cable or reduce bandwidth, and the relative complexity of field terminations compared to twisted pair. I’ve 25+ years experience with fiber, and trust me, it’s great.
————
Outside of cost, both crush twisted pair like an ant. The power consumption per port is also far lower. However, this is only going to matter if you focus on limiting power consumption (not for cost, on principle), have very high-bandwidth applications where latency matters (I do!), and/or just want field experience with things other than twisted pair.
I use DAC and fiber for some things as I try to get every scrap of capability out of my hardware. For example, I have VERY low power (silent or near silent) hardware where I can push 5GB (so ~40Gbps) / sec storage. Not just sending it over the wire, but actually committing it to disk without buffering in RAM. So I have the capability of “PCIe 3.0/4.0 x4 NVMe” speeds across the network… from the (mostly silent) storage server, to anything else that can send or ingest the data that fast. Despite the storage server having very little flash (a few TB vs 100TB+ disk). That’s harder to do with twisted pair, or at least the power consumption of the network connectivity itself starts to add up for a few virtualization cluster nodes.
———-
Generally, “DAC in the rack, fiber to out back” is a reasonable approach. Though “fiber-only” works if you want to limit complexity!
[1] Fiber and DAC tend to trade places on latency every generation or so. It’s a very close race, but they crush twisted pair.
I'm getting a really bad taste in my mouth for 10GBASE-T RJ45 SFP+ modules mostly due to the god awful heat they pump out (which scales linearly with power draw): So when we are forced to use them (for example to connect to a far end that only accepts 10GBASE-T) we often cannot use any SFP cage directly adjacent.
I had the Twisted Pair/DAC/Fiber discussion with one of my an engineer that we peer with recently and they made the pitch for Fiber that "being able to see the light levels takes a lot of the guesswork out of their troubleshooting". So I do have to admit that that is one utility lost when choosing a DAC over Fiber even in a rack, but my counter is that DAC has fewer moving parts and thus a lower surface area for failure in the first place.
We have had some gear refuse to work with some DACs (just as some gear refuses to work with some SFP+ modules of various stripes) but aside from that I have yet to see a DAC that starts working stop working, while I get that with fiber runs (though usually the longer ones running outdoors) quite frequently.
It's truly bananas especially in a homelab environment. Nearly every time I've thought to myself "oh I can just do a copper gbic for this" it's been the wrong thing to do.
However that was mainly because I always thought fibre was much more fragile. I mean in a sense it is, it really doesn't like being pull through a thatch of other wires. The solution of course is to do nice cabling.
Nowadays I'm not actually sure if DACs are cheaper than fibre now
Yep still are, although fiber transceivers are surprisingly affordable, and of course you can usually re-use jumpers when you upgrade transceivers so TCO over a couple generations might be comparable.
This is generally good enough for intra-rack runs, especially if you put your top-of-rack (ToR) switch in the middle: you basically only have to go about 1m/3' in either direction (up/down).
There is also a new generation of "invisible" fiber that is interesting for home use. Basically looks like fishing line so you can run it somewhere that you just don't see unless you're really looking. Fragile tho
Downside the cables are kind of heavy and sometimes folks dont like that.
You make sure all of the cabling has ends that can be pulled safely through the rat nests in the ladders (EG especially RJ45 cabling needs competent clip guards) and Bob's your uncle