For a project we have 50 servers all equiped with (generally) the same hardware. The issue we have here is very serious and happens on all machines. Despite a lot of effort and contacting manufacturs and the software developpers everyone points to eachother and even refuses to give me a clue about what is going on.
First let me describe the setup. This is 'servergrade' hardware. For my first experience, servergrade is the largest dissappointment in my life.
* SuperMicro X10SDV-8C+-LN2F
* Intel Xeon D-1540 (embedded on the motherboard)
* Custom designed 1U case or SuperMicro original case
* 480 watt server PSU or 200 watt SuperMicro original PSU
* Samsung Evo 850 500 GB SSD
* 32 GB DDR4-2133 ECC or NON-ECC (but not mixed in the same server) (officially supported)
* Asus GT730 4GB DDR3 GPU
* GPU is mounted with a PCIe riser card (not ribbon), nameless from China or SuperMicro original
Running on the system
* Windows Server 2012 R2 Enterprise
* VMWare Workstation 12
* VM's run GPU intensive tasks
* This system is stock, there's not over/underclocking at all
* Random BSOD 0x09c (aka Machine_Check_Exception) or 0x000000124
* Random as in sometimes the system runs for a week with no problems, sometimes in crashes after just 10 minutes, but mostly it runs for a few hours.
We have the exact same *software* setup running on desktop computers (with desktop hardware). These system can run fine with 1 our of 100 PC's crashing every month.
* BIOS updated to latest version (I would think now that this improved the time for the system to be stable, but that could have been random.
* Windows updated to the latest version
* VMWare updated to the latest version
* Swapped all components and tried every different option, even tried a desktop ATX PSU and M.2 SSD.
* Installed all systems from scratch with Ubuntu. I'm not familiar with Linux and have never seen a Linux BSOD and I still didn't since server systems are headless and I tried this in the DC. RESULT: system would hang and after reboot Linux reported XORG crash (GPU related).
* Changed GPU setting in BIOS to 'Above 4G', the rest of the BIOS is factory default.
* Tried both W2012 R2 and W7 Pro
* Disabled all C states as recommended by Supermicro support
* Tried different GPU's
* Tried different GPU drivers. Also tried the driver that is used on the stable ATX pc's.
* Bluescreenview says that it is caused by hall.dll on the W7 pc's and ntoskrnl.exe on the W2012 servers, but there is no real cause for it to find online.
* Systems are located in a datacenter. Temperature, air, power and network are optimal.
* Temperatures are well below the factory maximum
* We have the exact same software setup running on desktop computers (with desktop hardware). These system can run fine with 1 our of 100 PC's crashing every month.
* I have contacted VMWare, they say this is a hardware issue
* I have contacted SuperMicro, they say nothing really except some things and already tried and also that this could still be a software issue.
* Below the latest mini dumps. I can also provide a complete memory dump.
* We've started to have this issue aswell on our newly installed ATX pc's. These SSD's are cloned to make it easier to install. But we've also tried a fresh install (unless SSD's somehow stay corrupted from the cloning even after cleaning it completely?)
We are desperate here. The application we run luckily is sort of redundant. If a server and it's VM's on it drop, it's not such an issue, other servers will take over the load within 5 minutes, but at this rate I am required to be online all day to restart servers. Crashes happen multiple times per day.
I have a large hardware knowdledge but this goes past it, I've search on this all day for over a month trying all sorts of different things.
The fact that these motherboards are used with hosting providers on a large scale makes me suspect that the board on itself is ok. This is definately not a specific hardware issue for RMA as all 50 boards have the same symptoms. The only thing different with us is the GPU. This in combo with the Linux experiment makes me suspect that this is definately something on the PCIe lane. The GPU itself is stable on desktop mobo's. Despite it's large memory capacity this is a small GPU that does not draw much power. I would suspect the Chinese riser cards, but then again we also use SuperMicro certified risers and they show no improvement at all.
I am very desperate to find a solution here. This will start with determing the exact cause.
We are willing to pay a nice bounty to an expert who can analyse some dumps and give us more details (or even better yet, a solution).