I posted on another thread but after further basic testing decided to put it in separate thread as i am not sure where the problem is.
Below was ordered in march 2019 and haven't been setup or fully used yet.
rockpro64 4gb
nas case with 12v 5a psu and fan
Sata pcie card ASM1062 Serial ATA Controller (rev 01)
sd card imaged with stretch-openmediavault-rockpro64-0.7.9-1067-arm64.img
Main problem is when i first plugin power connector to rockpro64 it automatically starts to boot from sd card (which is good) but once os boots hard drives are not showing up in omv and it takes much longer to boot. I have to reboot at least once for hard drives to show up. It also throws a lot of errors when trying to copy a number of files.
My findings:
After first boot no disks are seen but i hear hard drives starting to spin up 2x 4TB WD red, i tried with only 1 drive but the same problems so power shouldn't be an issue.
If i execute "halt -p" when disks are not seen in omv i can still hear them spinning when rockpro64 power on light is off.
During reboot disks do not stop spinning
Based on the above it looks like sata card or whatever controls disk recognition need to be reconfigured to add delay but i do not know how
Before i go and start purchasing different sata cards (they are not cheap here, around 40-50 euro for marvel 88se9215) i would like to know is it a problem with sata card or time it takes for hard drives to spinup and start talking to sata card. I would also like to know if asmedia 1602 rev01 is stable and performing well for others.
Below are the errors i see when copying some files to disk
[ 80.509588] ata1: SATA link down (SStatus 1 SControl 310)
[ 80.534423] ata1: irq_stat 0x00000040, connection status changed
[ 80.540057] ata1: SError: { CommWake DevExch }
[ 80.545230] ata1: limiting SATA link speed to 1.5 Gbps
[ 80.550431] ata1: hard resetting link
[ 212.084097] ata2: EH complete
[ 212.097400] ata2: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[ 212.106333] ata2: irq_stat 0x00000040, connection status changed
[ 212.110266] ata2: SError: { CommWake DevExch }
[ 212.114038] ata2: limiting SATA link speed to 1.5 Gbps
[ 212.117866] ata2: hard resetting link
[ 214.326623] ata2: SATA link down (SStatus 1 SControl 310)
05-28-2019, 02:18 PM (This post was last modified: 05-28-2019, 03:00 PM by Nikolay_Po.)
Are the metal parts of the HDD cases connected to board ground? Somewhere was a notice that without ground connection between the HDD case and board GND there may be a lot of errors. I've ordered (23 of May) ROCKPro64 and the same ASMedia SATA adapter as yours but without of a case. Still awaiting, the delivery will be not quite fast. I will try too.
There may be several places where an error may arise. Besides the software problem and overheating, the signal integrity may be the cause of an errors.
Try to:
Connect DC IN ground at DC iN socket to the metal case. And ensure the drives have a contact to the case, metal by metal. Add a wire or sand of the paint/coating from metal parts of the case to ensure good electrical contact. As a quick test you may connect RJ-45 shield to the NAS case by shortest wire possible.
Take a wire and try to additionally connect the lower screw of SATA card to the ground of ROCKPro64. You may connect this wire to metal shield of RJ-45 or USB connector. The best is to connect to ground contact of DC IN. The is a GND mark at DC IN socket ground pin. It is between the DC socket and aluminum can capacitor. The aim is to connect the controller board ground to the same ground where the HDDs are supplying.
Change SATA wires.
After the heating was not confirmed I'm suspecting the signal integrity problem. I'm wondering the SBC board has insulated mounting openings. So there may be none ground contact to the metal case as it is in case of traditional PCs or laptops.
The absence of a good ground contact allows the noise to be induced between the case (the drives) and the board because there is none good direct contact between the board and the case.
The disk drives requiring more power than SSDs we saw on demo video of a good man with a big glasses. That is why supply converters on HDD supply wires may produce more noise being more loaded. And this noise voltage may become too high to be suppressed by SATA differential pair interfaces, both HDD and a controller.
The good ground contact is crucial. Even if Ohmic (Galvanic) contact is good and your Ohm-meter shows below 1 Ohm between the SBC and HDD, the wire loops and power converters may induce high frequency noise on a wires and even on PCB traces.
Good practice for digital circuits is to connect the board ground to the chassis extensively, at each mounting screw. You may see this approach in any desktop or laptop PC. Again, I'm wondering why Pine64 PCB is different.
(05-28-2019, 02:18 PM)Nikolay_Po Wrote: Are the metal parts of the HDD cases connected to board ground? Somewhere was a notice that without ground connection between the HDD case and board GND there may be a lot of errors. I've ordered (23 of May) ROCKPro64 and the same ASMedia SATA adapter as yours but without of a case. Still awaiting, the delivery will be not quite fast. I will try too.
There may be several places where an error may arise. Besides the software problem and overheating, the signal integrity may be the cause of an errors.
Try to:
Connect DC IN ground at DC iN socket to the metal case. And ensure the drives have a contact to the case, metal by metal. Add a wire or sand of the paint/coating from metal parts of the case to ensure good electrical contact. As a quick test you may connect RJ-45 shield to the NAS case by shortest wire possible.
Take a wire and try to additionally connect the lower screw of SATA card to the ground of ROCKPro64. You may connect this wire to metal shield of RJ-45 or USB connector. The best is to connect to ground contact of DC IN. The is a GND mark at DC IN socket ground pin. It is between the DC socket and aluminum can capacitor. The aim is to connect the controller board ground to the same ground where the HDDs are supplying.
Change SATA wires.
After the heating was not confirmed I'm suspecting the signal integrity problem. I'm wondering the SBC board has insulated mounting openings. So there may be none ground contact to the metal case as it is in case of traditional PCs or laptops.
The absence of a good ground contact allows the noise to be induced between the case (the drives) and the board because there is none good direct contact between the board and the case.
The disk drives requiring more power than SSDs we saw on demo video of a good man with a big glasses. That is why supply converters on HDD supply wires may produce more noise being more loaded. And this noise voltage may become too high to be suppressed by SATA differential pair interfaces, both HDD and a controller.
The good ground contact is crucial. Even if Ohmic (Galvanic) contact is good and your Ohm-meter shows below 1 Ohm between the SBC and HDD, the wire loops and power converters may induce high frequency noise on a wires and even on PCB traces.
Good practice for digital circuits is to connect the board ground to the chassis extensively, at each mounting screw. You may see this approach in any desktop or laptop PC. Again, I'm wondering why Pine64 PCB is different.
I did a continuity test between sbc mounting bolts and hdd case and it is good. There are mounting posts in the nas case that you screw sbc to. See below youtube video and forward to 1 minute. https://www.youtube.com/watch?v=_UeeklKo0Og
As i tried with single hard drive and still get the same issues it has to do something with timing for drive to spinup.
What you listed regarding ground may help with errors during copying of the files but looking at it as none electronics specialist, connector that has 4 pins on pcb close to dc jack feeds the drives with power, yes there is a converter of some type in between to reduce 12v to 5v or 3v but i would think ground should be solid.
I am using sata cables provided by pine64 but i had to bend them a bit at the connectors on pci card and hdd side as otherwise they either get in contact with case fan or with case cover. I will try to look around for spares at home and do tests to exclude those.
One thing to note is that i added zalman fanmate 2 that i had laying around to this 12 feed as i couldn't control case fan with cpu temperatures, it didn't make sense.
I got mine in 20 days since placing the order but it all depends what time and where and customs.
(05-28-2019, 05:09 PM)vecnar Wrote: I did a continuity test between sbc mounting bolts and hdd case and it is good.
Have you a network analyzer up to GHz range? The noise that can disturb SATA operation is high frequency. And usual continuity testers doesn't work at high frequencies. They are DC usually and can't help gere.
The case has metal mounting posts. But has the SBC a metal contacts at these posts? Have the case posts an electric contact with SBC ground?
Quote:As i tried with single hard drive and still get the same issues it has to do something with timing for drive to spinup.
What you listed regarding ground may help with errors during copying of the files but looking at it as none electronics specialist, connector that has 4 pins on pcb close to dc jack feeds the drives with power, yes there is a converter of some type in between to reduce 12v to 5v or 3v but i would think ground should be solid.
I can't say about the timings.
Saying about good contacts you're saying about a DC path. Yes, it is good. But long black wires of DC supply can't forward high frequency energy good enough. Additional contacts are needed. That is why the motherboard in generic PC is screwed by many screws and the mounting holes of PCB are metalized.
Anyway try to place a link between SATA controller board screw and an RJ-45 shield just to check will it change something. Also place SATA cables togeter with power that manner the look like a common single cable.
If nothing above helps, the problem is definitely a software problem
(05-28-2019, 05:09 PM)vecnar Wrote: I did a continuity test between sbc mounting bolts and hdd case and it is good.
Have you a network analyzer up to GHz range? The noise that can disturb SATA operation is high frequency. And usual continuity testers doesn't work at high frequencies. They are DC usually and can't help gere.
The case has metal mounting posts. But has the SBC a metal contacts at these posts? Have the case posts an electric contact with SBC ground?
Quote:As i tried with single hard drive and still get the same issues it has to do something with timing for drive to spinup.
What you listed regarding ground may help with errors during copying of the files but looking at it as none electronics specialist, connector that has 4 pins on pcb close to dc jack feeds the drives with power, yes there is a converter of some type in between to reduce 12v to 5v or 3v but i would think ground should be solid.
I can't say about the timings.
Saying about good contacts you're saying about a DC path. Yes, it is good. But long black wires of DC supply can't forward high frequency energy good enough. Additional contacts are needed. That is why the motherboard in generic PC is screwed by many screws and the mounting holes of PCB are metalized.
Anyway try to place a link between SATA controller board screw and an RJ-45 shield just to check will it change something. Also place SATA cables togeter with power that manner the look like a common single cable.
If nothing above helps, the problem is definitely a software problem
Hello Nikolay,
By continuity i meant small resistance i had on my multimeter which was done when everything was off. I do not have network analyzer.
Sbc doesn't have metal contacts on where screws attach. The only link between case and sbc was pcie bracket screw. So the link between sbc and hdd cages is through power cables, sata cables and sata controller mounting bracket.
Strangely enough today i didn't have problems when writing to disks, i only had problems on first startup when power is connected first time and even one reboot didn't fix at times, possibly because i removed hdd cage and all cables where moved. Write speeds were from disk to disk were from 40-60 MB/s on large files.
While i had it opened i changed sata cables that i had laying around, they looked a bit old with 90 degrees angle on one side (sata cables have to be bent close to contact points in the case on both sides). I also connected sata screw to a wire and wire to ground cable that feeds one of the drives (it is connected to ground pin on pcb).
You mentioned putting sata cables and power cables in one bunch but is it really a good idea to put power cables that have converter in them with data cables? Or should I place them as far apart as possible as it has voltage converter behind shrink wrap?
Sometimes when unplugging power plug from socket and holding power button to discharge the current didn't help to reproduce the problem, I assume it is because some current is stored in capacitors as leaving it for longer and coming back is reproducible.
I think i have armbian on another sd card that i will try and see if i get the same results.
(05-29-2019, 01:12 PM)vecnar Wrote: Hello Nikolay,
By continuity i meant small resistance i had on my multimeter which was done when everything was off. I do not have network analyzer.
Because you have none network analyzer you can't guess wither high freqency ground network conductivity is enough or not.
When integrating the board into the case, the developer need to ensure good ground contacts. It seems to me Pine64 efforts were not enough here.
Quote:Sbc doesn't have metal contacts on where screws attach. The only link between case and sbc was pcie bracket screw. So the link between sbc and hdd cages is through power cables, sata cables and sata controller mounting bracket.
This is called a "ground loop" and ia bad because makes the setup sensitive for electromagnetic interferences.
Ground loops should be as small as possible. To achieve that the chassis grounding should be performed at each interface or supply inlet.
I recommend you to try to tie all available ground surfaces of a board to the chassis.
Quote:You mentioned putting sata cables and power cables in one bunch but is it really a good idea to put power cables that have converter in them with data cables? Or should I place them as far apart as possible as it has voltage converter behind shrink wrap?
The SATA cable is shielded. Ts will not suffer from convertrer noise. But placing cables together you will decrease the ground loop cross-section thus will decrease inductance and high frequency impedance. It may make your sertup less suspectable to EMI.
Also when testing look into the logs for SATA errors. Thi interface may work in peesence of errors.
(05-29-2019, 01:12 PM)vecnar Wrote: Hello Nikolay,
By continuity i meant small resistance i had on my multimeter which was done when everything was off. I do not have network analyzer.
Because you have none network analyzer you can't guess wither high freqency ground network conductivity is enough or not.
When integrating the board into the case, the developer need to ensure good ground contacts. It seems to me Pine64 efforts were not enough here.
Quote:Sbc doesn't have metal contacts on where screws attach. The only link between case and sbc was pcie bracket screw. So the link between sbc and hdd cages is through power cables, sata cables and sata controller mounting bracket.
This is called a "ground loop" and ia bad because makes the setup sensitive for electromagnetic interferences.
Ground loops should be as small as possible. To achieve that the chassis grounding should be performed at each interface or supply inlet.
I recommend you to try to tie all available ground surfaces of a board to the chassis.
Quote:You mentioned putting sata cables and power cables in one bunch but is it really a good idea to put power cables that have converter in them with data cables? Or should I place them as far apart as possible as it has voltage converter behind shrink wrap?
The SATA cable is shielded. Ts will not suffer from convertrer noise. But placing cables together you will decrease the ground loop cross-section thus will decrease inductance and high frequency impedance. It may make your sertup less suspectable to EMI.
Also when testing look into the logs for SATA errors. Thi interface may work in peesence of errors.
Hello Nikolay,
Thank you for the above information. I have bundled sata and power cables as much as i could. I tried to reproduce it on armbian but couldn't, left off overnight and plugged in the afternoon and no problems, no errors in dmesg output regarding ata. But after changing sd card with omv on it i couldn't get them recognized, i had to reboot twice. I was getting below errors sames as before
Code:
[ 7.461809] ata2: SATA link down (SStatus 1 SControl 300)
[ 7.476399] ata2: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[ 7.483279] ata2: irq_stat 0x00000040, connection status changed
[ 7.489419] ata2: SError: { CommWake DevExch }
[ 7.495414] ata2: hard resetting link
I will try to reproduce on armbian to be sure it is software and not hardware.
Regarding grounding points on sbc, you meant connect cable to rj45 metal shield, 1 hdmi and 2 usb shields and wrap wire around the bolts that hold sbc to the case or just pcie bracket screw?
Also what is the easiest way to attach cable to flat metal surfaces on sbc shields?
05-30-2019, 01:20 PM (This post was last modified: 05-30-2019, 02:00 PM by Nikolay_Po.)
(05-30-2019, 11:48 AM)vecnar Wrote: Thank you for the above information. I have bundled sata and power cables as much as i could. I tried to reproduce it on armbian but couldn't, left off overnight and plugged in the afternoon and no problems, no errors in dmesg output regarding ata.
Can you confirm that there is none difference for armbian how the cables are placed and attached? Does armbian operation reacts for HDD case connecting or disconnecting from a chassis?
Quote:But after changing sd card with omv on it i couldn't get them recognized, i had to reboot twice. I was getting below errors sames as before
Code:
[ 7.461809] ata2: SATA link down (SStatus 1 SControl 300)
[ 7.476399] ata2: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[ 7.483279] ata2: irq_stat 0x00000040, connection status changed
[ 7.489419] ata2: SError: { CommWake DevExch }
[ 7.495414] ata2: hard resetting link
Quick look into SStatus register description, SStatus=1 means "Device presence detected but Phy communication not established". This is an information (a SStatus register read) from the cnotroller. It is hardware status. The link just completely is gone. The signal is present (device detected), but interface information is destroyed too badly to keep SATA communication up.
You need to check somehow at which SATA speed operates armbian and at which OpenMediaVault.
It may be that one is trying to operate at higher speed than another. And higher speed fails.
It is hard to say anything without of seeing full log about SATA initialization.
Quote:I will try to reproduce on armbian to be sure it is software and not hardware.
When hardware fails using one distributive and not using another, it may be still hardware problem. One need to know all details like interface speeds and initialization sequences to judge. Of course it may be software problem. But SATA is rather well established protocol and it uses well established source codes. It is harder imagine there is software bug rather than hardware.
Quote:Regarding grounding points on sbc, you meant connect cable to rj45 metal shield, 1 hdmi and 2 usb shields and wrap wire around the bolts that hold sbc to the case or just pcie bracket screw?
Also what is the easiest way to attach cable to flat metal surfaces on sbc shields?
The best is to solder a wire to the minus of DC socket (at the top side of board) then to squeeze this wire (shorter is better) by the SATA card bottom screw. At the bottom side to solder another wire to the minus and squeeze this wire by a mounting screw to the mounting pin of the chassis. This will tighten the grounds of the components together into one point (DC socket minus) with a good contact.
If you can't or wish not to solder, just ensure that Ethernet socket have good contact and try to place a clamp filters on the wires. Something like ZCAT20D from TDK on SATA interface cables. See clamp filter catalog for a reference.
And... It may turns out to be software problem. So don't be afflicted too much after following my advices if follow.
(05-30-2019, 11:48 AM)vecnar Wrote: Thank you for the above information. I have bundled sata and power cables as much as i could. I tried to reproduce it on armbian but couldn't, left off overnight and plugged in the afternoon and no problems, no errors in dmesg output regarding ata.
Can you confirm that there is none difference for armbian how the cables are placed and attached? Does armbian operation reacts for HDD case connecting or disconnecting from a chassis?
Quote:But after changing sd card with omv on it i couldn't get them recognized, i had to reboot twice. I was getting below errors sames as before
Code:
[ 7.461809] ata2: SATA link down (SStatus 1 SControl 300)
[ 7.476399] ata2: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen
[ 7.483279] ata2: irq_stat 0x00000040, connection status changed
[ 7.489419] ata2: SError: { CommWake DevExch }
[ 7.495414] ata2: hard resetting link
Quick look into SStatus register description, SStatus=1 means "Device presence detected but Phy communication not established". This is an information (a SStatus register read) from the cnotroller. It is hardware status. The link just completely is gone. The signal is present (device detected), but interface information is destroyed too badly to keep SATA communication up.
You need to check somehow at which SATA speed operates armbian and at which OpenMediaVault.
It may be that one is trying to operate at higher speed than another. And higher speed fails.
It is hard to say anything without of seeing full log about SATA initialization.
Quote:I will try to reproduce on armbian to be sure it is software and not hardware.
When hardware fails using one distributive and not using another, it may be still hardware problem. One need to know all details like interface speeds and initialization sequences to judge. Of course it may be software problem. But SATA is rather well established protocol and it uses well established source codes. It is harder imagine there is software bug rather than hardware.
Quote:Regarding grounding points on sbc, you meant connect cable to rj45 metal shield, 1 hdmi and 2 usb shields and wrap wire around the bolts that hold sbc to the case or just pcie bracket screw?
Also what is the easiest way to attach cable to flat metal surfaces on sbc shields?
The best is to solder a wire to the minus of DC socket (at the top side of board) then to squeeze this wire (shorter is better) by the SATA card bottom screw. At the bottom side to solder another wire to the minus and squeeze this wire by a mounting screw to the mounting pin of the chassis. This will tighten the grounds of the components together into one point (DC socket minus) with a good contact.
If you can't or wish not to solder, just ensure that Ethernet socket have good contact and try to place a clamp filters on the wires. Something like ZCAT20D from TDK on SATA interface cables. See clamp filter catalog for a reference.
And... It may turns out to be software problem. So don't be afflicted too much after following my advices if follow.
Hello Nikolay,
I didn't open the box or move the cables, just took sd card from outside the case, had to tilt it slightly but not opening it or touching any cables. if i move hdd case all cable connections will be affected so it may affect it but i haven't tried it yet.
I tried one thing a few times and i can constantly reproduce it so i wouldn't want to open the case now. I found that after booting to armbian all is fine, halt -p and power off device from socket, change sd card with omv and it doesn't show disks, it needs 2 reboots or 2 times to halt -p (as i use watt meter i could see that during shutdown state at these stages it was consuming 6-7 watts where if it shuts down normally it consumes 1-2 watts) and use turn on button for disks to be seen again.
Armbiand disk transfer speeds around 75 Mb/s on large file between disks where in omv i was getting around 60 Mb/s
I have dmesg outputs from both armbian and omv. One post has limit of 65k characters so i uploaded them to uguu.se, the links are valid for 24 hours https://a.uguu.se/moY4l8LGYpEq_ArmbianDmesg.txt https://a.uguu.se/mghdEbXWg0gQ_OMVdmesgOutput.txt
I have a small gas soldering iron but not sure if i am comfortable doing it. I will probably destroy dc socket by soldering around it as there is no access to pins from top at least i didn't see on pictures.
Finding clamps for cables is time consuming, waiting for deliveries from abroad.
Should i just take everything out of the box and place on cardboard and test or it will not help?
05-31-2019, 12:16 PM (This post was last modified: 05-31-2019, 01:13 PM by Nikolay_Po.)
Do not solder the board. It is very tiny, many small components that is too easy to damage.
First of all it need to compare the logs and find the difference to get a clue about problem. I can solder mine boards just because I have a soldering station and enough experience. So I can eliminate possible hardware problem to examine a software problem. But in your case is better to examine (and exclude if not actually present) a software problem first. Solder the wires only after we define the hardware problem is definitely persist. More testing, probably of more users, more boards needed. Don't damage your board right now.
Have attached your logs for history.
There is no difference in logs until file system initialization starts:
Armbian (good):
Code:
[ 10.306823] systemd[1]: Starting Create Static Device Nodes in /dev...
[ 10.310112] systemd[1]: Starting Load/Save Random Seed...
[ 10.315855] systemd[1]: Starting udev Coldplug all Devices...
[ 10.320900] systemd[1]: Mounting FUSE Control File System...
[ 10.327677] systemd[1]: Mounting Configuration File System...
[ 10.333960] systemd[1]: Starting Apply Kernel Variables...
[ 10.346053] systemd[1]: Mounted Configuration File System.
[ 10.349502] systemd[1]: Mounted FUSE Control File System.
[ 10.354745] systemd[1]: Started Nameserver information manager.
[ 10.359864] systemd[1]: Started Load/Save Random Seed.
[ 10.365358] systemd[1]: Started Create Static Device Nodes in /dev.
[ 10.370737] systemd[1]: Started Apply Kernel Variables.
[ 10.389286] systemd[1]: Starting udev Kernel Device Manager...
[ 10.476560] systemd[1]: Started udev Kernel Device Manager.
[ 10.535139] systemd[1]: Started Set the console keyboard layout.
[ 10.538736] systemd[1]: Reached target Local File Systems (Pre).
[ 10.548600] systemd[1]: Mounting /tmp...
[ 10.560632] systemd[1]: Mounted /tmp.
[ 10.565920] systemd[1]: Started Journal Service.
[ 10.621985] systemd-journald[310]: Received request to flush runtime journal from PID 1
OMV (bad):
Code:
[ 4.668895] systemd[1]: Mounting FUSE Control File System...
[ 4.677607] systemd[1]: Mounting Configuration File System...
[ 4.683865] systemd[1]: Starting Create Static Device Nodes in /dev...
[ 4.691704] systemd[1]: Starting Initial Check File System Quotas...
[ 4.700326] systemd[1]: Starting udev Coldplug all Devices...
[ 4.706770] systemd[1]: Starting Load/Save Random Seed...
[ 4.714409] systemd[1]: Starting pNFS block layout mapping daemon...
[ 4.727700] systemd[1]: Mounted FUSE Control File System.
[ 4.737374] systemd[1]: Mounted Configuration File System.
[ 4.741549] systemd[1]: Started Journal Service.
[ 4.786973] systemd-journald[337]: Received request to flush runtime journal from PID 1
[ 4.822384] ata1: SATA link down (SStatus 1 SControl 300)
[ 4.826507] ata1: exception Emask 0x10 SAct 0x0 SErr 0x4040000 action 0xe frozen t4
[ 4.830837] ata1: irq_stat 0x00000040, connection status changed
[ 4.834850] ata1: SError: { CommWake DevExch }
[ 4.838688] ata1: hard resetting link
The problem is arising right at the start of file system operation. The start sequences are differ.
Had never investigated SATA problems before. Need to look at exception mask bits of ATA library and SATA Error register of a controller.
