JBOD/RAID issue?

[ktulaman]

Since last week, it has been excruciatingly slow to transfer files out of my N2100. When i encountered this when i was trying to watch a ripped DVD stored on the N2100 using my Xbox media center. I then tried to transfer the ripped DVD to my local harddrive. At times, the transfer rate was less than a few kilobytes a second, pretty much grinding to a halt. After rebooting the N2100 a few times, things appeared to be working again. I thought it was probably a good idea for me to upgrade the firmware from 2.1.01 to 2.1.03.1 (beta). I also thought it may be a good idea for me to get rid of all the files the ripped DVD share folder and recreate it.

My first attempt in removing all the files in the shared folder from the shell prompt failed. I was getting a lot of error messages telling me that the directories and files were in read-only mode, even though the permissions on the directories/files indicated otherwise. After rebooting the N2100, i managed to get rid of all the files. My upgrade to 2.1.0.3.1 was uneventful.

Unfortunately the upgrade and recreation of the shared folder did not resolve my problem. After transferring all the deleted files (about 300GB) back to the shared folder, the same issue came back. For some directories/files in the shared folder, i have no problem transferring the files. But for some directories/files, any attempt in transferring them to my local harddrive will cause the CPU on the N2100 to peak at 100% and the file transfer rate will be extremely low, if the files transfer at all. It makes no difference if i try to transfer them via SMB or AFS. I even attempted to transfer the problem directories/files to another shared folder on the N2100 via the "cp" command on shell prompt but the same thing happens.

The two Seagate HDDs in my N2100 are configured in JBOD.

Has anyone else experienced this?

Is it possible that my Seagate HDDs are too fragmented ? My next step is to transfer all the files (if at all possible) to my local HDD and then reformat the HDDs in the N2100. Hopefully, this will resolve my problem.

[marty]

So you have 300 GB under one share folder?
How many files do you have in that share folder? Do you have those files separated in several sub directories or everything is under the share folder?

[ktulaman]

Looks like i may have a drive that is failing

The default Thecus N2100 error reporting made no mention of this failing drive!!!

Is /dev/sdb the top drive or the bottom drive?

This is the smartctl output:

peanuts:/img/bin# smartctl -a -d ata /dev/sdb
smartctl version 5.33 [armv5l-unknown-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is smartmontools.sourceforge.net/


=== START OF INFORMATION SECTION ===
Device Model: ST3400633AS
Firmware Version: 3.AAD
User Capacity: 400,088,457,216 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 7
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Wed Feb 28 18:57:28 2007 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 430) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 179) minutes.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x0a0f 119 083 006 Pre-fail Always - 234172864
3 Spin_Up_Time 0x0003 097 096 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 098 098 020 Old_age Always - 2513
5 Reallocated_Sector_Ct 0x0033 096 096 036 Pre-fail Always - 174
7 Seek_Error_Rate 0x000f 081 061 030 Pre-fail Always - 158857379
9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 5918
10 Spin_Retry_Count 0x0013 097 097 097 Pre-fail Always FAILING_NOW 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 49
187 Unknown_Attribute 0x0032 001 001 000 Old_age Always - 854
189 Unknown_Attribute 0x003a 092 092 000 Old_age Always - 8
190 Unknown_Attribute 0x0022 058 037 045 Old_age Always In_the_past 60836937770
194 Temperature_Celsius 0x0022 042 063 000 Old_age Always - 42 (Lifetime Min/Max 0/29)
195 Hardware_ECC_Recovered 0x001a 056 046 000 Old_age Always - 4462907
197 Current_Pending_Sector 0x0012 001 001 000 Old_age Always - 47038
198 Offline_Uncorrectable 0x0010 001 001 000 Old_age Offline - 47038
199 UDMA_CRC_Error_Count 0x003e 200 199 000 Old_age Always - 2
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 3

ATA Error Count: 0
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 0 occurred at disk power-on lifetime: 64760 hours (2698 days + 8 hours)
When the command that caused the error occurred, the device was in an unknown state.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
9d 43 e0 51 01 00 00 Error: ABRT

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
00 9d e0 25 4b d7 cd 01 03:44:11.264 NOP [Reserved subcommand]
00 85 e0 35 c0 cd 6f 03:43:54.880 NOP [Reserved subcommand]
00 84 e0 35 47 ae cd af 03:43:54.880 NOP [Reserved subcommand]
00 1b e0 35 09 98 cd af 03:43:54.880 NOP [Reserved subcommand]
00 94 e0 35 11 85 cd bf 03:43:54.880 NOP [Reserved subcommand]

Error -1 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was in a reserved state.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
04 00 36 00 e0 51 01

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
d8 23 35 00 e0 25 33 00 41d+23:33:41.150 [RESERVED]
c8 24 31 00 e0 25 75 00 38d+22:11:51.070 READ DMA
aa 74 2e 00 e0 25 a2 b0 33d+03:12:23.710 [RESERVED]
9a 1c 2d 00 e0 25 40 58 29d+22:51:25.726 [VENDOR SPECIFIC]
89 1c 29 00 e0 25 23 00 26d+15:06:12.510 [VENDOR SPECIFIC]

Error -2 occurred at disk power-on lifetime: 5675 hours (236 days + 11 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 04 bf 96 e0 Error: UNC at LBA = 0x0096bf04 = 9879300

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
25 00 08 04 bf 96 e0 00 7d+08:31:26.270 READ DMA EXT
25 00 08 04 bf 96 e0 00 7d+08:31:24.470 READ DMA EXT
25 00 08 fc be 96 e0 00 7d+08:31:24.469 READ DMA EXT
25 00 08 f4 be 96 e0 00 7d+08:31:24.468 READ DMA EXT
25 00 08 ec be 96 e0 00 7d+08:31:24.467 READ DMA EXT

Error -3 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was in an unknown state.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
00 00 00 37 21 01 00

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
04 b9 80 02 00 40 00 2a 18:39:07.455 [RESERVED]
04 b9 80 02 00 00 08 36 18:47:51.743 [RESERVED]
fc b9 80 02 00 00 08 35 48d+22:33:41.310 [VENDOR SPECIFIC]
f4 b9 80 02 00 00 08 34 47d+09:16:43.582 SECURITY ERASE UNIT
ec b9 80 02 00 00 08 33 45d+19:59:45.854 IDENTIFY DEVICE

Error -4 occurred at disk power-on lifetime: 38590 hours (1607 days + 22 hours)
When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
51 01 01 00 00 37 21 Error: UNC, IDNF, AMNF at LBA = 0x01370000 = 20381696

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
40 25 4a 9d 80 02 00 e0 12d+10:15:48.992 READ VERIFY SECTOR(S)
00 25 23 96 80 02 00 e0 00:00:07.384 NOP [Reserved subcommand]
00 25 b1 8e 80 02 00 e0 00:00:05.344 NOP [Reserved subcommand]
00 25 68 87 80 02 00 e0 00:00:03.304 NOP [Reserved subcommand]
00 25 57 87 80 02 00 e0 00:00:01.264 NOP [Reserved subcommand]

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[omega]

Correct me if I'm wrong but the N2100 is using a ext2/ext3 filesystem for the raid, isn't it?

Then fragmentation is not a real issue as these filesystems are handling fragmentation pretty well. At least it is unlikely that this would explain the bad performance.

And even if you have all the 300 GB in one folder, for bulk data transfer this is also not a real issue.

From what you're telling I suspect your network connection. What is your network speed?

Are you able to log into your N2100 via SSH?
If yes, just make a simple test by writing a 1 GB testfile to your RAID.
Just go into one of your share directories and try this command:

time dd if=/dev/zero of=testfile bs=64K count=16000

You'll see the elapsed time needed to write a 1 GB file then.
After the test just remove the file by "rm testfile".

Andreas

[omega]

Yeah, that's the reason for the extremely low speed.....

Did you try to initiate a smart self test (smartctl -t)? Don't know if that helps in any way but I would try it....

From what the smart output shows, your drive is having read errors.

For the N5200 I could tell you how to identify the failed drive, but I don't own a N2100.... sorry.

But maybe you can identify the drive by its temperature, it is 47 degrees and the good one is supposed to have max. 29 degrees. Check out the actual temperature by checking the smart values for /dev/sda.

Andreas

[ktulaman]

Thanks! Running the long test now and i should know a little more in less than 3 hours:

smartctl -t long -d ata /dev/sdb

I got the serial number off the failing drive from "smartctl -a -d ata /dev/sdb" so it should not be difficult to verify against the number on the disk drive itself.