HomeХобби и стильRelated VideosMore From: Lawrence Systems / PC Pickup

So How Hard Is It To Crash and Kill a FreeNAS 11 ZFS Raid Z1 Array?

805 ratings | 50629 views
Amazon Affiliate Store https://www.amazon.com/shop/lawrencesystemspcpickup Things we love including computers, software, services, gadgets, and of course hot sauces. https://www.lawrencesystems.com/things-we-love/ We spent some time trying to corrupt a FreeNAS ZFS Raid Z1 3 drive array to see just how hard it would be to corrupt the data on the drives.... https://www.lawrencesystems.com/ 14140 Pennsylvania Southgate MI 48195 313-299-1503 [email protected] https://twitter.com/TomLawrenceTech https://www.lawrencesystems.com/ 313-299-1503 [email protected] https://twitter.com/TomLawrenceTech
Html code for embedding videos on your blog
Text Comments (104)
Lino (16 hours ago)
Try pulling off a sata cable on a raidz1 or raidz2 setup and plug it back in after reboot. This is what killed my freenas pools. A bad sata cable decided to die on me. Resilvering happened after resilvering. See if your system recovers from such procedures.
kingneutron1 (3 days ago)
Test is incomplete without running a full zpool scrub and having it come up with no errors
graceoverall (30 days ago)
I didn't hear anything about the actual data integrity. Just booting up is not sufficient to prove that the actual data is in tact. :(
It does an integrity check on mount. Data was all there
Ewen Chan (1 month ago)
https://www.youtube.com/watch?v=6xHu1fmFeRo
Ewen Chan (1 month ago)
If you really want to corrupt it, set up a strip ZFS pool/array and then pull the power plug and then unplug one of the drives, start the system back up and plug the drive back in.
ohthecake (2 months ago)
Was this using ECC ram?
No
OpenGL4ever (3 months ago)
@Lawrence Systems / PC Pickup I have never seen something stupid as that. If you want corrupt a working memory, then use a radioactive probe of a beta or gamma emitter and place it directly close on the RAM.
Raphael Ferreira (3 months ago)
Yeah... Apply a sparking live wire to your motherboard and, yeah, you’ll fry it. (Why is it that this video reminded me of those Jackass videos where the guys spent hours smashing their balls in ever most creative ways...?)
Arpad Muller (3 months ago)
I would test it while writing data into the pools, because ZFS uses ARC. So there is a state when the PC get your data from network, storing in memory temporary, and ACK you back. If you shoot in the head in this moment, there is no time to sync to disks. That means you have an ACK to you data (ok I have stored), but actually sits int the memory and lost.
CrankyCoder (3 months ago)
I would like to see an updated version of this. But with something bigger. Like have 6 drives in a raidz2 and pull 3 out. Then put them back and see what happens. I would like to see how the system works for something like that. Example, if you have an external drive enclosure (e-sas) with a bunch of drives, and something happens to it (power, ect) but you can restore power and the drives are all there. would the system eat itself? or would it recover?
malayaleeking (3 months ago)
Next, we should try pouring water into the oil tank to see how long it takes the engine to seize.
Chris S (5 months ago)
I managed to crash FreeNAS on a hp server. If you change the options under the options field of a link aggregate member then it can crash. Nothing until a manual reset of the system.
Ron Laws (5 months ago)
if this was BTRFS it would have failed before you even hit record.
Elvio Jr. (6 months ago)
Excellent video. Do you know any software that audits files in the freenas?
QuadroNVS (7 months ago)
its nice to know that ZFS is a beast. My file backup server is runs FreeNAS 9.3(very old, i know) but I have never had an issue with the file system itself however if a drive were to be failing or have some kind of issue it makes the entire server misbehave. I know this because I have had a 2TB array quit in the middle of a file transfer and the server web interface was no longer accessible yet the server was still pingable. Windows would have just spit some errors and kept on trucking. I still love FreeNAS, it so powerful, I just need to learn the in's and outs so I can convert all of my servers over to FreeNAS.
Melvin Chong (9 months ago)
How about a power outage when a UPS is in place ? Will FreeNAS automatically shuts down after completing the writing from cache to disks ?
mdd1963 (1 year ago)
Data integrity is of paramount importance....unless we are paying for ECC RAM, then , not so much! :)
mdd1963 (1 year ago)
Electrocuting the motherboard for views.... :/
Edvin Olofsson (1 year ago)
That's amazing! Thanks for your sacrifice
Bob Dole (1 year ago)
Somewhere, an electrical engineer with a real job is punching himself in the balls.
Steve Bez (1 year ago)
"yell glitch"adds tree cull port tall store reads, Granvilles .stockings'sed,injun,eer...??Datey cakey.
Kenneth Nicklowicz (1 year ago)
you all are much like me! this was reassuring as I Have been using free nas sense version 8 and love knowing I wont loose any data :) Thanks for this.. made my day
Artemia Salina (1 year ago)
LOL "We're gonna science this motherboard."
blastman8888 (1 year ago)
I have been runnng a 10TB raid Z1 for about 7 years I have had to replace drives that fail even replace one then a 2nd one after it rebuilt never had any data loss I know it's possible to lose more then one drive but been lucky enough haven't had it happen.
Melvin Serrano (1 year ago)
I have a quick question, I have a few HDD's that have bad sectors on them, would they work on a FreeNAS server like this one? Thanks.
Sasquatch (1 year ago)
try corrupted(non ECC) ram and read 10+GB file from it pool, unrecoverable fail, Zvol won't import , Pool recovery impossible. Note here system will run happily for months with faulty ram and will fail only on huge file read, writing same huge file worked fine, working with files below 1GB is fine too. Found out hard way, luckily had backup 2 days before crash. So your test was....staged?
Brisys Consulting (1 year ago)
Easiest way to corrupt a Freenas ZFS array? Building them with cheap mutha-fudging power supplies!!!
David Grishko (6 months ago)
Yep
You should try to disconnect one drive, then start the process of array rebuild with "new" clean drive and try to corrupt the process of rebuilding the array (by removing memory, disabling power etc). Anyway, great video!
James Monfregola (1 year ago)
Excellent video! We use a lot of FreeNAS in the field and they are definitely hardy boxes.
Justin Grazulis (1 year ago)
dd'd a few drives for a few minutes, popped 10 out of 12 drives into a completely different build and freenas still was able to recover the array, at that point, I was sold on ZFS
Jay P (1 year ago)
I don't think freezing the system is much of a worrry for ZFS. The real concern is flipped bits writing the wrong data for a period of time going unnoticed. Crashing the system like you are isn't likely to cause the problems a flipped bit does. Still fun though haha.
John Castillo (1 year ago)
One thing that has happen was running FreeNAS with a duo-core mobo and 80Gb ide pimary drive & 4 1.5Tb data drives. Running Plex Media Server extension where the data partition got full and crashed the entire system. BUT the data/movies did not get corrupted. Had to reinstall FreeNAS to recover access to drives and had to rebuild the raid setup since the drives got swapped in drive table of new Setup of FreeNAS. no data loss yet. Found out the PMS ext was the fault of the overload in the data partition and since move the Metadata storage for PMS to a regular data drive.
Orfescu Horatiu (1 year ago)
if you had redundant memory :)) !!!!
aymen ali (1 year ago)
I had hdd lost in it 😭😭😭because the sA/C it drops water on the board it's restarting for hours the nothing's pair z1
aymen ali (1 year ago)
Only motherboard forums.freenas.org/index.php?threads/motherboard-dead.49539/
aymen ali (1 year ago)
Only motherboard forums.freenas.org/index.php?threads/motherboard-dead.49539/
The hard drive or the motherboard got wet?
remle ikawura (1 year ago)
scrub the drive using a bad memory, that sure will corrupts the filesystem.
Sasquatch (1 year ago)
scrubbing works fine with upper 1/4 of memory dead. reading big 10GB+ files with bad memory kills pool
Bo Diddly (1 year ago)
I agree with +remle ikawura. I have had RAM modules that I would call "Flaky", that ruined data by sometimes working and sometimes not work completely. On more than one occasion, the OS or software would randomly experience mysterious activity and a (chkdsk, e2fsck) would always find corrupted data. When running a memtest, sometimes the memory would pass, sometimes it would pass some parts of the test. This memory would fail in such a way that the OS doesn't detect it because the OS checks don't happen often enough or do not log/compare re-occurrences of an error that comes and goes. So, the OS continues to write possibly corrupted data to the HDs and it is not noticed until there is a mysterious failure.
remle ikawura (1 year ago)
nah, removing the memory modules will make the system hang while a bad memory keeps the system at least running while ruining the file system. This is by my experience, I have a getho core2quad freenas back then and the array gone kaput while resilvering the array and guess what, one the ram sticks is bad. RIP files.
+remle ikawura we had a scrub running while we yanked out the memory, takes more than that.
SkiLoveRock Beats (1 year ago)
Similar logo to Linus Tech Tips, probably wasnt on purpose, just fyi! The more you know (specifically the WAN show)
SkiLoveRock Beats (1 year ago)
Then again I do graphic design so im paying a lot of attention to logos :/ Thanks for the quick response and Ill definitely be watching more of your videos!
SkiLoveRock Beats (1 year ago)
A long time ago, I just started watching your channel and that was my first impression just fyi. (Specificaly LTTs B role set and WAN show logos) PS: I love all of your videos ive seen so far, nothing against your content at all just noticed the similarity in your logo and color scheme!
That has been our logo and shirts for years, I watch Linus, but I don't pay much attention to his wardrobe. When did he start wearing them?
Demetrius Cassidy (1 year ago)
This plays out like one of those horror movies, but with PCs.
Steve Bez (1 year ago)
Yeah if you Agatha Kriz tea retro micro Hitchco psyco crows like MacOS System simpling 'bell life'belief Ed of simple actions micro mac arent relayed edp in word,just lwrote email message too win10 from system4' tape decnet Qnap QuantumnSnap scsi greeting too early HP A600 fiber net for diverse exclamation mark,but it snowed, offering no further interferences,eh,screwdriver,cocktails, Blair Witch kennedy dead trillion Trump tens virt node spring simpson addictions,why,MGM...
newSogn (1 year ago)
So pulling out the memory doesn't destroy it yet it's the worst mistake to not use ecc memory? Like I get why you SHOULD use ecc but this dude just pulled it out while it's on and it still recovered
Sasquatch (1 year ago)
pulling memory most likely instantly reset/hang up CPU. bad memory will feed bad data into your file system. I had bad non-ecc memory running like a champ until I tried reading 500GB file. it killed my file system.
12345678987654321dog (1 year ago)
ECC is just more of a precaution. Like wearing a seatbelt. Sure you'll survive many crashes without wearing one but you never know when it'll be that one crash where you fly through the windshield.
Yup, we did it many times over and over.
Brian Fraser (1 year ago)
overclock the sata bus....
Crosstalk Solutions (1 year ago)
Great video Tom!
timarc124 (1 year ago)
A suggestion for another video : How to install phpvirtualbox on FreeNAS 11
Davy Laronne (1 year ago)
That was so coooools
Brett Passmore (1 year ago)
map an iscsi connection and run a VM on it. While that VM is running, see it is un corrupted. Be a great test.
Steve Bez (1 year ago)
During VM use active directories,port samba, run access client term services on all disks in use,then swap out too other netw pooling for extensive uses an tests ,p.c\servers multimedia ,protocols,smp traps,dns attacks, metasploit?
Randall Powell (1 year ago)
Great Demo. OT question: What do you use for Audio and Video? Your videos are very clear and crisp.
Thanks! Canon 70D and a Rode Video Mic Go I need to do an updated video, but it's mostly the same gear I have in this video. https://www.youtube.com/watch?v=urTTuZArhlQ
Nicolai Syvertsen (1 year ago)
Pure science coming out of those red wires.
Olaf (1 year ago)
Hi Can you make one more test: Install Freenas on new PC and import harddrives in Raid 1 or 5?. WIll the data survive? I mean swapping hard drives to new PC with new operating system.
Sasquatch (1 year ago)
it will survive if you add HDD's in correct order. Done it with 4 hdd array, swaping 2 hdd's will make your pool fail to import, if you swap them back it will work, at least in RaidZ1
Kaushal Beladiya (1 year ago)
Kindly share briefly configuration for this setup.
TNBigHorn (1 year ago)
WOW... I'm convinced !!!
francesco nandi (1 year ago)
Taking off ram stick when oh.. Heart Attack.
smileymattj (1 year ago)
Do more attacks on the disks. RAID is all about the disks failing. I think attacking the memory is showing the stability of FreeBSB. - Unplug a drive, and reconnect it; (sata and power, 2 tests one at a time) - Build the array with a known failing drive; - Clone one of the disks to a spare and swap it while running; - Swap in a disk with random data on it not part of the array; - Swap around the arrangement of the disks; - Put strong magnetic next to one of the drive; (Ex. Large sub) - Simulate a fall, drop/shake one of the disks; - Use known bad stick of ram. One that will let the pc run but causes consistent blue screens;
Steve Bez (1 year ago)
Hows the pool swapping out too net tars etc  over the net, smb,ldap,active directories?
Sasquatch (1 year ago)
Known bad non ECC ram will kill pool. Had 8GB stick failing on me(urper 2GB dead) and FreeNas worked fine untill i tried reading 500GB file from it over SMB share. Kernel panic, and unrecoverable pool error. I managed to run same faulty stick with new pool and it worked fine so long as read files were smaller than 5 GB. files between 5 and 10 GB would fail to read, files bigger than 10GB kill pool on read. Writing any size file (500GB max tried) worked fine, veified with good ram. My ram was non ECC(testing FreeNas HW I had lying around)
Christian Vazquez (1 year ago)
Also "simulating" a drop/shake doesn't directly test the filesystem but instead challenges hardware failure on the drive.
Swaping in a disk with random data on it not part of the array & Swap around the arrangement of the disks does not mess with FreeNAS at all. When you move the drives between the SATA cables it figures out based on a signature on them which ones belongs to which Volume and if the drive does not have a signature you can use it as a replacement for an offline drive in a volume.
Allan Jude (1 year ago)
Even if you manage to make it write bad data or metadata, it will most likely detect the checksum mismatch, and use a different copy of the metadata instead. ZFS never overwrites data in place, so it can always fall back to one version earlier (new version every 5 seconds or less), and have a consistent system. ZFS is Amazing.
Steve Bez (1 year ago)
But do you notice\miss any other kind of use(s) compared too PC or wyw?
Roger Airgood (1 year ago)
ZFS jesus is correct.
thegreatga (1 year ago)
You should have randomly pulled then reattached (quickly) the sata cables during writes.
rogejedib333 (1 year ago)
I've got a stick of bad ram (bit stuck high) I could send you.
Glenn Hurse (1 year ago)
one area I have encountered (Not with zfs - not tested) Faulty video cards.. had these things can corrupt BIOS
Dazdigo (1 year ago)
Try submerging it in distilled water and then slowly add salt to it.
Kerry Murphy (11 days ago)
+befru Distilled water may destroy hard discs, but it won't be from a short. Distilled water does not conduct electricity, which was @Dazdigo's point.
whyomgwhywtf (6 months ago)
+befru as long as the water doesn't contain any conductive materials it wont short it out...which is why OP said slowly add salt to distilled water.
Steve Bez (1 year ago)
Aah, tiz a bed bug relate
befru (1 year ago)
I would assume the water would instantly cause short circuits and destroy the electronics. I think most hard drives also have small breather ports on them and would be destroyed by submerging them in any liquid. I think they even have to be left outside of mineral oil PCs for the same reason.
Externalzz (1 year ago)
Would it be worth doing a scrub to be sure?
+Externalzz did that a few times, all went well
Andi Biront (1 year ago)
You could overclock non-ECC memory until is stable enough to boot but generating random errors. Then... see what happens. That's a possible scenario of a memory going bad.
asb2106 (2 months ago)
i have nightmares like this
G H (3 months ago)
FN does *not* specifically require ECC RAM. ECC is highly recommended, and is considered essential on 'critical' infrastructure, but it depends on the use case; a home media server does not for example. If your happy to restore corrupted files from a backup once in a blue moon (RAID is not a backup!), then go ahead and use non-ECC. Read the section under 'ECC RAM or not?' https://www.freenas.org/blog/a-complete-guide-to-freenas-hardware-design-part-i-purpose-and-best-practices/
Rodrigo J. Da Silva (4 months ago)
FreeNAS specifically requires ECC memory...I don't know where you got that claim from
Steve Bez (1 year ago)
Wanna kno how use is 'pooling' ,netw,pxe,all protocol types,with apple \decnet & windows p.c ntfs netw? Any gen?
Phillip Morris (1 year ago)
nice stress test. hey you are aware that the os lives in memory while the ox is running....hence the shut off when you yank out the stick...well i have had zfs save my bacon before...its good stuff
Steve Bez (1 year ago)
Ya,the sun will always shine!
Goran Jordanov (1 year ago)
FreeBSD never DIES! Haha :)
Steve Bez (1 year ago)
Network sony ps4 multimedia backup,any?
Harryo (1 year ago)
Thanks for testing this
Steve Bez (1 year ago)
Any SSD known use, netnodes as hd?
bmxriderforlife1234 (1 year ago)
i knbow how to corrupt a zfs array. dont use ecc ram and overclock it.... lol
Steve Bez (1 year ago)
No,looking for 'use'tests, stress,multipul combined 'overtime'and complex mixed platform,arches,apps,networking protocols, mobile,cloud, etc as 'benchtest' not 'performed under conditions in a lab' but used over time by average performance developer\user . Where OS maybe linux bsd socket related too zfs pool multipul bottlenecks at different parameters giving bug rides or weak points of NTFS networking while p.c rig windows or VM \cloud ,backup types of 'nas' related & compared too database backed development work on usual geared p.c i5/i7 Xeon or mac ppc\pro,etc etc
bmxriderforlife1234 (1 year ago)
no im talking about have freenas running, now start up a vm inside of freenas. passthrough the disc controller that freenas is using. apparently it kills the array. this is why freenas hasnt added hardware passthrough.
Mat Man (1 year ago)
ZFS wants to read the smart data to check the state of the drive. If you have a VM running and pass thru controller to VM ZFS will run like champ. Check out Napp-In-One- https://www.napp-it.org/setup_en.html
bmxriderforlife1234 (1 year ago)
go into the shell and passthrough a disc controller the hdds are connected to. i hear thats why freenas hasnt implemented hardware passthrough in the gui yet. cause theyd have to figure out a way to make the os check which devices are used and needed for the host so you dont passthrough something the os needs and destroy all your data.
goohman (1 year ago)
1st?

Would you like to comment?

Join YouTube for a free account, or sign in if you are already a member.