Meltdown and Spectre Analysis and Current Status

There seems to be a lot of complacent or feel-good news that Meltdown and Spectre will solve themselves or that no worry or care should be taken from users but this couldn’t be further from the truth.  In reality while CPU makers say “there are no known cases of exploits” doesn’t do much to allay fears of those in the know.  This is because Spectre and Meltdown will not leave any trace or evidence that you’ve been hacked.  Although it can be argued that there may be some signs of unauthorized access if that was how access was gained.

However, the nature of Spectre and Meltdown allow for normal authorized users, programs and even scripts on websites to exploit you.  This is why it is so scary as there’s really no way to be certain you haven’t been breached.

It’s an issue for everyone because these exploits could impact anything from your bank, transportation/transit, airplanes, nuclear power plants, and basically anything else that relies on computing security since Meltdown and Spectre are a complete breakdown of those barriers.  I won’t go into more of the basic details but I did make a quick “take on the issue here“.

The good news

There were patches quickly released for a lot of Linux, Windows and Mac devices.  However this doesn’t mean that the users installed the patches or that all users have the ability or access to do so.  Take for example physically remote computers, devices and perhaps some that are running headless that may not be easily accessible or that for some reason have patches disabled (this is more common than you’d think in production or mission critical environments).

Then what about old and unsupported versions of operating systems or that old security system, phone, or TV box, or even ATM whose manufacturer may not be around anymore or is just simply not offering support?

It’s the same issue with many common worms and viruses, patches, and fixes may be issued but millions or more are often still affected long after for various reasons.

The bad news

Even if we assume that Google discovered these flaws first, and if we assume they weren’t mandated to be put there via ARM, AMD and Intel what about insiders who know about this back in June or even earlier on?   From that point a number of individuals and groups could have compromised or damaged sensitive data and computer systems.  There’s still time since a lot of devices and people will not be patched yet.

And to make things worse, the only true way to solve this issue is with a CPU microcode update, which is not simple to deploy especially on embedded devices and any mistake can lead to a bricked device.

These OS patches are just that “patch work”, a hack or work around to mitigate the issue.

Then there’s the question of “we know there are 3 variants or vectors of attack”.  What if there are others that are not yet discovered?  You can be well equipped and funded organizations/hacking groups are working on this as we speak and they certainly won’t be disclosing it.  Until all devices have microcode updates there’s no way to certain we are safe from unknown vectors related to Spector and Meltdown.

What can you do?

Simply look out for the latest updates for your devices/phones/computers and install the update but don’t falsely assume a new update means you are protected unless you’ve read so that “this update fixes the Spectre and Meltdown” issue.

My Take On Meltdown and Spectre Computer Security Flaws

Spectre and Meltdown allow a non-privileged user (non-root/non-Admin)  to access memory they aren’t supposed to essentially dissolving the majority of computing security and privacy barriers.  This could be a guest user collecting sensitive information/passwords for an entire database, group of users, network etc..

If you are using any computing device whether it be an ARM based device, Intel CPU (although Intel is the worst offender at this point), AMD CPU this issue affects you and billions of other devices and users around the world.  Whether you are on Linux, Unix, Windows, Mac this applies to you.  It is really an unmitigated scandal and disaster for both privacy, security and even safety with long lasting and wide ranging ramifications that will continue to playout for years.

I’ve made a comment in the past about security, IOT and how there are many devices that are now unsupported or can’t be updated leading to huge security issues.  We are now unfortunately there and have been since 1995.

This issue was first reported by Google Project Zero and they are known as the Meltdown and Spectre Vulnerabilities that affect all microprocessors made since 1995 (the modern computing era).

To make it worse there are 3 known “variants” or attack vectors known (I suggest there may be more that are undisclosed or not yet known to the public).  With variants 1,2 being very similar (known as Spectre) and variant 3 known as Meltdown.

  • Variant 1: bounds check bypass (CVE-2017-5753)
  • Variant 2: branch target injection (CVE-2017-5715)
  • Variant 3: rogue data cache load (CVE-2017-5754)

The attack is possible due to “speculative execution” where CPUs (computer chips) essentially try to predict future work needed and will actually do sometimes unneeded work as the performance hit for doing this is less than waiting to execute the instructions later.   This means the computer sometimes performs work that isn’t needed and not used to increase performance, where things have gotten bad is through this feature, it’s possible for a normal user/process to gain unrestricted access to memory that you shouldn’t have access to.

What is Spectre?

The primary variants (1,2) that make up Spectre  rely on the user exploiting the speculative feature of the CPU to write to memory under their control.  This allows a normal user to read basically all memory processes allowing keys, passwords and confidential data to be intercepted.  AMD Claims that Variant #2 does not impact them as well.

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5753

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5715

What is Meltdown?

Meltdown is the third and more serious and nasty variant that still relies on the speculative execution exploit/flaw but actually allows the attacker to read arbitrary memory (so basically anywhere at will).  The key feature of Meltdown is that it is the easiest attack to perform and it has been demonstrated on the Intel platform already.

The only good news is that apparently this Meltdown attack only affects Intel and not AMD.

https://access.redhat.com/security/vulnerabilities/speculativeexecution

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5754

Redhat has also done an excellent writeup about the issue here:

https://access.redhat.com/security/vulnerabilities/speculativeexecution

How To Protect Yourself

First and foremost you should update your devices as soon as patches become available.  In Linux enabling KPTI can protect you.   However for some major distributions of Linux users are still waiting for a patch.

If you are vulnerable and performing critical operations it’s time to make tough choices including possibly turning off your machines or denying all non-admin users access to a server/services if possible.

Ensuring rotation of keys and passwords can also mitigate your risks even if passwords have been compromised.

It comes down to good security practices all around such as segregating services to different physical machines, restricting physical and virtual user access.

If possible remove all non-essential or untrusted applications from your device/computer/server.

Dedicated Servers Will Become More Popular

There has been a huge trend to put everything into the Cloud, one that I have reservations with despite owning companies that offer our own private Cloud.

Fortunately we haven’t been impacted by Spectre and Meltdown and are not vulnerable but it does raise questions from our clients that we’ve mentioned before.

I’ve always advocated for physical segregation, which means that if possible you should have your own physical dedicated server that is encrypted and running a minimum set of services with as a few users as possible.  By doing this you significantly reduce your risk in a scenario like this by putting your company database, e-mail, VPN, websites, file server on physically different servers.

Serious Questions and Concerns Raised

I would raise the question that is it really possible that such a wide-ranging exploit was completely unknown for this long until a team from Google discovered it?  Considering the budgets of major intelligence agencies around the world who are constantly looking to find exploits of their own it is conceivable that this vulnerability may have been exploited for far longer than it was publicly known by specific groups.

Another one is Intel’s response to it by apparently being accused of singling out AMD when as of now, Intel is far more vulnerable.

Since these chip makers are all US based is it possible they were mandated by law to introduce speculative execution in such a similar way that this vulnerability would be possible?  Considering recent revelations I don’t think it would be inconceivable.

Are there more than 3 variants and if we assume that no one else really knew about Variants 1-3 is it not possible that a well-armed team could find new ways to exploit them?

Long-term Value for Intel, AMD and ARM

At the time of writing Intel’s stock was down about 3% but this could get worse for either of these companies if one’s vulnerabilities keep increasing and/or one of them is hit with a larger exploit.

Conclusion

It’s hard to give an honest conclusion as we’re just getting started and this is all we know about the Variants 1,2 (Spectre) and Meltdown.  So far it looks like we were lucky to choose AMD.  The key issue that will come out of this is how many devices and users will remain vulnerable by being unable to patch or if they have a device that cannot be easily patched or there is no longer any support from the vendor?  This would increase the amount of zombies and data security breaches several fold.

This is also a good time and a wakeup call for all companies to do a security audit and if they don’t have dedicated security staff, to bring in some good IT and security auditors to assess and mitigate these risks before they become costly losses.

Why Don’t I/We Use RHEL Red Hat Enterprise Linux Instead of Centos?

This a question one of our good client’s asked me one day and I have to admit I wasn’t prepared for this one, it’s something we’ve never put active thought in but was rather a matter of instinct. While we do use various OS’s for different platforms including our own in-house Linux, Cloud, Hosting Control Panel, applications and clients including BSD based OS’s such as FreeBSD, this is something we’ve never been asked.

RHEL (Red Hat Enterprise Linux) has been a clear leader since the early days of providing a standardized, mission critical platform for business applications in the Linux/Unix environment. It was actually my first Linux install back when I was in highschool and I’ve personally maintained Unix/Linux systems for over 16 years now. In that time I have found the strengths and weaknesses of RHEL in terms of our business and clients. However, the Centos project is a legal clone of RHEL with the only difference being artwork and the name Centos it functions identically as RHEL and is completely Open Source.

Since our team of experts does everything in-house we don’t really on Vendor support, it means when something is going on with a server we can solve the problem on our own very quickly with our own team. We don’t have the need to call a third party and ask how-to fix the problem and in fact it’s quicker for us to just do it ourselves whereas I’ve learned many organizations heavily rely on this type of third-party support. The goal with my ventures has always been that our teams should be self-sufficient for both security and efficiency.

Centos being Open Source is a huge advantage for us, we can customize and redistribute OS’s and deploy them on servers without having to touch a button or connect to monitor or KVM (yes I do realize RHEL can be installed headless/kickstart but not in the way we deploy our custom OS images-possibly for another blog). The only con with Centos is that there is a small delay in updates since it depends on the upstream source which is RHEL but this is a minor issue and all major updates are done to Centos almost instantly.

To conclude, my hats off to Linus Torvalds for inventing the Linux Kernel, the RHEL team and especially the CentOS team and I hope this explains why CentOS for my company’s is the best fit for our needs at the moment.

Green Low-Power Alternative Computing Intel NUC Boxes

The key features of low power NUC boxes are that they are small,lightweight,portable,efficient and the low power means less heat and energy savings which all mean “Green Efficient Computing for the Environment”.  In addition consider that you could run these units without power on a UPS for a much longer period of time than most laptops or Desktop computers.  In an emergency or any issue with lack of power, these units will be first and foremost.

As a bit of continuation of my Green Computing talk these NUC boxes take it to the next level from Intel using low power laptop DDR3L SODIMM memory.  I recently bought a barebone Intel NUC J3455 Box for my wife and was impressed at the power usage (literally 10W to the wall at 110V!) and it is still a Quad-Core albeit with slightly lower CPU frequency of just 1.5Ghz but it works great for most functions.  I was able to upgrade it easily to 8GB of RAM (2x4GB), it has an SD card slot built-in, HDMI, VGA and 4 USB 3.0 ports and a 2.5″ SATA 3.0 port that I plugged a 256GB SSD into and installed Ubuntu/Linux Mint on.  It works quite well but there appear to be a few bugs and fidgeting required, for example the NIC cable came loose and it wouldn’t work until I replugged and rebooted it (actually the BIOS stopped showing the NIC at all so somehow it got disabled on its own and there was no option to rThe cheaper one on the left is bare-ones and e-enable or disable it in the BIOS).  You also have to disable the C-Step functions or the CPU doesn’t work properly.  In Linux it looks like there is a bug in the Intel Graphics driver for this model that sometimes causes the graphics/mdm to be restarted. Aside from the tinkering it is well worth the cost saving and works well and reliably.

One thing I will say I am a little surprised at is that the unit does get fairly hot if you are using it heavily and there is no fan in the Intel NUC which means things do get a bit hot (but nothing compared to a laptop).  It comes down to HDD, RAM and CPU being cramped into an incredibly small package.

The Vorke V1 J3160 is basically the same thing as above but does not have the 2 RAM slots (only a single).  However it is priced well and only uses 6W instead of 10W, quite the power savings!  I have purchased the unit below but have not had a chance to test it but I am hoping for the same or better results as the Intel J3455 unit above.  I love how this one comes with 4GB RAM and 64GB SSD out of the box and includes Windows so it’s ready to go out of the box (many like me will just be installing Linux though).  It can serve as an excellent backup box (eg. plug it in somewhere else and hook up a bunch of large USB 3.0 HDDs and keep another copy of your data). Or in my case it could just be an excellent “stand-by” computer with a mirror of your current config “just in case” your main unit goes down you could get going instantly with the backup one.  One other feature I am hoping for in this unit is to see the BIOS is a little more stable and less buggy than Intel’s.  Finally I am hopeful the unit will run cooler due to the 6W CPU and built-in case fan which I think Intel may want to consider too.   Time will tell but I am looking forward to getting a chance to open and test the Vorke V1 and hope they will keep producing similar units.

Dedicated Server Uptime Samples

I just logged into two random dedicated servers and I am always happy about the time uptimes we have:

13:05:37 up 960 days, 21 min,  1 user,  load average: 0.00, 0.01, 0.05

14:11:14 up 835 days, 18:01,  6 users,  load average: 0.09, 0.02, 0.01

In the case of both servers they have never been down, they were literally installed on a rack from the time shown above.

The reason our uptime is always fantastic is not only because our facilities being out of the core disaster areas.  We never overload or oversell our servers.  We are not a budget provider, but still offer excellent value in my opinion.  We’ve had a lot of clients switch to us from other hosts primarily based on the reasoning “no amount of features or gimmicks in the world matter if you have an unreliable service”.

 

Rebooting a Linux Dedicated server with no hard drives from the shell

I just thought I would finally test this so I simulated a complete RAID array failure by pulling all of the drives at once.

This results in an input/output error when trying to do anything so the question is can you still reboot in this situation?

[root@testserver /]# reboot
-bash: /sbin/reboot: Input/output error
[root@testserver /]# shutdown -rn now
-bash: /sbin/shutdown: Input/output error
[root@testserver /]# shutdown
-bash: /sbin/shutdown: Input/output error
[root@testserver /]# uptime
13:47:10 up 41 min,  1 user,  load average: 0.00, 0.00, 0.00

Reboot by sending commands directly to /proc

[root@testserver /]# echo 1 > /proc/sys/kernel/sysrq
[root@testserver /]# echo b > /proc/sysrq-trigger

And sure enough the server rebooted, it could be handy if someone has a remote server without remote hands or remote-reboot (in this case we have both on-site so there was no risk and this was a test server).

What dmesg looks like when the drives are removed and arrays degraded:
[  559.302943] ata3: exception Emask 0x10 SAct 0x0 SErr 0x1810000 action 0xe frozen
[  559.302988] ata3: SError: { PHYRdyChg LinkSeq TrStaTrns }
[  559.303048] ata3: hard resetting link
[  559.303054] ata3: nv: skipping hardreset on occupied port
[  560.024048] ata3: SATA link down (SStatus 0 SControl 300)
[  565.024048] ata3: hard resetting link
[  565.024054] ata3: nv: skipping hardreset on occupied port
[  565.327053] ata3: SATA link down (SStatus 0 SControl 300)
[  565.327064] ata3: limiting SATA link speed to 1.5 Gbps
[  570.327045] ata3: hard resetting link
[  570.327050] ata3: nv: skipping hardreset on occupied port
[  570.630048] ata3: SATA link down (SStatus 0 SControl 300)
[  570.630059] ata3.00: disabled
[  570.630078] ata3: EH complete
[  570.630087] sd 2:0:0:0: rejecting I/O to offline device
[  570.630104] ata3.00: detaching (SCSI 2:0:0:0)
[  570.630125] sd 2:0:0:0: [sda] killing request
[  570.630153] md: super_written gets error=-5, uptodate=0
[  570.630159] md/raid10:md2: Disk failure on sda2, disabling device.
[  570.630162] md/raid10:md2: Operation continuing on 1 devices.
[  570.630257] end_request: I/O error, dev sda, sector 58605128
[  570.630291] md: super_written gets error=-5, uptodate=0
[  570.633517] sd 2:0:0:0: [sda] Synchronizing SCSI cache
[  570.633651] sd 2:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[  570.633659] sd 2:0:0:0: [sda] Stopping disk
[  570.633680] sd 2:0:0:0: [sda] START_STOP FAILED
[  570.633684] sd 2:0:0:0: [sda]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[  570.655206] RAID10 conf printout:
[  570.655210]  — wd:1 rd:2
[  570.655214]  disk 0, wo:0, o:1, dev:sdb2
[  570.655217]  disk 1, wo:1, o:0, dev:sda2
[  570.659025] RAID10 conf printout:
[  570.659029]  — wd:1 rd:2
[  570.659032]  disk 0, wo:0, o:1, dev:sdb2
[  570.659313] md: md1 still in use.
[  570.738752] md: md2 still in use.
[  570.739106] md/raid1:md1: Disk failure on sda3, disabling device.
[  570.739109] md/raid1:md1: Operation continuing on 1 devices.
[  570.739380] md/raid10:md0: Disk failure on sda1, disabling device.
[  570.739382] md/raid10:md0: Operation continuing on 1 devices.
[  570.739412] md: unbind<sda2>
[  570.747449] md: export_rdev(sda2)
[  570.868144] RAID1 conf printout:
[  570.868148]  — wd:1 rd:2
[  570.868168]  disk 0, wo:0, o:1, dev:sdb3
[  570.868175]  disk 1, wo:1, o:0, dev:sda3
[  570.873025] RAID1 conf printout:
[  570.873029]  — wd:1 rd:2
[  570.873032]  disk 0, wo:0, o:1, dev:sdb3
[  570.999292] md: unbind<sda3>
[  571.007119] md: export_rdev(sda3)
[  573.633246] ata4: exception Emask 0x10 SAct 0x0 SErr 0x1810000 action 0xe frozen
[  573.633292] ata4: SError: { PHYRdyChg LinkSeq TrStaTrns }
[  573.633331] ata4: hard resetting link
[  573.633335] ata4: nv: skipping hardreset on occupied port
[  574.354052] ata4: SATA link down (SStatus 0 SControl 300)
[  579.354032] ata4: hard resetting link
[  579.354037] ata4: nv: skipping hardreset on occupied port
[  579.657041] ata4: SATA link down (SStatus 0 SControl 300)
[  579.657052] ata4: limiting SATA link speed to 1.5 Gbps
[  584.657032] ata4: hard resetting link
[  584.657038] ata4: nv: skipping hardreset on occupied port
[  584.960047] ata4: SATA link down (SStatus 0 SControl 300)
[  584.960058] ata4.00: disabled
[  584.960076] ata4: EH complete
[  584.960086] sd 3:0:0:0: rejecting I/O to offline device
[  584.960094] ata4.00: detaching (SCSI 3:0:0:0)
[  584.960124] sd 3:0:0:0: [sdb] killing request
[  584.960148] md: super_written gets error=-5, uptodate=0
[  584.960220] end_request: I/O error, dev sdb, sector 58605120
[  584.960265] md: super_written gets error=-5, uptodate=0
[  584.960322] end_request: I/O error, dev sdb, sector 58605128
[  584.960357] md: super_written gets error=-5, uptodate=0
[  584.960393] end_request: I/O error, dev sdb, sector 58605128
[  584.960428] md: super_written gets error=-5, uptodate=0
[  584.962495] sd 3:0:0:0: [sdb] Synchronizing SCSI cache
[  584.962765] sd 3:0:0:0: [sdb]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[  584.962772] sd 3:0:0:0: [sdb] Stopping disk
[  584.962786] Buffer I/O error on device md2, logical block 524292
[  584.962805] sd 3:0:0:0: [sdb] START_STOP FAILED
[  584.962810] sd 3:0:0:0: [sdb]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
[  584.962824] lost page write due to I/O error on md2
[  584.962841] end_request: I/O error, dev sdb, sector 58605128
[  584.962877] md: super_written gets error=-5, uptodate=0
[  584.962921] md: md1 still in use.
[  584.962931] Buffer I/O error on device md2, logical block 524293
[  584.963007] lost page write due to I/O error on md2
[  584.963020] Buffer I/O error on device md2, logical block 1048646
[  584.963095] lost page write due to I/O error on md2
[  584.963104] Buffer I/O error on device md2, logical block 1048647
[  584.963179] lost page write due to I/O error on md2
[  584.963188] Buffer I/O error on device md2, logical block 1048648
[  584.963274] lost page write due to I/O error on md2
[  584.963280] md: md2 still in use.
[  584.963299] Buffer I/O error on device md2, logical block 1048694
[  584.963381] lost page write due to I/O error on md2
[  584.963391] Buffer I/O error on device md2, logical block 1056863
[  584.963468] lost page write due to I/O error on md2
[  584.963478] Buffer I/O error on device md2, logical block 1056864
[  584.963553] lost page write due to I/O error on md2
[  584.963562] Buffer I/O error on device md2, logical block 6299690
[  584.963635] lost page write due to I/O error on md2
[  584.963800] Aborting journal on device md2-8.
[  584.963836] EXT4-fs error (device md2) in ext4_delete_inode: Readonly filesystem
[  584.963868] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 987 pages, ino 28972747; err -30
[  584.963877] md: super_written gets error=-19, uptodate=0
[  584.963883]
[  584.963888] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 7896 pages, ino 28972690; err -30
[  584.963893]
[  584.963953] EXT4-fs warning (device md2): ext4_end_bio: I/O error writing to inode 28972747 (size 36864 starting block 689771)
[  584.964303] JBD2: I/O error detected when updating journal superblock for md2-8.
[  584.964309] EXT4-fs error (device md2): ext4_journal_start_sb: Detected aborted journal
[  584.964316] EXT4-fs (md2): Remounting filesystem read-only
[  584.972785] md0: detected capacity change from 30005002240 to 0
[  584.972794] md: md0 stopped.
[  584.972810] md: unbind<sdb1>
[  584.979298] md: export_rdev(sdb1)
[  584.979344] md: unbind<sda1>
[  584.987280] md: export_rdev(sda1)
[  585.165084] md: super_written gets error=-19, uptodate=0
[  585.165102] md: super_written gets error=-19, uptodate=0
[  589.309845] EXT4-fs error (device md2): ext4_find_entry: reading directory #262476 offset 0
[  589.963162] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 1024 pages, ino 263495; err -30
[  589.963314]
[  599.310238] EXT4-fs error (device md2): ext4_find_entry: reading directory #262476 offset 0
[  604.963046] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 1024 pages, ino 262464; err -30
[  604.963153]
[  609.310592] EXT4-fs error (device md2): ext4_find_entry: reading directory #262476 offset 0
[  614.963071] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 1024 pages, ino 28186168; err -30
[  614.963176]
[  614.963181] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 8192 pages, ino 28186171; err -30
[  614.963298]
[  614.963301] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 8192 pages, ino 28972747; err -30
[  614.963405]
[  614.963408] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 8192 pages, ino 28972690; err -30
[  614.963507]
[  619.310906] EXT4-fs error (device md2): ext4_find_entry: reading directory #262476 offset 0
[  619.963133] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 1024 pages, ino 263495; err -30
[  619.963244]
[  629.311267] EXT4-fs error (device md2): ext4_find_entry: reading directory #262476 offset 0
[  634.963038] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 1024 pages, ino 262464; err -30
[  634.963144]
[  639.311561] EXT4-fs error (device md2): ext4_find_entry: reading directory #262476 offset 0
[  644.963069] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 1024 pages, ino 28186168; err -30
[  644.963172]
[  644.963176] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 8192 pages, ino 28186171; err -30
[  644.963288]
[  644.963291] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 8192 pages, ino 28972747; err -30
[  644.963395]
[  644.963397] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 8192 pages, ino 28972690; err -30
[  644.963499]
[  649.311846] EXT4-fs error (device md2): ext4_find_entry: reading directory #262476 offset 0
[  649.963202] EXT4-fs (md2): ext4_da_writepages: jbd2_start: 1024 pages, ino 263495; err -30
[  649.963319]
[  653.202216] ata3: exception Emask 0x10 SAct 0x0 SErr 0x50000 action 0xe frozen
[  653.202317] ata3: SError: { PHYRdyChg CommWake }
[  653.202379] ata3: hard resetting link