reading SBR from 2208 shows the data to be offset by a few bytes

[ezonakiusagi]

I know this lsirec tool was not intended to be used with LSI SAS2208 chipset, but since there are lots of similarities between 2008/2108 and the 2208/2308, i tried using lsirec to read the SBR off a 2208 card. The tool complained that there were 2 different copies and it chose to use the 1st one. When I used your 'sbrtool parse' command to read the SBR, all the segments looked wrong. But when I looked closer, I found that everything was offset by a few bytes, .e.g, i found the 0x0107 for PCI device type, but it was not where it should be located.

I haven't read your code to see how you are reading the SBR, but do you know what might cause this?

[marcan]

So two things. One, you should make sure to ./lsirec 0000:01:00.0 halt before reading the SBR, as I found that the MegaRAID firmware will try to use the I2C bus and mess things up. If that's not the issue though, it's entirely possible that these cards use a slightly different SBR format.

[ezonakiusagi]

just as follow up on this. i have now discovered that LSI 2208 and 2308 chipsets can both have either 256B or 512B SBRs, often depending on the chip revision and whether the chip supports PCIe 2.0 vs 3.0. the formats are similar, but different and have completely different offsets for the various PCI id values.

to support writing 512B SBR should be simple, since you just get the size of the SBR file and adjust the buffer and write the size of the buffer. reading SBRs might be a little tricky. you could choose to always read 512B, then check bytes 0x100-0x1FF to see if they are just padding. but I've found both 0x00 and 0xFF padding patterns, not sure why that is. if interested, i can contribute this kind of change.

[strongholdmedia]

Those are not "padding bytes" but the presumed place of the SAS address.
0xFF means uninitialized.
But it seems not all firmware use them. If one doesn't, the area is filled with 0x00.
You should keep in mind that other firmware, however, may use it if you plan crossflashing.
If so, you will need the address checksum to match (or set them to 0xFF, that is, uninitialized).

[strongholdmedia]

To add further to the confusion, I have dumps that have second copy at 0x80, others at 0xe0..

[Fohdeesha]

@marcan How can I help to solve this so lsirec will work with 2208 cards? namely the Dell H710, there is lately a lot of demand to flash these to IT mode, and someone on ebay is making a lot of money selling them flashed over to IT mode and keeping his method a secret (probably just using an EEPROM reader/writer, or a patched version of lsirec). Would dumps of the H710 eeprom (a 2208 based card) help? Perhaps a cash donation?

[Fohdeesha]

I could also provide a dump of a stock dell H710, and an H710 from ebay that has been cross flashed to IT mode, so we could diff them and see which offsets have been modified to result in the new PCI vendor data

[strongholdmedia]

2208 - yes, H710 - no.
Dell cards have two eeproms on it plus the flash.
These eeproms have write protect ability on them.
You cannot just power the chip as the bootloader starts and blocks it from being written, by driving pin 7 high, when there is firmware (any) on it (including bootloader).

You can erase the firmware but then it won't be flashable by the sasflash, only lsi_rec, and that is bad, as there is no lsi_rec format firmware available. Also, Dell servers won't boot in this state.

There could be a method if one could patch the kernel driver to recognize and bootstrap recovery mode devices during boot. All my attempts failed to do this after the system booted.

Plus you have to calculate Dell-specific CRC.
If you can do this, you may be able to use linux/solaris sas2flash utility, or may reverse the EFI utility (but EFI reversing is quite annoying with radare2, so YMMV). Because these look for the official CRC, not the Dell one.
If any of these are done, you anyhow need two working cards at the same time for this. You will need to reboot into EFI after erasing the flash, something you cannot do with mismatched (non-Dell) ID.

Basically, you need to power both cards, then hot-swap them back and forth while keeping the power attached to both of them.
I managed to put one card into "degraded 4x" mode with this sadly.

Since then, though, I do not have any servers nor cards to work with these ATM.

[strongholdmedia]

This took over a week of my time for two items, and in all seriousness, I did contemplate creating a PCIE adapter for the socket, were I to find the pinout.

[Fohdeesha]

Looks like it should be the same as our Dell h310 crossflash guide, the Dell crcs were not an issue there, the only issue is the new offsets. An sth forum member has already figured those out, we are going to do some testing tonight, but in general we've already been using lsirec to crossflash Dell mini cards, the only difference with the h710 is the changed offsets (double sbr size) https://forums.servethehome.com/index.php?threads/perc-h710-mini-to-it-mode.25448/#post-249878

The person that originally opened this ticket also fixed the new offsets in lsirec, kept them a secret (did not contribute back to the project) and has been using it to make money crossflashing h710s and selling them on eBay (you can see if you search eBay for H710 IT). There's also at least one other person (in Bulgaria) using lsirec with patched offsets to sell crossflashed H710s on ebay

[Fohdeesha]

Our H310 guide, for reference: https://doku.phoxden.net/plugins/servlet/mobile?contentId=7208964#content/view/7208964

[strongholdmedia]

I don't remember fixing offsets in the lsirec though. I only remember hacking the flash utility.
I went with the EFI one because I did not have the patience for consecutive boot times (much slower when no SBR).
I am, however, sure that I could not make the DOS one to do what I wanted.

[strongholdmedia]

Had to write my thesis, and will have grad exam tomorrow; as such, this whole ordeal has been forgotten, thanks for remindig me anyway.

[Fohdeesha]

For starters, it looks like lsirec isn't even dumping the whole SBR for D1 revision cards (I realize this was covered above a long time ago, but just writing out my thought process) (9207-8i, SAS2308, Dell H710 Mini (5CT6D), etc), so that's the first issue. These cards SBR is 512 bytes instead of 256. Thankfully MegaRec properly dumps the whole thing. You can see the two attached dumps of a stock Dell 5CT6D, and how lsirec is missing half of it.

For pre-D1 revision cards (like B0) (eg the PCIe 2.0 variants, 9205-8E, SAS2208, Dell H710 Mini (MCR5X & FRH64)) the SBR is still 256 bytes, so lsirec at least dumps the whole thing

D1 dumps.zip

[Fohdeesha]

looking at the source it's quite obvious why it's reading/writing 256 bytes, this should be easy to fix (at least in a D1-specific repo, adding autodetection logic to support both card types in one will be more work):

static int do_readsbr(lsi_dev_t *d, const char *filename)
{
    uint8_t sbr[256];
    int ret;

[strongholdmedia]

That's not how I2C works. :)
If reading another 0x100 bytes above the default 0x100 bytes yields 0xFFs, it is quite probable you have a 2Kbit eeprom, otherwise it is probably 4Kbit.
I skipped that part for I have like 16384 different i2c EEPROM readers, and again, you cannot write EEPROM with anything firmware side without deprotecting the EEPROM first (after booting the card in non-recovery mode, that is).

The real issue, however, is not this.
You cannot boot a Dell firmware with the non-Dell SBR, and the system doesn't boot with a non-Dell firmware. If the card does not have any firmware at boot, the PCIE subsystem and the driver won't initialize it so lsirec will not recognize it (neither) (creates no device node in /sys/bus/pci/blah).

Also, the signed firmware may only be written in regular mode (not recovery).
You also have to ensure that the type value in the SBR is matching the module type, it won't work with a PCIE slot type.

[Fohdeesha]

I'm quite aware how i2c works, and I can now read/write the newer 512 byte SBRs with the above modification.

I 'm not sure why you keep saying it can't be written to, we have already done it using lsirec. I can paste the dumps of the sbr when I return home if you like. As I mentioned, two different people are doing it on eBay as well, for nearly a year now. There's no Dell signing on the sbr's, just the standard checksum, again we have already modified them and dell boots them just fine, you just have to maintain the sub vendor and product IDs. Check the h310 crossflashing guide I linked above for details. The system boots with the non-dell firmware (lsi IT mode firmware) perfectly fine, all it looks for is that the subvendor IDs in the sbr point to Dell. This post is literally being routed through an R720 with an h310 in the mini slot that was crossflashed with non Dell firmware. Boots great

[strongholdmedia]

I 'm not sure why you keep saying it can't be written to, we have already done it using lsirec. I can paste the dumps of the sbr when I return home if you like.

I have mines as well, thanks. After all, I am not saying it "can't be written to"; I wrote "is write protected by the firmware". That is, with firmware (i.e. without erasing flash), you may not even easily use EE programmer ICP (unlike desoldering, which may not be something you'd want to do routinely).

Also, while toying around with lsirec, I found that I could not write SBR with it without entering MPT mode (that is, erasing the flash) neither.
Reading worked okay though.

As I mentioned, two different people are doing it on eBay as well, for nearly a year now.

But neither mention using lsirec, nor indeed any software for doing so, and that is for a reason.

There's no Dell signing on the sbr's, just the standard checksum, again we have already modified them and dell boots them just fine, you just have to maintain the sub vendor and product IDs.

The official LSI firmware is signed, not the SBR.. It also has partitions and stuff, and there were some partition mismatch issue if I remember well.

[Fohdeesha]

Actually one of the eBay listings not only mentions lsirec, but had it in their eBay screenshots. Just follow along the h310 crossflashing guide I linked above, our h710 method is identical except for modified sbr size and offsets. Everything else is the exact same

[strongholdmedia]

I doubt that. It is highly unlikely you could hostboot the h710 (or, indeed, any 2218) with 2118it.bin.
Also, for me, the Sun/Linux lsiutil (the one mentioned) did not at all work with the h710 (firmware or otherwise). It did not even recognize that.
Moreover, the SAS address is also at a different offset, and there is two of it.........

[Fohdeesha]

The h710 (d1 pcie 3.0 revision) gets flashed to 9207-8i, not 2118. No need to even bother with the SAS address manually, you leave it blank (0x00) in the SBR as recent firmware doesn't care, then you just use sas2flash to set the SAS address to what you want. No idea why lsiutil did not see your card, it has seen all of our testbed h710s without issue. Not sure why you have had so many issues, it's really quite simple. You can keep "doubting" all you want but it will be demonstrated and published very soon depending on my free time. I can even send you an example sbr to flash

[strongholdmedia]

The basis of my doubt is predominantly based on you having "blamed" others on not sharing their "solution" with everybody while yourself keeping talking about free time and stuff.

And also that I am fairly sure that I would not start reversing EFI bytecode with cutter as a routine weekend pastime until ascertainment of having been tried every probable permutation of whatever is written here, there, over there, and even on Russian sites.

Also, the "original ebay guy" mentioned something about hardware revisions. It may be possible that you had success with a different one.

[Fohdeesha]

We've had success on both h710 revisions. The "original eBay guy" has contacted me actually (he's also the OP of this GitHub issue), and he confirmed what we already knew: literally nothing you have said is correct or required, all that is needed is a modified SBR. He even offered to send me them. There has also never been signed dell or lsi firmware for these series of cards, you can drag the firmware into ghidra as I did and confirm that for yourself. Please stop bothering everyone subscribed to this issue with your ramblings, if you wish to keep telling me something both the OP and myself have already done is impossible, please just email me directly. I'm sorry you were never able to figure out how to modify an SBR properly, however that doesn't change the reality of the situation.

[Fohdeesha]

everything is much easier once you wipe the 16mb flash of anything dell: https://i.imgur.com/YqsmZ5m.png

we'll have a guide published in maybe a few days, we might try to streamline/script the process for dummies

[strongholdmedia]

We've had success on both h710 revisions.

Both of B1, C1, D1?

literally nothing you have said is correct or required, all that is needed is a modified SBR.

IF you are not BSing (at this point I don't have anything left to believe this) then maybe they weren't required for him.

He even offered to send me them.

Nice for you then.

There has also never been signed dell or lsi firmware for these series of cards, you can drag the firmware into ghidra as I did and confirm that for yourself.

You are so self-indulged, perhaps you could prove with ghidra or whatever that it is not needed. This is just BS.

Please stop bothering everyone subscribed to this issue with your ramblings, if you wish to keep telling me something both the OP and myself have already done is impossible, please just email me directly.

I don't know if you are dyslexic or anything, but I said literally nothing like this. (This is not the first time I realized you have serious text recognition problems, but until now I didn't say, because I tried to be polite. But your level of disrespect brought me to cease this.)

I'm sorry you were never able to figure out how to modify an SBR properly, however that doesn't change the reality of the situation.

Again, nobody wrote or said that.

I spent like 10 years in the GSM unlocking industry to have quite some experience in telling if someone is bullshitting about RCE / security / technology issues.
Now that we stopped being polite, I have to tell that you are exactly this type.

Again, all I've written is how I did succeed doing the task you are bragging about, and how I did not.
I also expressed that had I some time earlier, I would have been glad to help.

Again, to note, you are bragging about AoS could have been sending you modified SBRs - an offer you clearly rejected, since you have already been accomplished this very task, and you just need time to assemble your guide whatsoever.

If he finally ends up sending you his complete method in the background so that you can win your first argument ever on the Internet, please be a man and admit where you got your information from (provided they did not request you not to do so).

Finally, to reiterate, as a bottom line, for people having problems with understanding English text: all the above is my experience of the process. I am not wanting to deter anyone from attempting to accomplish this. I do not sell cards, nor work for Broadcom/LSI.

I just wanted to mention what obstacles I did end up having, while starting exactly the same way ("if anyone can do it, I can, too"). I do consider myself wise not proclaiming it up front, however (or, rather, at all, even in retrospect, provided that no script kiddie had been turning up to annoy me).

Meanwhile, please rest assured, as Dostoyevsky put it, "Talking nonsense is the sole privilege mankind possesses over the other organisms.". Who am I to deprive you of this foremost evolutionary achievement.

[Fohdeesha]

My lord, it took less time to find a method then it did for you to write that mess. All you have to do is use megarec or similar to zero the onboard flash, then you can use lsiutil or similar to hostboot IT firmware. It's that simple, I still do not understand why you're making such a big deal out of it, compared to the other crossflash guides I have published this was probably the simplest. The thread OP can confirm himself he did not send me any methods lmao. You literally stated "these will not boot without Dell firmware", got schooled, and are using word salad to backtrack. Our discord is getting a kick out of it to be fair

[strongholdmedia]

Definitely.
Please come back with your guide.
We eagerly await it.

[strongholdmedia]

For anybody else, I now checked - to avoid misleading others - and both of my cards are Rev. D1.
I am unsure if it matters (but if @Fohdeesha makes sense, it probably would).

[Fohdeesha]

D1s are the pcie 3.0 capable units and require the 512byte SBR (and 9207-8i firmware), the others (C0 B0 etc) are the older pcie 2.0 rev and take a 256byte SBR and 9205-8E firmware, other than those differences the guides are the same. The SBR struct became obvious after looking at SBR rips from the two respective LSI cards, transplanting the Dell subvendor IDs and updating the checksums got us our SBRs. A couple days later the OP of this thread sent me his, and they matched, which was reassuring. The key is to use megarec or similar to zero out the cards entire 16mb onboard flash and reboot, otherwise with the dell firmware still present lsiutil can not hostboot the card properly. However after zeroing, you just use lsiutil to hostboot it, flash the lsi image, and you're done. My screenshot above is the result of following this process on a D1 h710 mini

[strongholdmedia]

The SBR struct became obvious after looking at SBR rips from the two respective LSI cards, transplanting the Dell subvendor IDs and updating the checksums got us our SBRs. A couple days later the OP of this thread sent me his, and they matched, which was reassuring.

This is true. But you cannot boot with the Dell firmware and such SBR, you cannot boot with the LSI firmware and the original SBR, and you cannot flash H710 (D1 at least) in recovery mode (with megarec), nor after hostbooting it.
At least this was my mileage.

[strongholdmedia]

Also no version of sas2fl[a]sh wanted to flash it either. (But the EFI one at least recognized it.)

[Fohdeesha]

The above guide is from exactly a SAS2308 / D1: https://i.imgur.com/YqsmZ5m.png

The super long initialization and issues with lsirec not seeing the card is exactly the behavior we saw when there were still traces of dell software on the card. They all went away when we used the master "cleanflash" command with megarec in freedos. I can almost guarantee if you follow the above guide verbatim, on debian, with the same versions of software in the zip, it will work. We did D1 cards, and others have followed it now as well, and nobody is having the issues you are claiming - well, we did before we completely zero'ed the flash. Another big thing is to ensure in the system bios, I/OAT DMA Engine and SR-IOV global are disabled. These being enabled will interrupt lsirec properly talking to the card

[strongholdmedia]

Did surely clear the flash with that.

The two setting you mentioned I vaguely remember that I turned off, due to someone else recommending it.
Used Debian Buster with kernel 4.19 (the live installer image).

It is quite great if nobody else had these issues (even though I can neither confirm nor refute this statement).
If someone would still run into such, at least they will definitely find this issue.
Perhaps then they'll have better luck using your guide.

[strongholdmedia]

megarec -m0flash 0 H710MMSTOCK.rom

As a side note, you could ask whoever you got this file from to dump their IT flash in the same way.
I suppose it would make all the hassle unnecessary.

[uffsalot]

@Fohdeesha Thanks for the writeup and the modified sbr, works perfectly on my H710 in my R520:

[Fohdeesha]

megarec -m0flash 0 H710MMSTOCK.rom

As a side note, you could ask whoever you got this file from to dump their IT flash in the same way.
I suppose it would make all the hassle unnecessary.

Like the comment above it says, that file is straight from dell (just renamed for clarity): https://www.dell.com/support/home/us/en/04/drivers/driversdetails?driverid=kkr9j&oscode=ws8r2&productcode=poweredge-r720

We did a full 16MB image dump of the IT cards we crossflashed as we had the same idea (just laying down the entire ready to go firmware from megarec would be much faster) but megarec is only built to flash megaraid firmware images, not IT/IR images. It's a simple metadata check that could either be patched out of the megarec binary, or just spoofed and added to the IT image, but that was more work than just publishing the already working lsirec method. If another enterprising individual wants to improve on the guide with megarec patches or similar to speed it up, they have my full support

[Fohdeesha]

@uffsalot glad to help :) Note for anyone else looking at this, my post was not a conclusive guide, just the main required bits. If you're not really comfortable with this stuff I would wait until a full guide is out - for instance those commands are for the D1 card only (Dell 5CT6D) - for B0 revision cards for example you would use the b0 sbr and 9205-8e.bin

[strongholdmedia]

@Fohdeesha Thanks for the writeup and the modified sbr, works perfectly on my H710 in my R520

Could you please mention your BIOS version?
Also could you change (back) your SAS address with sas2flash afterwards?

[strongholdmedia]

If another enterprising individual wants to improve on the guide with megarec patches or similar to speed it up, they have my full support

I am more than willing to spend time on this, and now that I finished my thesis, I will presumably also have time.
I do not, however, have any relevant hardware at the moment, as the two servers I mentioned are running in production, but seriously consider getting another R720 for "research issues".

(Unsure if anyone is interested in it at all, especially here, but I consider very important to mention that the Linux kernel drivers H710, at least D1, but I suppose all of them, will cause kernel panic on regular shutdown, and will require local intervention / power cycling to restart. I am not sure if iDRAC may help with this, but anyone concerned should run at least kernel 5.2 where these are fixed somewhat.)

[Fohdeesha]

Huh? I nor anyone I've asked has ever had kernel panics on shutdown or otherwise, crossflashed or stock firmware. I have like 4 of these in production and have never seen nor even heard of that

[WildOne69]

@strongholdmedia either you're an incompetent or your one of the eBay sellers selling these cards flashed and you don't want this info published. I've flashed half a dozen since @Fohdeesha posted how to do this here and NOT once have I had an issue with reboots in Linux or FreeNAS. IF you don't have something positive to add, GO away.

[uffsalot]

@Fohdeesha Thanks for the writeup and the modified sbr, works perfectly on my H710 in my R520

Could you please mention your BIOS version?
Also could you change (back) your SAS address with sas2flash afterwards?

I'm running bios version 2.7.0 and changing the sas address was no problem.

[strongholdmedia]

@strongholdmedia either you're an incompetent or your one of the eBay sellers selling these cards flashed and you don't want this info published.

That is YOUR prejudice.

I've flashed half a dozen since @Fohdeesha posted how to do this here and NOT once have I had an issue with reboots in Linux or FreeNAS.

Nice for you (even though nobody talked about any kinds of reboots).

IF you don't have something positive to add, GO away.

Please be a pain in your own respective body parts, have you nothing else to do.

[strongholdmedia]

Huh? I nor anyone I've asked has ever had kernel panics on shutdown or otherwise, crossflashed or stock firmware. I have like 4 of these in production and have never seen nor even heard of that

Regardless, it did happen, and I had to upgrade to 5.2.
It only happened on shutdown (halt, -h ) and not on reboots or similar though.
Here is the whole kernel development discussion.

https://patchwork.kernel.org/patch/10734933/

The guy "Kashyap Desai" proposed a fix for it back in time ages before it has finally been fixed in the mainline.
I felt sorry for him, for his patches have been rejected multiple times, for reasons like disagreement about whether the command queue architectural issues in the Linux kernel - which are screwed indeed - should be changed just because "one broadcom driver" "was malfunctioning".
In fact he was right but as the patches are rejected, a lot of efforts have been needed to rectify the issue, so it has only that late they have been mitigated.

Then again, YMMV.

NB. The very same version of the mpt3 driver, below kernel 5.0.7, has a DoS vulnerability (not only in RedHat, all mainline kernels are affected).
I suspect this is the very same problem but had not the time to verify this, so this may be speculation.
I also suspect that it may be used for RCE even if not in the wild.

https://access.redhat.com/security/cve/CVE-2019-11810

[WildOne69]

@strongholdmedia either you're an incompetent or your one of the eBay sellers selling these cards flashed and you don't want this info published.

That is YOUR prejudice.

I've flashed half a dozen since @Fohdeesha posted how to do this here and NOT once have I had an issue with reboots in Linux or FreeNAS.

Nice for you (even though nobody talked about any kinds of reboots).

IF you don't have something positive to add, GO away.

Please be a pain in your own respective body parts, have you nothing else to do.

Nice, YOU get defense just proves how truly incompetent you TRULY are at understanding anything written.

Thank you @Fohdeesha for making IT firmware work on the H710 for Free.

[devinirv]

I have a quick question reguarding this flashing procedure.
Has anyone tested or confirmed the same sbr file being used on the regualar dell h710 in pcie 8x rather than the mini monos. Ive xflashed quite a few regular lsi cards the sas 2208 to 2308 IT. The 9271 8i to a 9207 8i however i recently came across a dell h710 d1 in the same pcb layout as the stock lsi 9271 however when i clear the sbr and write a stock one at 256bytes it esecially doesnt clear propperly and the card goes jnto recovery when i try to flash with sas2flsh.
No im not into hexediting nor do i have an eeprom reader, im just looking for a clear answer if the 512 sbr in your zip would esencially work? Ive also thought of doing an empty 512 sbr file by not to sure of the repocusions with these h710s.

Any help would be appreciated
Thanks

[Fohdeesha]

It uses a slightly different sbr, which my full guide has. Just follow the instructions: https://fohdeesha.com/docs/perc/

[devinirv]

Thank you so much, this worked perfect, was abble to flash 2 of these for my dell systems, going to attempt the mini monos on my buddies 2 r720xd in next few days.

[galaris]

It uses a slightly different sbr, which my full guide has. Just follow the instructions: https://fohdeesha.com/docs/perc/

I followed your guide and was able to flash a H710p Mini in my R720xd without any issues, thank you so much!

[bcraft1901]

hi @Fohdeesha I have followed your fantastic guide on flashing the Dell H710 D1 Mini to IT mode for which I thank you.
Whilst the flashing was very straightforward and im able to install and boot centos 8, once loaded the console shows many "bad sector errors", the ssd is brand new and i have put it in another machine to verify it is working correctly, which it is however i cannot get away from this fault when installed in my R620. Are you able to suggest any way i can debug this or if its a known issue?

Thanks, Ben.

[bcraft1901]

Hi - just in addition to the above I have attached a screenshot taken today.

[Fohdeesha]

@bcraft1901 That's definitely not a known issue and totally unrelated to the flash process - if the ssd does not throw these errors on a different *nix system, it could be the mini mono card is not fully seated correctly (try reseating it), or the backplane cables could not be seated or faulty

[UnixRonin]

I have one question for you which I hope is straightforward. I'm looking at a Dell R720 which I plan to reflash to IT/JBOD to use with ZFS. It is a 16-drive chassis. I know that it uses the H710P controller, though I don't yet know what revision.

I see from its specs that the H710P supports up to 32 drives. Does it STILL support 32 drives after flashing the IT firmware?

Just constructive paranoia here. I'd hate to buy a 16-bay server and only be able to use 8 of them.

[kim-bjoern]

I’m quiet certain you’ll be fine. One of my servers is a R720 SFF with an IT flashed H710 mini. I haven’t had both expanders filled with disks, but 12 SSD’s are running just fine. Sent from the palm of my hand! /Kim Bjoern

…

Den 4. jan. 2022 kl. 00.00 skrev Phil Stracchino ***@***.***>:  I have one question for you which I hope is straightforward. I'm looking at a Dell R720 which I plan to reflash to IT/JBOD to use with ZFS. It is a 16-drive chassis. I know what it uses the H710P controller, though I don't yet know what revision. I see from its specs that the H710P supports up to 32 drives. Does it STILL support 32 drives after flashing the IT firmware? Just constructive paranoia here. I'd hate to buy a 16-bay server and only be able to use 8 of them. — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.

[devinirv]

The dell r720 fully supports 16 ssd and hdd through the mini card built in in IT mode, there should be a quick flashing guide that is posted here graciously by Fohdeesha, who has worked hard to find an easy 123 click solution rather than the harder way we have been flashing these cards before. i have successfully flashed over 16 regular dell perc710s and various other dell cards. The onboard daughter card in the dell r720 and 730 are no different. I am currently running a Dell r720xd with the onboard perc as well as the dual 10gb onboard nic card with 16 lff drives and gtx 1070 as well as a second lsi megarec card and another 4gb nic as well as the 2.5 inch ssd slot in the back with some micron slc nand flash ssds. these systems are great for home servers and cheap, they are however starting to get dated. one thing to keep in mind if the acoustic level is dependent on the cpu tdp.

[UnixRonin]

Thank you both! Yes, I came here FROM Fohdeesha's quick click solution.

So that would be a H720P mini then?

Background:
My SunFire X4540 failed to come back after a power outage (despite being cleanly shut down while still on UPS power), an attempt to replace it with a current-model QNAP NAS was one of the more horribly disappointing experiences I can remember, and I'm looking at frobbing an R720 into a proper replacement. It has fewer drive bays than the X4540 of course, but also lacks the X4540's 2TB drive limit, so I should be able to have a lot more storage for a lot less power draw. I'm looking at the SFF model though.

[devinirv]

It's a personal choice to go sff but I came from having a cisco ucs c240 sff and it was hard to source large capacity 2.5 inch disk if not going all ssd. Largest I managed to aquire was 4tb Seagate and these are eco models from portable enclosures. It was just to costly and economically and logistically not feasible for a Nas enclosure. Atleast with lff 16 bay you can also put in 2.5 as the dell caddies do support mounting 2.5inch ssd and hdd. The backplane also supports larger than 2tb and the perc710 mono supports larger than 2tb disks. The dell raid card in the r720 should be a mini monolithic card. Best of luck.

[Fohdeesha]

Guys, this is not the place to discuss crossflashing or dells, this is Marcan's lsirec repo, and a completely unrelated github issue. Everyone is just spamming his inbox by using this to discuss my mostly unrelated guide. For guide discussion choose any one of the following:

https://forums.servethehome.com/index.php?threads/guide-flashing-h310-h710-h810-mini-full-size-to-it-mode.27459/
https://www.truenas.com/community/threads/guide-flashing-h310-h710-mini-to-it-mode.82196/
https://github.com/Fohdeesha/lab-docu/issues

[walterav1984]

Small success with careless cross flashing the Fujitsu D3116C1 1GB from "iMR" to "IT-mode":

$ lspci #output edited
#original
27:00.0 0107: 1000:005b (rev 05)
	Subsystem: 1734:11e4
27:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2208 [Thunderbolt] (rev 05)
	Subsystem: Fujitsu Technology Solutions MegaRAID SAS 2208 [Thunderbolt]

#post crossflash sbr
27:00.0 0107: 1000:0087 (rev 05)
	Subsystem: 1028:1f34
27:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
	Subsystem: Dell SAS2308 PCI-Express Fusion-MPT SAS-2

#post firmware it-mode flash sas2hax.efi
27:00.0 0107: 1000:0087 (rev 05)
	Subsystem: 1000:3020	
27:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
	Subsystem: Broadcom / LSI 9207-8i SAS2.1 HBA

Similar to the other 2208 cards mentioned earlier it was hard to find a compatible MegaRec/Cli tool that was compatible with the 512 bytes sbr and 16MB flash. Also the original and modified sas2flash utils and lsiutil didn't recognize the card before modifying the sbr!

The following Youtube URL from AGTech suggested the right Megarec.exe from HP which detects card and could read/backup/write sbr and erase whole 16MB and MegaCLI.exe from Broadcom for backing up txt info.

md5sum MegaRec.exe
5588ac9f07f67bb8a2c61a2995230714

md5sum MegaCLI.exe
8c1d85401dbc27ba64605e01c626b5d9

#backing up original Fuji D3116C1 sbr/spd/info from FreeDOS
megarec.exe -readsbr 0 fd3116sbr.bin
megarec.exe -readspd 0 fd3116spd.bin
megacli.exe -adpallinfo -a0 > fd3116cli.txt

After making a backup of the sbr, spd and cli-info of the original Fujitsu roms with both MegaRec/CLI.exe and lsirec, I acknowledged that I simply didn't know how to backup the full 8/16 MB ROM without a functioning sas2flash which I successfully did before with other original LSI cards...

By modifying lsirec.c and changing [256] into [512] at line 667 and 694, lsirec started to read/backup and write the same data/checksums from within Linux as the original MSDOS MegaRec.exe tool did while using FreeDOS.

So I dared to flash the sbr using lsirec with a Dell 710PD1md.sbr from @Fohdeesha since the Dell card shared a lot similarities with the Fujitsu using marcan's lsirec instructions by using hostboot mode.

#Ubuntu Linux 23.10
# echo 16 > /proc/sys/vm/nr_hugepages #as root user!

$ sudo ./lsirec 0000:27:00.0 info
Device in MPT mode
Registers:
 DOORBELL:       0x00000000
 DIAG:           0x000001b2
 DCR_I2C_SELECT: 0xc003ff0a
 DCR_SBR_SELECT: 0xb5000009
 CHIP_I2C_PINS:  0x00000003
IOC is RESET

$ sudo ./lsirec 0000:27:00.0 unbind
Trying unlock in MPT mode...
Device in MPT mode
Kernel driver unbound from device

$ sudo ./lsirec 0000:27:00.0 halt
Device in MPT mode
Resetting adapter in HCB mode...
Trying unlock in MPT mode...
Device in MPT mode
IOC is RESET 

$ sudo ./lsirec 0000:27:00.0 readsbr sbrfuji_backup512.bin

$ md5sum  710PD1md.sbr
44e32ff0415e48c05598bae78f77c32d

$ sudo ./lsirec 0000:27:00.0 writesbr 710PD1md.sbr
success

$ sudo ./lsirec 0000:27:00.0 hostboot 9207-8.bin
Device in MPT mode
Resetting adapter in HCB mode...
Trying unlock in MPT mode...
Device in MPT mode
IOC is RESET 
Setting up HCB...
HCDW virtual: 0x7f7075e00000
HCDW physical: 0x161400000
Loading firmware...
Loaded 809436 bytes
Booting IOC...
IOC is READY 
IOC Host Boot successful.

$ lspci -vns 0000:27:00.0 -A linux-sysfs | head -n 2
27:00.0 0104: 1000:005b (rev 05)
	Subsystem: 1734:11e4

#nothing shows...
$ lspci -vns 0000:27:00.0 -A intel-conf1 | head -n 2
$ lspci -vns 0000:27:00.0 -H1 | head -n 2

$ sudo lspci -vns 0000:27:00.0
27:00.0 0104: 1000:005b (rev 05)
	Subsystem: 1734:11e4
	Physical Slot: 7
	Flags: bus master, fast devsel, latency 0, IRQ 174, NUMA node 1
	I/O ports at 8000 [size=256]
	Memory at fbef0000 (64-bit, non-prefetchable) [size=16K]
	Memory at fbe80000 (64-bit, non-prefetchable) [size=256K]
	Expansion ROM at fbe00000 [virtual] [disabled] [size=128K]
	Capabilities: [50] Power Management version 3
	Capabilities: [68] Express Endpoint, MSI 00
	Capabilities: [d0] Vital Product Data
	Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] MSI-X: Enable- Count=16 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1e0] Secondary PCI Express
	Capabilities: [1c0] Power Budgeting <?>
	Capabilities: [190] Dynamic Power Allocation <?>
	Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
	Kernel modules: megaraid_sas

#not sure anymore if I did a modprobe -r megaraid_sas before
$ sudo ./lsirec 0000:27:00.0 rescan
Device in MPT mode
Removing PCI device...
Rescanning PCI bus...
PCI bus rescan complete.

$ sudo lspci -vns 0000:27:00.0
27:00.0 0107: 1000:0087 (rev 05)
	Subsystem: 1028:1f34
	Physical Slot: 7
	Flags: fast devsel, IRQ 174, NUMA node 1
	I/O ports at 8000 [size=256]
	Expansion ROM at fbe00000 [virtual] [disabled] [size=1M]
	Capabilities: [50] Power Management version 3
	Capabilities: [68] Express Endpoint, MSI 00
	Capabilities: [d0] Vital Product Data
	Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] MSI-X: Enable- Count=16 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1e0] Secondary PCI Express
	Capabilities: [1c0] Power Budgeting <?>
	Capabilities: [190] Dynamic Power Allocation <?>
	Capabilities: [148] Alternative Routing-ID Interpretation (ARI)
	Kernel modules: mpt3sas

After noticing this success, I rebooted the machine and started to repeat the same (cross)flash steps in FreeDOS with MegaRec.exe and the Dell modified sbr and wiping the whole 16MB flash.

#FreeDOS
megarec.exe -writesbr 0 710PD1md.sbr
megarec.exe -cleanflash 0

Than used a UEFI computer with a modified sas2hax.efi, since original MSDOS sas2flash failed with Mfg / PAL errors, than flashed stock "9207-8.bin" and altered sas-address.

#efi shell
fs0:
sas2hax.efi -o -f 9207-8.bin

#md5sum of original lsi firmware
532ece37507a51feb8e78ff94ddfdc83  9207-8.bin

After this real flash, original sas2flash and lsiutil programs start to recognize the card ootb and can dump/write parts of the firmware so I can verify the flash was successful by comparing the md5sum of the it-mode firmware file.

The card "somehow" functions now in IT-mode with mpt3sas instead of megaraid_sas and detects disks on multiple ports but not all connectors (not sure if it did before with original fuji firmware). If I read marcan's original instructions there is more to this and probably a real checksum matched from the original fujitsu sbr could make a difference, but also (re)writing the SBR again after flashing to it-mode firmware could improve port detection... You can see that after flashing the it-mode firmware the SBR changes again like the device-id from Dell changed to LSI 9207-8i, not sure if that was intended.

@Fohdeesha could you hint how you changed the Dell SBR with correct checksumming, Id rather reflash and verify with a modified Fujitsu sbr than Dell one? Understood now how to calculate checksum.
For 512 bytes eeprom the 2 repeated checksum value is "a+b=c" where "c=2a5b", and "a=bytes 0-222" "b=byte 223". Using mate-calculater in scientific mode it shows a "mod" button which results in "2a5b mod 100 = 5b" as opposed to 256 bytes version where it is "a5b" discovered by marcan. When doing "c - b=a" where "a" represents the value of bytes "0-222" which are gonna be modified because of changing "device-id" byte and "imr/it-ir-mode" byte can be known. For me 2a5b - a2 = 29b9 = old "a" value of bytes 0-222, than minus old device id value "5b" plus new device-id "87" minus old "01" plus new value "00" creates a new valua for "a" and than "c" minus new "a" gives new checksum value "b"= 77 in my case.

Made a minimal 3 byte changed modified original fujid3116sbr (6bytes total because 2 times editing repeated data):
changed device id at offset 20 from "5b > "87"
changed offset 96 from "01" > "00" probably imr to it/ir mode
changed checksum byte at offset 223 " from "a2" > "77" fix checksum

This file is called "mod3116sbr.bin"

After these changes, the card still behaved the same as dell 710 firmware, but I narrowed the port detection issue down to 1 dead port (SAS-MCL1, hdd connector 3 at top of the card, SAS-MCL2 connector at bottom near motherboard is fine) and 1 sata 1.0 hdd 100GB disk that doesn't work at all on any of my other original lsi cards.

In the order of "rinse and repeat" from what I can still remember and reconstruct I finally dared to introduce a self constructed 512 bytes "demptysbr.bin" as a first step before cross-flashing inspired by most online cross flash tutorials. By hex editing a 512 byte file to have only 00 from bytes 0-447 and only FF from 448-511 you can create a "demptysbr.bin" file. Which resulted after "cleanflash 0" and reboot in a raid card that has device-id "1000:0089" which can only be used for recovering by "MegaRec.exe" by supplying again a working SBR, since lsirec, lsu-util or any sas2xxx tool won't recognize the card anymore after that...

#FreeDOS
megarec.exe -writesbr 0 demptysbr.bin
megarec.exe -cleanflash 0

#reboot to FreeDOS 
megarec.exe -writesbr 0 mod3116sbr.bin

#reboot again to efi shell
fs0:
sas2hax.efi -o -f 9207-8.bin

#reboot to FreeDOS again it may fix the port detection issue because of not yet understood reasons...
megarec.exe -writesbr 0 mod3116sbr.bin

My guess is that this 2nd hand Fujitsu D3116C1 card was shipped with a single dead hdd connector instead of the crossflash procedure (could not verify before flash), and using the 1.0 sata hdd as test disk gave me the wrong idea that more ports were not working... If this dead port is used, it can kernel panic the mpt3sas driver, or halt the system from booting! Like normal LSI card, led is also green blinking, smart status works and normal IO tested no heavy benchmarking yet! Also removing the 1GB extension module on the card has no effect, currently running without it!

[walterav1984]

For now 7 ports seem to work from the 8 on the Fujitsu D3116C1 card, updated my previous post and my guess is that crossflash was successfull but the Fujitsu card has 1 broken connector and I was using a too old sata 1.0 HDD to verify function of the other remaining ports!