.. title: PCI Express development on the cheap (well, sort of)
.. slug: pci-express-development-on-the-cheap-well-sort-of
.. date: 2020-08-22 21:04:53 UTC+02:00
.. tags: artix, eval board, fpga, nuc, pcie, pci express, picoevb, xilinx
.. category: 
.. link: 
.. description: 
.. type: text

So I've been meaning to play with FPGAs and PCI Express for such a long time already now and, in light of some recent events, I decided that it's about time to finally take action.
This post gives an overview of the hardware setup I arrived at using the `PicoEVB development board`_ with a Xilinx `Artix 7`_ FPGA and a PCIe x1 Gen 2 host interface.
The best thing about this is that you can get it starting at just $199 + shipping!

I was looking for a Xilinx kit specifically, and one that is also supported by `Enjoy-Digital`_'s open source `LitePCIe`_ project ("a small footprint and configurable PCIe core powered by `Migen`_ & `LiteX`_") because I'd like to be able to experiment with these "new-fangled" HDLs, too (eventually).
At work we have an `AC701 evaluation kit`_ from Xilinx, also with an Artix-7, but I didn't bother asking if I could borrow it because it's quite expensive ($1.295) and I'd be constantly worried not to damage the thing.
Not fun.
So I started searching the Web and stumbled upon the PicoEVB.
I have to admit that I was rather judgemental of the board, mostly because of the unusual M.2 2230 form factor and the obvious scarcity of I/Os, connectors, anything.
There just wouldn't be that much to evaluate, I thought.
But then it dawned on me.
For evaluating/prototyping *PCI Express* designs, the board *does* have everything that's needed *at least* to get started: An FPGA; flash memory; a built-in USB JTAG "cable"; a PCI Express interface, of course; and a couple of LEDs.
No frills.
It's supported by Vivado's free WebPACK edition and Enjoy-Digital's open source LiteX framework and LitePCIe in particular; probably also `Symbiflow`_.
Sample project, Linux drivers, various open source utilities all available.
What more do I actually want?
`CrowdSupply offer`_ was $219 + $25 shipping to Germany.
Deal!
Placed the order, and 5 (!) days later a small parcel arrived from some Mouser facility in Munich, Germany.
(Now I do wonder about those shipping fees though!?)

So far, so good.
But there's still the unusual (for an eval board that is) M.2 form factor.
Well, it was clear to me from the beginning that I would have a setup with *two* boxen, one main workstation for development & simulation etc. and one "dumb" secondary host with the eval board (whichever it would be) plugged in to do the actual live testing on.
I remember this being mentioned in some blog post way back when I first started looking into this kind of stuff, and I *strongly* recommend you follow this advice, too (at least for the time being).
The thing is that the FPGA, via PCI Express, has access to the entire (physical) address space of its host, including kernel space.
So when you make just the tiniest of mistakes e.g. in the implementation of your DMA engine, the FPGA will gradually clobber main memory until it hits something vital, giving you *just* enough time to realize what's going down before the system freezes or crashes.
Then you have to reboot, fsck your hard drives (and hope for the best), and while trying to restore your session you discover that you've lost the last 5 minutes of work.
I've been involved in the implementation of two DMA designs in my career and I can tell you first-hand that *even the pros mess up* (from time to time ;-)).
I know *I* will mess up.
Even if you're lucky and don't lose any changes, it can be utterly frustrating and kill your morale pretty quickly.
Imagine it's late Tuesday night, you're trying to wrap your head around a particular issue and your workstation crashes for the third time in a row during a test and each time you have to start all over again, twiddling your thumbs, thinking about what went wrong, until Vivado is ready again—not to speak of the joy when manual recovery is necessary.
It's a productivity killer if nothing else.

Anyway, I did have a spare (and sufficiently modern) PC for the PicoEVB at hand, but I still needed an adapter from M.2 2230 Key A/E to PCIe x1.
These kind of adapters seem to be quite rare, but I eventually found an Amazon Warehouse deal for the `Delock PCI Express Card > 1 x internal M.2 key E`_ (Item No. 89889) for €18.03 total.
While waiting for the adapter, I tried to find a spare hard drive that I could use to set up the secondary host PC for the eval board.
What I dug up was a 120 GB 2.5" SSD, together with a vintage 2017 `Intel NUC Kit NUC5CPYH`_ featuring a Celeron N3060 (1.60 GHz) and 4 GB of RAM.

.. image:: /images/nuc5cpyh-inside-view.jpg
   :alt: Inside view of the NUC5CPYH

I originally bought all that stuff a couple of years ago for a media center project that (somehow) never materialized, but what proved advantageous now because the NUC supports PCIe Gen 2 aaand... has an M.2 2230 A+E slot.
Populated with an Intel Dual Band Wireless-AC 3165 (+ Bluetooth 4.2?) adapter card.
It does also have wired Ethernet though, and I really liked the idea of having a small, quiet little box sitting on my desk instead of some cobbled together, unwieldy and probably rather noisy midi-tower taking up half of my real estate.
It's also much easier to stow away or transport.
Now a wireless connection to the test box would've been the icing on the cake, but I was willing to make compromises.
The Delock adapter hadn't arrived yet, so I decided to just go ahead with the NUC setup.
I carefully removed the wireless adapter with the two antennas still connected.
The antennas' female U.FL connectors only have a rating for a precious few mating cycles and I've already made some bad experiences even with larger connectors (probably of bad quality to begin with) deteriorating after just a couple of reconnects.
So I shoved a small plastic zipper bag over the assembly and keep it softly bent out of the way, which seems to work pretty good so far apart from that it's not straightforward anymore to close the case cover (well, case bottom actually, although it's feasible with a bit of effort).
Next I installed the latest stable Debian 10.5 (with SSH server enabled) and finally (cold-)plugged in the PicoEVB.

.. image:: /images/nuc-with-picoevb-soft-off.jpg
   :alt: NUC with PicoEVB in Soft Off state

When the NUC is in Soft Off (S5) state, three red LEDs ('B', 'C', and 'A'—go figure!) on the PicoEVB board are on.
Once the power button is pressed, LED 'B' (leftmost in the picture above) immediately starts to blink and continues to do so as long as the NUC is powered on.
Furthermore two new devices show up, an FTDI USB to (ultimately, I guess) JTAG bridge and a Xilinx "serial controller":

- lsusb

  .. include:: files/posts/2020/pci-express-development-on-the-cheap-well-sort-of/lsusb-picoevb.txt
     :literal:

- lspci

  .. include:: files/posts/2020/pci-express-development-on-the-cheap-well-sort-of/lspci-picoevb.txt
     :literal:

So apparently the PicoEVB FPGA is already configured.
I assume that the flash memory is pre-loaded with the (only, at this writing) sample `Project-0`_, which includes an `spi-loader`_ implementation (the "serial controller"?).
The `README explains the behavior of LED 'B'`_ quite nicely, but it also says that LED 'A' "lights up if FPGA is overtemperature or one of the FPGA voltage rails are out of spec"—which sounds rather worrisome.
If that actually turns out to be the case, then there's nothing I can do about it right now, and probably at all I'm afraid.
The FPGA is definitely not hot, not even warm, and also LED 'A' is on even in S5!?
Maybe I'll double-check this with the Delock adapter in my workstation... we'll see.

.. _PicoEVB development board: https://rhsresearch.com/collections/all/products/xilinx-artix-fpga-dev-board-m-2-fits-in-a-laptop
.. _Artix 7: https://www.xilinx.com/products/silicon-devices/fpga/artix-7.html
.. _Enjoy-Digital: http://www.enjoy-digital.fr/
.. _LitePCIe: https://github.com/enjoy-digital/litepcie
.. _Migen: https://m-labs.hk/gateware/migen/
.. _LiteX: https://github.com/enjoy-digital/litex
.. _AC701 evaluation kit: https://www.xilinx.com/products/boards-and-kits/ek-a7-ac701-g.html
.. _Symbiflow: https://symbiflow.github.io/
.. _CrowdSupply offer: https://www.crowdsupply.com/rhs-research/picoevb
.. _Delock PCI Express Card > 1 x internal M.2 key E: https://www.delock.de/produkt/89889/merkmale.html
.. _Intel NUC Kit NUC5CPYH: https://ark.intel.com/content/www/us/en/ark/products/85254/intel-nuc-kit-nuc5cpyh.html
.. _Project-0: https://github.com/RHSResearchLLC/PicoEVB/tree/master/Sample-Projects/Project-0
.. _spi-loader: https://github.com/RHSResearchLLC/PicoEVB/tree/master/spi-loader
.. _README explains the behavior of LED 'B': https://github.com/RHSResearchLLC/PicoEVB/blob/0638868fbccc7a2a6de8d0032eae4fcde978b888/Sample-Projects/Project-0/README.md#project-details