firmware/protocol development

bushing · 2010-01-03 02:17:52

Moving from the Linux client forum to here:

iZsh wrote:

bushing wrote:
I ran into Roel today at CCC and spoke with him about this for just a little bit, hopefully we can continue the conversation. There's a lot of work that I would like to see done on the Proxmark software (both the host and device-side) -- much of which I will happily volunteer to do myself! -- but I'd like to get a bit of consensus on how to proceed, since there are a few different approaches that can be taken here.
1) The simplest fix for this immediate problem would be to clear the prompt before printing anything (say, debug output) to the screen, then redraw it. Anything that involves "bracket all parts of the code that might..." seems a bit fragile.
2) In general, we have a problem where we have a big long list of commands that operate in different ways that are not necessarily obvious from their name or help text. For example:
Most commands just send a packet to the device; any response will come back as debug "print text" packets from the device. Some (mainly 'tune') will take a special response packet type, interpret it, and print that. Others expect that earlier commands will have put data in the sample buffer, and they may either process this data and display some interpretation, or process that data and put it back into the buffer (e.g. any demod function). Others will send a command that takes the existing sample buffer and uses that as an input. It's hard to really sort out what does what; we really need some sort of interface that defines what commands operate on what data, and how they report status.
3) We have too many commands, in general. As someone who doesn't really understand RFID, but wants to try to read a HF tag -- do I want to use 'hi14areader', 'hi14read', 'hi14bdemod', 'hi15demod', 'hi15read' or 'hi15reader'? Why is there no 'hi14breader' or 'hi14ademod'? etc. Ideally, we'd have just a few general commands -- detect reader, detect card, snoop traffic between card and reader, read card, send raw commands to card, spoof card. As a user, I shouldn't have to -- and often cannot -- tell the software exactly what kind of environment to expect (what kind of card or reader, what the existing contents of the sample buffer will be, etc) -- it should be able to make a good guess about this, and perhaps allow for a manual override. (This is one of the main things that I have no idea how I would even approach, from an RF-processing standpoint).
4) Carrying #3 further, do we even want to keep the same command-response model? Maybe we want to switch to a 2-paned gui, where there's a command line and a log window (like the Windows client, or an IRC client). Maybe we want most functionality to be hidden behind big, brightly colored "read / spoof" buttons. I don't know, I'm just not sure that the current model is the cleanest one.
There are also issues of how analysis work is split between the host and the device, and how exactly the data is sent over USB (do we keep the same current model, and add support for larger (bulk?) packets? or do we go the other direction and use a "USB Serial" (CDC) model, where we have dumb bytes being sent back and forth at relatively high speed?
I'm starting to ramble, so I'll stop here. I think it would be good for us to decide on what an ideal interface style would look like (or would we have two -- one for users, one for development?) -- and then work out a path to get there.
Any thoughts on this would be welcome, even from non OS X users. (Maybe I should start a new thread...)
bushing you might want to repost that in the firmware dev->General (USB driver, Framework, Protocol) part,insofar as it would get lost in the middle of OS X issues in here. Especially since I'll probably throw my couple of cents too.
As you know I started to rewrite the proxmark code (in a top-bottom / pc-fpga swipe algorithm way ) and created a branch for it. From what I saw with the current code, here are my first thoughts about what I'm going to change first in my sandbox:
the asynchronous usb messages from the arm code is kind of nice to be able to output debugging messages whenever we want without having to change anything on the pc/client side, but it makes it very confusing/hard to read (because of non-linear reading of the code)/hard to maintain for normal commands. So I think I'm going to have 2 out endpoints : one for the debugging/asynch messages, and one for bulk messages such that, normal commands will have a classic send message/recv reply code. Easier to read IMHO.

Yeah. I think one easy approach (which only partially solves this problem) is to refactor the use of the debug packets. Right now, we have these packets:

#define CMD_DEBUG_PRINT_STRING                          0x0100
#define CMD_DEBUG_PRINT_INTEGERS                        0x0101
#define CMD_DEBUG_PRINT_BYTES                           0x0102

The first one and second one are used to print debug messages; they are all supposed to be human-readable, but the second one is silly because it prints numbers with no context.

I added a simple printf() implementation to the armsrc, and I moved everything from using CMD_DEBUG_PRINT_INTEGERS to using sprintf() + CMD_DEBUG_PRINT_STRING (this is already done in trunk). Now, nothing uses PRINT_INTEGERS, and nobody ever used PRINT_BYTES (not even implemented in client!). I propose we rename DEBUG_PRINT_INTEGERS to RETURN_CODE (or something), and then use it to report command status (failure, success, buffer length, etc.)

If we do that, we even stay more-or-less compatible with the current client (not that I think that is critical). I agree that we should move to bulk pipes for most things, though.

iZsh wrote:

I'm going to move any signal computations on the arm side, such as the demod commands for instance. In general I think the client should just be a "stupid" client which just sends order to the proxmark. That would enable more standalone features with the proxmark. Of course we should still have some commands to extract raw data from the proxmark whenever we want to do signal analysis while reversing unknown stuff (it might even be useful to be able to save the data in format compatible with gnuradio. Just a side-though), but once something is known, I think it would be best if it's implemented on the arm side. the flash size might be an issue at some point, but I don't think we're there yet.

I agree. If there are processes that take a long time (more than 1 second?), we may want to send status update (progress info) packets to the host, as well as blinking LEDs. Flash size should not yet be a problem -- we are only using about half of our 256K of flash, and there's still plenty we could do to slim down our current memory footprint (e.g. compressing the FPGA bitstream).

henryk · 2010-01-03 14:45:22

Yeah, I saw your printf code and like it. However, there is one major problem: Currently string size is limited to the USB packet payload size, which is 48 bytes (include 0 termination), which really is not a lot (see e.g. my version command where I tried to cram as much info as possible into that size, after 'discovering' the limitation). This will of course go away with some bulk mechanism which we need anyway to stream data to the host.

As a general reminder: When breaking the protocol, please break it thoroughly (so that the old part will refuse to work with a new part), detectably (so that the new part may detect an old part and choose to use a compatibility mode) and extensibly (so that as far as possible further breaks are not necessary).

bushing · 2010-01-03 17:37:03

henryk wrote:

Yeah, I saw your printf code and like it. However, there is one major problem: Currently string size is limited to the USB packet payload size, which is 48 bytes (include 0 termination), which really is not a lot (see e.g. my version command where I tried to cram as much info as possible into that size, after 'discovering' the limitation). This will of course go away with some bulk mechanism which we need anyway to stream data to the host.
As a general reminder: When breaking the protocol, please break it thoroughly (so that the old part will refuse to work with a new part), detectably (so that the new part may detect an old part and choose to use a compatibility mode) and extensibly (so that as far as possible further breaks are not necessary).

Is that really better than breaking it mildly and in a backwards-compatible way? (E.g. adding an optional command from the host to the device to enable bigger packets / bulk packets)

iZsh · 2010-01-03 21:11:53

If we're going to rewrite the whole code anyway, I don't see much point trying to keep backwards-compatibility (especially since it would not force ppl to move to the new protocol). IMHO, it would be better to agree on some common specs/design, and just trash the whole thing and start with a clean slate.

If we don't do it sooner than later, we'll never do it anyway.

henryk · 2010-01-03 21:42:24

Currently I'm seeing two big changes in the future:

What has been discussed here, changing the protocol. Here I would suggest a radical change, since we want to include a number of new features that are impossible with the old protocol. Do away with HID and do everything with libusb, multiple endpoints, bulk traffic, clearer command structure.
Privately I'm also contemplating breaking the partition layout to enable two new features: Enlarge the bootloader partition so that it occupies a whole lock section alone (would make it possible to basically foolproof the bootloader by setting the lock bit) and adding an nvdata partition to enable the device to store measurements in standalone mode. To future-proof this I'll also add an explicit partition table to store the current size and position of the partitions. However, since this will make images incompatible anyway I want to combine it with a break to the image format: Drop the S19 stuff and only use ELF files. I haven't quite yet researched how I would do this, e.g. library-wise.

bushing · 2010-01-03 22:40:45

henryk wrote:

What has been discussed here, changing the protocol. Here I would suggest a radical change, since we want to include a number of new features that are impossible with the old protocol. Do away with HID and do everything with libusb, multiple endpoints, bulk traffic, clearer command structure.

I started looking at this today. I'm no USB whiz, but as far as I can tell, we're currently using bulk endpoints -- well, stepping back, check out page 495 of http://www.atmel.com/dyn/resources/prod … c6175.pdf. Ignoring EP0 and EP3, we only have two endpoints (EP1, EP2) -- each of which support sending 64 bytes at a time, and support double-buffering. We are currently using both of them to send/recv 64-byte packets (command ID + 3 arguments + 48 bytes of data) in 8-byte chunks (see max packet size in USB descriptor). I guess the 8-byte chunk thing is a HID thing?

It seems like the main low-level change that needs to be made is to increase the max packet size in the descriptor to the full 64 bytes, and optionally take advantage of the buffering. As for the protocol, we just need to start using variable-length packets -- I propose that we use a simple header that has a packet length (16 bits?) and a command ID (16 bits?) -- that way, we can just do 2 reads for each packet, one to get the size / command ID and then one to get the rest of the packet. I don't know that it really makes any sense to standardize any of the rest of the packet (like how we dedicate 12 bytes in the current format to 3 arguments) -- thoughts?

henryk wrote:

Privately I'm also contemplating breaking the partition layout to enable two new features: Enlarge the bootloader partition so that it occupies a whole lock section alone (would make it possible to basically foolproof the bootloader by setting the lock bit) and adding an nvdata partition to enable the device to store measurements in standalone mode. To future-proof this I'll also add an explicit partition table to store the current size and position of the partitions. However, since this will make images incompatible anyway I want to combine it with a break to the image format: Drop the S19 stuff and only use ELF files. I haven't quite yet researched how I would do this, e.g. library-wise.

No need to be shy about this Sounds like a good idea to me. A lock section is 64 * 256 = 16K, we currently use a bit over 4K for the bootloader, right? We're nowhere close to running out of flash, but if we run tight, we could probably stuff some reusable stuff into the rest of the 16K (USB descriptor, decompression routine for FPGA bitstream, etc). Or, we could throw some LCD init code in there, wheeee.... Could the partition table sit in the bootloader sector? I'm thinking of something MBR-like, where the first few bytes at address 0 are a jump instruction to the bootloader, and then that's followed by the partition table.

I'm not quite sure what you mean about "library-wise", but a simple ELF loader will fit a few hundred bytes of code -- see e.g. http://gitweb.bootmii.org/?a=viewblob&p … der/stub.c, load_elf(). We could either modify "flasher" to parse ELF files, or, hell, we could even just make flasher send over the whole ELF file and the bootloader could parse it and flash the appropriate parts of memory. I'd be happy to implement either.

How much space do you want for the nvram? Does it need its own lock section, or just to sit in some well-known location in flash?

d18c7db · 2010-01-03 22:53:56

I agree with the protocol change, this is a much needed and long overdue change.

Second point, I have some reservations, in that the flash is a limited resource. Each sector is 16k, so using an entire sector for the boot seems like a waste, given the current boot is under 7k. Will have to decide if sacrificing 10K is worth the page locking mechanism.

Dropping S19, not a biggie, openocd flash write_image supports writing elf files

d18c7db · 2010-01-03 23:23:07

Actually, since there's so much recent interest in firmware development, is anyone interested in the standalone version PM3 ? I finished a prototype recently (the one I posted on youtube) and I was going through the final testing steps (all good btw), then I ended up sending it to someone and due to the holidays the PCB and components I need to build another one are held up in transit. The one in the vid above is actually an even eariler prototype, the final version has the hirose antena connector so as to be compatible with the standard PM3 antenas.

If you like the sound of a board with a color LCD, Gigs of microSD for storage, more than one button backwards compatible with the original PM3, standalone (battery powered) then I could upload the schematic and BOM. The PCB can be ordered from batchpcb for under $30.

iZsh · 2010-01-04 00:56:12

bushing wrote:

It seems like the main low-level change that needs to be made is to increase the max packet size in the descriptor to the full 64 bytes, and optionally take advantage of the buffering. As for the protocol, we just need to start using variable-length packets -- I propose that we use a simple header that has a packet length (16 bits?) and a command ID (16 bits?) -- that way, we can just do 2 reads for each packet, one to get the size / command ID and then one to get the rest of the packet. I don't know that it really makes any sense to standardize any of the rest of the packet (like how we dedicate 12 bytes in the current format to 3 arguments) -- thoughts?

Yeah, I'd have gone for a simple packet length header only, but I guess since most of the time we'll need to include a command ID, it won't hurt to make it part of the common packet header.

Then each command would be responsible to decide how to interpret the packet content. Yes that would probably mean we'd need to write more struct (for each command), but I see this as a upside actually : the code would probably be easier to read. We won't need to dig around to know what arg0, arg1... contain. It would be explicit for each command since each command would provide a proper struct with explicit member names.

bushing · 2010-01-05 01:02:57

d18c7db wrote:

If you like the sound of a board with a color LCD, Gigs of microSD for storage, more than one button backwards compatible with the original PM3, standalone (battery powered) then I could upload the schematic and BOM. The PCB can be ordered from batchpcb for under $30.

Sure, I'd love to see that

I just committed changes to SVN to use ELF files instead of S19 files; I tested it for a few minutes, but I'm sure there are some bugs lurking in there. Be Warned.

proxcat · 2010-01-14 15:59:27

Pardon my ignorance, but what is the advantage in switching to bulk transfers? We are presently using interrupt transfers (common/usb.c:132,140). I'm no USB expert, but based on my understanding of http://www.beyondlogic.org/usbnutshell/usb4.htm, both bulk and interrupt transfer mechanisms have the same bandwidth capabilities. The primary difference between them being latency (bulk has higher latency).

On a separate note, I was wondering what you guys thought about coming up with a central shared task list. This would allow our community of developers to divide and conquer. I've shared a googledoc spreadsheet here: https://spreadsheets.google.com/ccc?key … UlE&hl=en. It is pretty bare right now and I haven't assigned edit rights to anyone yet as I wanted to see what you guys thought of the idea.

I plan on dedicating resources to firmware development in the coming months so I'd like to know what the community thinks are the most important features / enhancements before doing so. Any and all feedback is appreciated.

bushing · 2010-01-14 18:39:17

proxcat wrote:

Pardon my ignorance, but what is the advantage in switching to bulk transfers? We are presently using interrupt transfers (common/usb.c:132,140). I'm no USB expert, but based on my understanding of http://www.beyondlogic.org/usbnutshell/usb4.htm, both bulk and interrupt transfer mechanisms have the same bandwidth capabilities. The primary difference between them being latency (bulk has higher latency).

Upon further examination of the source code, the datasheet, and the USB descriptor, the source of the slowness in the protocol seems to actually be a result of the fact that we have a hard-coded packet size of 64 bytes, transmitted 8 bytes at a time. The chip supports 2 64-byte ping-pong buffers per endpoint, so we should be able to switch to arbitrary-length packets, transmitted 64 bytes at a time using the ping-pong buffers. I think we can do that either using interrupt or bulk packets.

Proxmark3 community

Announcement

#1 2010-01-03 02:17:52

firmware/protocol development

#2 2010-01-03 14:45:22

Re: firmware/protocol development

#3 2010-01-03 17:37:03

Re: firmware/protocol development

#4 2010-01-03 21:11:53

Re: firmware/protocol development

#5 2010-01-03 21:42:24

Re: firmware/protocol development

#6 2010-01-03 22:40:45

Re: firmware/protocol development

#7 2010-01-03 22:53:56

Re: firmware/protocol development

#8 2010-01-03 23:23:07

Re: firmware/protocol development

#9 2010-01-04 00:56:12

Re: firmware/protocol development

#10 2010-01-05 01:02:57

Re: firmware/protocol development

#11 2010-01-14 15:59:27

Re: firmware/protocol development

#12 2010-01-14 18:39:17

Re: firmware/protocol development

Board footer