Consolidating Makefile

henryk · 2009-08-20 10:58:03

I'd like to work on prettifying and simplifying the build system with some technology transfer over from the OpenPICC project. One of the things I'd like to get rid of is the dual Makefile/Makefile.linux. There are several options I'd like to open for discussion:

Least impact: I found a trick on how to make a Makefile that can be read by both nmake and GNU make and which has common and (n)make specific sections. It should be relatively straight-forward to consolidate each Makefile{,.linux} into a single one, with as much code shared as possible. Pro: Minimal changes to the overall system. Con: Somewhat ugly boilerplate code in Makefile (though it never needs to be touched). Less powerful nmake still used on Windows.
Most straightforward: GNU make will read a GNUmakefile instead of Makefile if present. So all make specific content would go there, all nmake specific content would go to Makefile, and then I'd add an additional file to be include from these two for common content. Pro: No magic involved. Con: Multiple different files. GNU make documentation explicitly discourages using GNUmakefile.
Most powerful: Since you're already shipping an arm-elf-gcc, perl and nmake with the Windows compile environment, why not ship a GNU make? Platform dependent settings could still be set within one Makefile, using the much more powerful capabilities of GNU make. Pro: Full power of GNU make for both Linux and Windows. Single Makefile. Con: Needs new release of the Windows compile environment.

henryk · 2009-08-26 12:36:06

Ok, I have ported the Makefiles from bootrom and armsrc over, with a new file common/Makefile.common for common settings and rules. The new Makefile is based on the Windows version, so it will do some things different from the Linux version, but consistently on both operating systems:

Output files (except for bootrom-merged.s19) are in the obj/ dir
All files built in thumb mode, as was the case with the windows build. The Makefile makes a vague reference to a possible bug. If that really is the case, the Makefile will have to be changed to build the affected files in ARM mode. I have verified that the new files create the same binaries as the old files, on the Windows platform.
No flash commands in the Makefile, may be re-added.

For Windows you'll need a new version of the development environment. I've made one by simply copying the binaries and dependencies from http://gnuwin32.sourceforge.net/packages/make.htm
Temporarily you may find it here, but it should be moved to the files section on this site. Also, somebody may want to make a proper update, with a newer gnuarm version :-)

Linux note: You may now set the path to your libgcc.a in your ARMLIB environment variable, no need to modify the Makefile anymore.

henryk · 2009-08-27 17:34:23

Some more build system news:

Most duplicate stuff moved to common/Makefile.common. The armsrc and bootrom Makefiles now just list their source files in ASMSRC, ARMSRC and THUMBSRC (respectively: assembler sources in ARM mode, C sources in ARM mode and C sources in thumb mode) and have definitions for the images they build
Automatic dependency generation for the sources
Common linker script in common/ldscript.common defines the memory regions for the complete project, for greater clarity.
Build process of the bootrom simplified: merge-srec.pl is gone, instead everything is handled in the linker script. This drops the perl dependency for ordinary users.

Note that the linker now knows about the AT91s memory sizes and will complain if you overstep your boundaries. Note also that the elf files now all contain the correct addresses and so will be more fun to use with gdb. The great end result should be a process that builds one monolithic image with bootrom, fpgaimage and osimage for use with gdb and openocd, and then just splits out the individually flashable parts with objcopy.

I re-stumbled upon the missing sanity checks in the flash process. It will happily overwrite the complete memory if it gets addresses that are outside of what it expects. Especially: It expects logical flash addresses with respect to the flash start, not physical addresses as are now used in the elf files. For the time being I have translated the addresses in the objcopy (elf to s19) step, until the flash code is fixed.

henryk · 2009-08-28 09:06:09

Getting better.

rbt2c.pl is now retired, instead objcopy is used to directly convert the raw bitstream file (fpga.bit) into an object file that can be linked with the image. This also provides symbols (_binary_fpga_bit_start and _end) so that there are less hardcoded memory addresses in the code.
The os and fpga images are now first linked into a fullimage.elf (so that the symbols are available) and then split with objcopy.
Since the .bit file has some headers, I implemented a rudimentary parser to locate the actual bitstream. Bonus: There is meta-data in the file. I added a version command to query the compile time (re-discovering the limited USB packet payload).
I also added version information to the osimage and bootrom which will too be returned with the version command.

Everything should be forwards and backwards compatible to the maximum extent possible. The new osimage will accept both an old FPGA image and an old bootrom (and say so when queried for the version). The only thing that clearly is not supported is new fpgaimage with old osimage.

Without rbt2c.pl all real dependencies on perl are gone. However, I added new soft dependencies on perl and the svn command when creating the version information. This should fall back gracefully to limited or no version information if perl or svn are not present. The result of this is: When building official images please have both present. The image will otherwise be marked as suspect or not versioned at all. The version information also contains a flag whether the SVN checkout was clean or had local modifications.

proxmark3> version
> version
#db# Prox/RFID mark3 RFID instrument
#db# bootrom: svn 184 2009-08-28 06:42:16
#db# os: svn 184 2009-08-28 06:42:09
#db# FPGA image built on 2009/ 8/27 at 20:20:35

Last edited by henryk (2009-08-28 09:07:07)

rule · 2009-08-28 12:15:00

Great work!

It helps to keep everything clean

adam@algroup.co.uk · 2009-08-30 19:56:21

Very nice, but it seems we're missing "hitag2.h":

lfops.c:9:20: error: hitag2.h: No such file or directory
lfops.c: In function 'SimulateTagLowFrequencyBidir':
lfops.c:529: warning: implicit declaration of function 'hitag2_init'
lfops.c: In function 'hitag_handle_frame':
lfops.c:675: warning: implicit declaration of function 'hitag2_handle_command'
make: *** [obj/lfops.o] Error 1

henryk · 2009-08-31 01:52:25

Right, sorry about that. I committed while on the road and didn't do my usual "check out on windows and see if it compiles" check.

d18c7db · 2009-08-31 05:33:48

Henry, in ldscript-flash you have a comment "Why doesn't this work..."

Is it perhaps that ORIGIN(bootphase1) should be ORIGIN(.bootphase1), the dot missing in front of bootphase1?

Awesome job you did with the new compile setup Also good catch in bootrom.c with the jump to flash address. I wonder if that was the culprit for some weirdness encountered recently where the firmware would fail to run properly then simply adding some irrelevant code would "fix" it.

Last edited by d18c7db (2009-08-31 05:40:57)

henryk · 2009-08-31 09:56:47

d18c7db wrote:

Is it perhaps that ORIGIN(bootphase1) should be ORIGIN(.bootphase1), the dot missing in front of bootphase1?

Nope. (Before, I even had both with the same name, I added the dot later to reduce confusion.) bootphase1 is the memory region that will contain the phase 1 bootloader, .bootphase1 is the file section that contains the phase 1 bootloader code. ORIGIN and LENGTH are specifically defined to work on memory regions, while ADDR and SIZEOF would work on sections. The issue I'm working around here is, I think, a case of bad magic. Within a section definition, the variable "." contains the current position relative to the start of the section (last {). It can be assigned to, but for obvious reasons it's illegal to set it backwards.

Now, I had previously defined the variable _bootphase1_version_pointer to contain the value 0x1001fc, which is the absolute address that I want the version pointer to end up at. As expected,

. = _bootphase1_version_pointer;

will not work and complain about overflowing the section since it contains code up to 0x2001ff, proving that in this assignment a relative value is needed.

Not a big deal, one might think, simply subtract ORIGIN(bootphase1) which is 0x100000 from the absolute address to get the relative address as in

. = _bootphase1_version_pointer - ORIGIN(bootphase1);

Curiously this won't work either and fails with "cannot move location counter backwards (from 00000000001000fe to 00000000000001fc)". Apparently this time, magically, an absolute address is required. Funky sidenote

. = 0x1001fc - ORIGIN(bootphase1);

*does* work.

So, somehow the appearance of the ORIGIN function changed the behaviour of the dot assignment to require an absolute address, therefore I could work around the strangeness by adding and substracting the ORIGIN from the absolute value I already had, essentially not changing the address:

.= _bootphase1_version_pointer - ORIGIN(bootphase1) + ORIGIN(bootphase1);

And yes, it took me some time to figure that out :-)

d18c7db wrote:

Also good catch in bootrom.c with the jump to flash address. I wonder if that was the culprit for some weirdness encountered recently where the firmware would fail to run

It shouldn't be. Both address areas (0x0 and 0x100000) are strictly identical, the first is simply an alias for the latter, until the remap command is issued to map RAM (0x200000) at 0x0. The current code doesn't use the remap (that's why I was initially confused by the comment), so it should never matter. It does matter when debugging, though, in order to have the program counter in the correct area. (Most, if not all, internal jumps are relative, so wherever you start, there you will stay.)

rule · 2009-08-31 10:11:15

Hey Henryk,

I'm sorry about the filesize limit. You can upload <= 64MB now.
Let me know when it is ready to move the compile environment to the destination directory.

Cheers,

Roel

adam@algroup.co.uk · 2009-08-31 12:18:48

Once again, very nice work!

For clarity, can you confirm which bootrom we should now be flashing? bootrom.s19 or bootrom-merged.s19?

henryk · 2009-08-31 14:55:41

Adam: There is no bootrom-merged.s19 anymore, so there's only bootrom.s19 now.

adam@algroup.co.uk · 2009-08-31 19:06:29

ok, great... need to delete all those old .s19 files that are lying around!

d18c7db · 2009-09-07 10:46:07

I've changed the winsrc Makefile so that now it makes with both gnu make and windows nmake, I also changed the batch files in the cockpit to use make for windows instead on nmake, not really a biggie, more for consistency. Also while in the cockpit I made a tiny change where the make batchfiles accept a parameter now, for example:

5makeall.bat clean

Will make clean in all directories, whereas

5makeall.bat

Will make everything (as before). The parameter you pass is propagated to the final make call so you could use it to make a specific target (if it's common to all 3 make files called) or you could be more specific, eg:

1makearm.bat obj/fpgaimage.s19

As an aside I tweaked the ldscript and ldscript-flash files by adding an eh_frame section which is needed by gcc 4.4.1 otherwise it bombs out. From testing it out this doesn't affect the code generation under the current compile environment which uses gcc 4.1.0, by all indication the lover wersion (lover wersion!? funniest dyslexic typo I made this week) gcc just ignores it. This last one won't mean much to most of you and you probably shouldn't care unless you plan to use a later toolchain to build the source like I do.

Proxmark3 community

Announcement

#1 2009-08-20 10:58:03

Consolidating Makefile

#2 2009-08-26 12:36:06

Re: Consolidating Makefile

#3 2009-08-27 17:34:23

Re: Consolidating Makefile

#4 2009-08-28 09:06:09

Re: Consolidating Makefile

#5 2009-08-28 12:15:00

Re: Consolidating Makefile

#6 2009-08-30 19:56:21

Re: Consolidating Makefile

#7 2009-08-31 01:52:25

Re: Consolidating Makefile

#8 2009-08-31 05:33:48

Re: Consolidating Makefile

#9 2009-08-31 09:56:47

Re: Consolidating Makefile

#10 2009-08-31 10:11:15

Re: Consolidating Makefile

#11 2009-08-31 12:18:48

Re: Consolidating Makefile

#12 2009-08-31 14:55:41

Re: Consolidating Makefile

#13 2009-08-31 19:06:29

Re: Consolidating Makefile

#14 2009-09-07 10:46:07

Re: Consolidating Makefile

Board footer