ramips: fix netgear r6120 factory image generation

as indicated in commit c5bf408ed6 "(ramips: fix image generation for mt76x8")
more rework was needed to fix the other issues.

Building on another machine, but using the same arch, showed
the application failing again for different reasons.

Fix this by completely rewriting the application, fixing following found issues:

- buffer overflows, resulting in stack corruption
- flaws in memory requirement calculations (too small, too large)
- memory leaks
- missing bounds checking on string handling
- non-reproducable images, by using unitilized memory in checksum calculation
- missing error handling, resulting in succes on specific image errors
- endianness errors when building on BE machines
- various minor build warnings
- documentation did not match the code actions (header item locations)
- allowing input to be decimal, hex or octal now

Signed-off-by: Koen Vandeputte <koen.vandeputte@ncentric.com>

target/linux/ramips/image/mt76x8.mk

@@ -115,10 +115,10 @@ define Device/netgear_r6120
   IMAGE_SIZE := $(ralink_default_fw_size_16M)
   DEVICE_TITLE := Netgear AC1200 R6120
   DEVICE_PACKAGES := kmod-usb2 kmod-usb-ohci
-  SERCOMM_KERNEL_OFFSET := 90000
+  SERCOMM_KERNEL_OFFSET := 0x90000
   SERCOMM_HWID := CGQ
   SERCOMM_HWVER := A001
-  SERCOMM_SWVER := 0040
+  SERCOMM_SWVER := 0x0040
   IMAGES += factory.img
   IMAGE/default := append-kernel | pad-to $$$$(BLOCKSIZE)| append-rootfs | pad-rootfs
   IMAGE/sysupgrade.bin := $$(IMAGE/default) | append-metadata | check-size $$$$(IMAGE_SIZE)