strange irom0 memory use

[pljakobs]

so I've moved all configuration access over to ConfigDB and am trying to get the app working.
Part of that was, as indicated earlier, to include the default config in the delivered image.
This is where I import files:

#define FILE_LIST(XX)                                        \
    XX(default_config,  "default_config.json")               \
    XX(index_html,      "webapp/index.html.gz")              \
    XX(VERSION,         "webapp/VERSION")                    \
    XX(index_jss,       "webapp/assets/index.css.gz")        \
    XX(index_js,        "webapp/assets/index.js.gz")         \
    XX(RgbwwLayout_css, "webapp/assets/RgbwwLayout.css.gz")  \
    XX(RgbwwLayout_js,  "webapp/assets/RgbwwLayout.js.gz")   \
    XX(i18n_js,         "webapp/assets/i18n.js.gz")          \
    XX(pinconfig_json,  "webapp/config/pinconfig.json")      \
    XX(favicon_ico,     "webapp/icons/favicon.ico")          \
    XX(network_wifi_3_bar_locked_FILL0_wght400_GRAD0_opsz24, "webapp/icons/network_wifi_3_bar_locked_FILL0_wght400_GRAD0_opsz24.svg") \
    XX(network_wifi_1_bar_FILL0_wght400_GRAD0_opsz24,        "webapp/icons/network_wifi_1_bar_FILL0_wght400_GRAD0_opsz24.svg")        \
    XX(network_wifi_FILL0_wght400_GRAD0_opsz24,              "webapp/icons/network_wifi_FILL0_wght400_GRAD0_opsz24.svg")              \
    XX(network_wifi_1_bar_locked_FILL0_wght400_GRAD0_opsz24, "webapp/icons/network_wifi_1_bar_locked_FILL0_wght400_GRAD0_opsz24.svg") \
    XX(network_wifi_locked_FILL0_wght400_GRAD0_opsz24,       "webapp/icons/network_wifi_locked_FILL0_wght400_GRAD0_opsz24.svg")       \
    XX(network_wifi_2_bar_FILL0_wght400_GRAD0_opsz24,        "webapp/icons/network_wifi_2_bar_FILL0_wght400_GRAD0_opsz24.svg")        \
    XX(signal_wifi_statusbar_null_FILL0_wght400_GRAD0_opsz24,"webapp/icons/signal_wifi_statusbar_null_FILL0_wght400_GRAD0_opsz24.svg")\
    XX(network_wifi_2_bar_locked_FILL0_wght400_GRAD0_opsz24, "webapp/icons/network_wifi_2_bar_locked_FILL0_wght400_GRAD0_opsz24.svg") \
    XX(wifi_lock_FILL0_wght400_GRAD0_opsz24,                 "webapp/icons/wifi_lock_FILL0_wght400_GRAD0_opsz24.svg")                 \
    XX(network_wifi_3_bar_FILL0_wght400_GRAD0_opsz24,        "webapp/icons/network_wifi_3_bar_FILL0_wght400_GRAD0_opsz24.svg")    

// Define the names for each file
#define XX(name, file) DEFINE_FSTR_LOCAL(KEY_##name, file)
FILE_LIST(XX)
#undef XX

// Import content for each file
#define XX(name, file) IMPORT_FSTR_LOCAL(CONTENT_##name, PROJECT_DIR "/" file);
FILE_LIST(XX)
#undef XX

// Define the table structure linking key => content
#define XX(name, file) {&KEY_##name, &CONTENT_##name},
DEFINE_FSTR_MAP_LOCAL(fileMap, FlashString, FlashString, FILE_LIST(XX));
#undef XX

and as long as I do that, it compiles file, with the following memory layout:

   Section|                   Description| Start (hex)|   End (hex)|Used space
------------------------------------------------------------------------------
      data|        Initialized Data (RAM)|    3FFE8000|    3FFE87E0|    2016
    rodata|           ReadOnly Data (RAM)|    3FFE87E0|    3FFE9BA4|    5060
       bss|      Uninitialized Data (RAM)|    3FFE9BA8|    3FFF17D8|   31792
      text|            Cached Code (IRAM)|    40100000|    4010706F|   28783
irom0_text|           Uncached Code (SPI)|    40202010|    402D8DB4|  880036
Total Used RAM : 38868
Free RAM : 43052
Free IRam : 4003

In my application.cpp, I have this code to intialize the db:

        if (general.getIsInitialized()) {
            debug_i("application init => reading config");
      	    //auto configStream = new FSTR::Stream(fileMap[F("default_config")]);
            //cfg->importFromStream(ConfigDB::Json::format, *configStream);
            debug_i("application init => config loaded, pin config %s",general.getPinConfig().c_str());  
        } 

and the compile result is with those two lines commented out.
As soon as I remove the // from the first line, I run out of irom0 space:

ld: region `irom0_0_seg' overflowed by 122676 bytes

How do I get from ~120kB free to missing 120kB just by instantiating this file as a stream? The config file is around 8kB

du -sh default_config.json 
8,0K	default_config.json

thinking about it a bit later:
since it's a define, the file itself will ony get inserted in the build once it's used, right? So it would be expected to use up more space, but that much more?

checking a bit deeper -
according to my hardware config, my irom0 should be 1016kB in size, hwconfig-edit also shows 121.5kB free when not using the default_config.json
Not sure I understand what's goign on here

[mikee47]

Any use of filemap will result in all the imported files getting linked into the binary, not just the one you've selected. Try this:

auto configStream = new FSTR::Stream(CONTENT_default_config);

[pljakobs]

hmmm.. of course, I need them all, the others are being used by the weberserver onFile() - will they be potentially mapped in more than once?

[mikee47]

Nope, the IMPORT_FSTR macros generates the data with a unique symbol, and the map is just a list of pointers.

[mikee47]

Sorting out memory usage issues like this would normally involve tools like nm:

nm -SC --size-sort out/Esp8266/debug/build/app_0.out | less

You could do a 'before and after' comparison (using kdiff3 or something) but it would need a little massaging as addresses would all be different and you'd need to compare by object name.

Anyway, problem here is that app_0.out won't exist because the link step failed. Edit out/Esp8266/debug/rom0.ld and change 1M to 2M.

Open up Sming/Components/rboot/component.mk and look at line 196, starting printf "%sorg = 0x40200010 + ($$($3) & 0xFFFFF), len = 1M - .... Change the 1M to 2M and run make -j rebuild. Remember to change it back to 1M before attempting to run on target hardware!

If you're able to build for Host, then you can use:

nm -SC --size-sort out/Host/debug/firmware/app | less

Since the problem is likely to be in generic code (not specific to the esp8266), then this might suffice.

[pljakobs]

so after looking at this quite some time, I realized that the whole flashString content is being written to the output file twice, I assume due to the fact that they were being used in two different translation units.

I was hoping that the content could be shared.

I have now defined the default_config.json file a single flashString in the correct translation unit, now my rom0 size is only 876436

[mikee47]

I was hoping that the content could be shared.

If you would provide a minimal example to demonstrate this behaviour I can investigate further.

[pljakobs]

I will try.