Not sure if import was successful

[jpan127]

1. I ran script/import.sh with the planet polylines file.

It seems like the process was killed because of a broken pipe (not sure about this).
I am guessing this is not expected behavior.
I modified the script to use my file edges_2017_01_08-15_25_01.0sv and to echo the command before executing it.

time bash script/import.sh 
cat /data/polylines/edges_2017_01_08-15_25_01.0sv | node /home/jpan127/p/interpolation/script/../cmd/polyline.js /data/street.db 1>/data/polyline.out 2>/data/polyline.err;
script/import.sh: line 42: 108174 Broken pipe             cat $POLYLINE_FILE
     108175 Killed                  | node $DIR/../cmd/polyline.js $STREET_DB > $PROC_STDOUT 2> $PROC_STDERR

real	4m24.487s
user	2m48.585s
sys	0m7.760s

I ran dmesg -T| grep -E -i -B100 'killed process' and got:

[Mon Aug 27 22:12:12 2018] Out of memory: Kill process 108576 (node) score 368 or sacrifice child
[Mon Aug 27 22:12:12 2018] Killed process 108576 (node) total-vm:3711788kB, anon-rss:2994380kB, file-rss:0kB, shmem-rss:0kB

2. Running ./interpolate polyline street.db < /data/polylines/edges_2017_01_08-15_25_01.0sv:

Exits after 3 minutes with no message but my `street.db` file is still completely empty.

3. Then running ./interpolate oa address.db < /data/openaddresses/us/ca/alameda.csv produces:

0	0/sec
events.js:167
      throw er; // Unhandled 'error' event
      ^

Error: SQLITE_ERROR: no such table: street.polyline
Emitted 'error' event at:

Which I assume is because street.db is empty.

Conclusion

It seems like none of the commands are executing successfully, but I am not sure what happens or what is supposed to happen.
I have ran concat_oa.sh and that has ran successfully.

[missinglink]

Sounds like this is directly related to #132

If generating the street.db fails there is no point in running the other steps, they will also fail.

I am still hoping the author of the libpostal library will fix the memory leak issue, if not, then we will take measures to patch the code in a fork.

We probably won't have time to produce a workaround for the memory leak until September at the latest, and we'd prefer if we didn't have to.

If anyone is willing to volunteer time or funding to fix this issue, please get in touch with us at [email protected]

[jpan127]

@missinglink

In the meantime, would you say an import of North America, which I would imagine does not have Russian characters, would be safe? That's all I need but I downloaded the planet polylines because it was readily available.

[missinglink]

@jpan127 I can say with absolute certainty that the North America file will contain Russian names :)

I'm also not sure if the bug in libpostal is restricted to only Russian characters, that may be the only case we are aware of.

Latest news on my end: I ran a full planet street.db build and it completed without error, so using the docker image tagged patch as I suggested in #132 (comment) will most likely resolve the issue.

[missinglink]

This issue should now be resolved, please see #132 (comment) for more info.