osm2pgsql openstreetmap planet-latest.osm.pbf import duration

1. 서버사양(Server Spec) :

 

Model : Dell Precision T5600 (PCIe v3)

ubuntu 20.04.6

cpu : Xeon E5-2643 3.3GHz Dual

ram : 32G

hdd : samsung nvme m.2 980 Pro with heatsink 2TB

 

2. planet-latest.osm.pbf (2024.09)

renderaccount@ellord-Precision-T5600:/home3/data$ du -sh planet-latest.osm.pbf 
77G	planet-latest.osm.pbf
renderaccount@ellord-Precision-T5600:/home3/data$

 

 

3. osm2pgsql

renderaccount@ellord-Precision-T5600:/home3/data$ osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -C 12000 --number-processes 10 -S ~/src/openstreetmap-carto/openstreetmap-carto.style /home3/data/planet-latest.osm.pbf 
osm2pgsql version 1.2.1 (64 bit id space)

Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=12000MB, maxblocks=192000*65536, allocation method=11
Mid: pgsql, cache=12000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using lua based tag processing pipeline with script /home/renderaccount/src/openstreetmap-carto/openstreetmap-carto.lua
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads

Reading in file: /home3/data/planet-latest.osm.pbf
Using PBF parser.
Processing: Node(9392643k 491.5k/s) Way(1050544k 3.70k/s) Relation(12528500 189.62/s)  parse time: 369432s
Node stats: total(9392643418), max(12179381022) in 19111s
Way stats: total(1050544832), max(1315819058) in 284248s
Relation stats: total(12528527), max(18049831) in 66073s
Sorting data and creating indexes for planet_osm_line
Sorting data and creating indexes for planet_osm_polygon
Sorting data and creating indexes for planet_osm_roads
Stopping table: planet_osm_nodes
Stopping table: planet_osm_ways
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_rels
Building index on table: planet_osm_ways
Building index on table: planet_osm_rels
Stopped table: planet_osm_nodes in 0s
Copying planet_osm_roads to cluster by geometry finished
Creating geometry index on planet_osm_roads
Creating osm_id index on planet_osm_roads
Creating indexes on planet_osm_roads finished
All indexes on planet_osm_roads created in 1385s
Completed planet_osm_roads
Copying planet_osm_point to cluster by geometry finished
Creating geometry index on planet_osm_point
Stopped table: planet_osm_rels in 2576s
Copying planet_osm_line to cluster by geometry finished
Creating geometry index on planet_osm_line
Creating osm_id index on planet_osm_point
Creating indexes on planet_osm_point finished
All indexes on planet_osm_point created in 6749s
Completed planet_osm_point
Creating osm_id index on planet_osm_line
Creating indexes on planet_osm_line finished
All indexes on planet_osm_line created in 13915s
Completed planet_osm_line
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.

Osm2pgsql took 389484s overall
node cache: stored: 1460253237(15.55%), storage efficiency: 92.84% (dense blocks: 168803, sparse nodes: 95018032), hit rate: 15.29%

 

 

흠...

 

마지막 polygon import 단계에서 오류발생하여 종료됨.

 

총 시간 : 389484 초 (4일 12시간 정도 소요됨)

 

 

 

4. polygon에서 오류는 발생하였지만 polygon을 제외하고 지도는 정상적으로 표시됨.

 

 

 

 

5. postgis database 용량

 

renderaccount@ellord-Precision-T5600:/home3$ sudo du -sh postgres-data/
1.5T	postgres-data/
renderaccount@ellord-Precision-T5600:/home3$ sudo du -s postgres-data/
1603660736	postgres-data/
renderaccount@ellord-Precision-T5600:/home3$ sudo du postgres-data/
4	postgres-data/main/pg_stat_tmp
20	postgres-data/main/pg_xact
4	postgres-data/main/pg_serial
4	postgres-data/main/pg_logical/mappings
4	postgres-data/main/pg_logical/snapshots
16	postgres-data/main/pg_logical
184	postgres-data/main/pg_subtrans
632	postgres-data/main/global
4	postgres-data/main/pg_commit_ts
4	postgres-data/main/pg_snapshots
4	postgres-data/main/pg_tblspc
12	postgres-data/main/pg_multixact/offsets
12	postgres-data/main/pg_multixact/members
28	postgres-data/main/pg_multixact
4	postgres-data/main/pg_wal/archive_status
1048584	postgres-data/main/pg_wal
1415201364	postgres-data/main/base/16385
7852	postgres-data/main/base/1
187386284	postgres-data/main/base/pgsql_tmp
7844	postgres-data/main/base/13462
7852	postgres-data/main/base/13463
1602611200	postgres-data/main/base
12	postgres-data/main/pg_notify
4	postgres-data/main/pg_stat
4	postgres-data/main/pg_dynshmem
4	postgres-data/main/pg_replslot
4	postgres-data/main/pg_twophase
1603660732	postgres-data/main
1603660736	postgres-data/
renderaccount@ellord-Precision-T5600:/home3$

 

polygon  까지 합하면 2TB 가까이 예상됨.