OpenSeaMap-dev Diskussion:Downloadable Raster Charts
Inhaltsverzeichnis
inland waterways (ideas)
web frontend
design (current)
- only zoom levels 7 - 15
- only corridor of tiles alongside waterways
- saving bandwidth, tiles not showing a navigable waterway are likely missing (eg. lakes beyond fairways)
workflow (not yet automated)
- query agains my EU-CEMT mbtiles-db to get tilenames
- expand tilenames-set to a corridor
- download tiles
- pack tiles into mbtiles-files
- upload mbtiles-files
bluewater charts (ideas)
web frontend
- die INT-chart Grenzen (als Basis fuer ein download-tool) bekomme ich inzwischen auf den Schirm: http://kap.grade.de/int-chart-ol2.html , sicherlich verbesserungsbeduerftig
- workflow
- INT-Chart durch klick auf die Karte waehlen
- download iniziieren
- Weiterleitung auf neue Seite
- pruefen, ob und in welchem Format vorhanden
- wenn ja und nicht zu alt, downloadlinks anbieten
- wenn nein Job in queue-db schreiben
- number, name, bbox, scale, sessionid, ip-address, email, microtime, status=open, ...
- das ganze garnieren mit
- Pruefung, ob das script fuer diese karte schon in der Warteschlange ist
- Limitierung auf x downloads/jobs pro session ...
- email notifier
- captcha
prerequisites
- apache
- php
- sqlite
tile server
- cronjob checkt queue-db alle x Minuten
- waehlt aeltesten job aus queue-db mit status=open
- tiledownload ueber proxy mit landez
- sinnvolles zoom-level anhand bbox errechnen (max. 10MB/chart)
- mbtiles erzeugen (siehe auch mbtiles implementation)
- stitchen und .png/.jpg erzeugen
- .kap mit imgkap erstellen
- ...
- status=done, zoom-level, date, ... in queue-db setzen
- temp-dir bereinigen und results in charts-dir kopieren
- erste Ergebnisse meiner Versuche: kap.grade.de
prerequisites
py script
- relies on landez
import logging
import shutil
import math
from landez import MBTilesBuilder, ImageExporter, TilesManager
import sqlite3
import os
import sys
import codecs
logging.basicConfig(level=logging.DEBUG)
# to be imported from sqlite-queue
con = sqlite3.connect('queue.sqlite')
with con:
con.row_factory = sqlite3.Row
cur = con.cursor()
cur.execute('SELECT * FROM jobs WHERE status="open" ORDER BY microtime DESC')
rows = cur.fetchall()
if len(rows) == 0:
logging.debug(" Nothing to do, exiting ")
sys.exit(0)
for row in rows:
sqMicrotime = row["Microtime"]
sqChart = row["Chart"]
sqName = row["Name"]
sqHO = row["HO"]
sqBottomGeoPre = row["BottomGeoPre"]
sqTopGeoPre = row["TopGeoPre"]
sqLeftGeoPre = row["LeftGeoPre"]
sqRightGeoPre = row["RightGeoPre"]
sqNumTiles16 = row["NumTiles16"]
# to be calculated from numtiles16, numtiles < 700
# numtiles16 is right now hardcoded at int-chart.gejson for ol depth of view reasons, there might be smarter ways in future
# https://github.com/makinacorpus/landez/issues/44
myNumTiles = sqNumTiles16 * 4 * 4
myZoom = 18
while (myNumTiles > 700):
myNumTiles = myNumTiles / 4
myZoom = myZoom - 1
# settings
# myUrlBase = "http://tiles.grade.de/tiles.py/openriverboatmap/{z}/{x}/{y}.png"
myUrlBase = "http://otile1.mqcdn.com/tiles/1.0.0/map/{z}/{x}/{y}.jpg"
myUrlSeamark = "http://tiles.grade.de/tiles.py/seamark/{z}/{x}/{y}.png"
if sqHO == "DE":
myUrlHydro = "http://tiles.grade.de/tiles.py/NAUTHIS_SkinOfEarth/{z}/{x}/{y}.png"
elif sqHO== "NL/GB":
myUrlHydro = "http://tiles.grade.de/tiles.py/ncp_diepte14/{z}/{x}/{y}.png"
else:
myUrlHydro = ""
myFileName = str(sqMicrotime)
myMbtilesFile = myFileName + ".mbtiles"
myImageFile = myFileName + ".jpg"
myKapFile = myFileName + ".kap"
myTextFile = myFileName + ".txt"
myBbox = (sqLeftGeoPre, sqBottomGeoPre, sqRightGeoPre, sqTopGeoPre)
myZoomLevels = [myZoom]
myZoomLevel = myZoom
myTileSize = 256
#myTileFormat = 'image/png'
myTileFormat = 'image/jpeg'
# download and mbtiles build
mb = MBTilesBuilder(tiles_url=myUrlBase, cache=True, filepath=myMbtilesFile, tile_size=myTileSize, tile_format=myTileFormat)
mb.add_coverage(myBbox, myZoomLevels)
if myUrlHydro != "":
hydro = TilesManager(tiles_url=myUrlHydro)
seamark = TilesManager(tiles_url=myUrlSeamark)
hydro.add_layer(seamark)
overlay = hydro
else:
seamark = TilesManager(tiles_url=myUrlSeamark)
overlay = seamark
mb.add_layer(overlay)
mb.run(force=True)
# export image from mbtiles
# not the best idea: causes IO at temp dir, again
ie = ImageExporter(mbtiles_file=myMbtilesFile)
ie.export_image(myBbox, myZoomLevel, imagepath=myImageFile)
# save the geoinfo
myGrid = ie.grid_tiles(myBbox, myZoomLevel)
sqWidth = len(myGrid[0])
sqHeight = len(myGrid)
sqWidthPix = sqWidth * myTileSize
sqHeightPix = sqHeight * myTileSize
# delete temp dir (ImageExporter does not!)
# bad: path is hard coded. how do I get hold of landez's temp path?
# https://github.com/makinacorpus/landez/issues/42
try:
shutil.rmtree('/tmp/landez')
except OSError:
pass
# geoinfo output
logging.debug(" grid: " + str(myGrid))
myLeftColumn = myGrid[0]
myTopLeft = myLeftColumn[0]
sqLeftTile = myTopLeft[0]
sqTopTile = myTopLeft[-1]
myRightColumn = myGrid[-1]
myBottomRight = myRightColumn[-1]
sqRightTile = myBottomRight[0]
sqBottomTile = myBottomRight[-1]
# calculate coords from tilenumbers
def num2deg(xtile, ytile, zoom):
n = 2.0 ** zoom
lon_deg = -180.0 + 360.0 * xtile / n
lat_rad = math.atan(math.sinh(math.pi * (1.0 - 2.0 * (ytile / n))))
lat_deg = 180.0 * lat_rad / math.pi
return (lon_deg, lat_deg)
sqBottomGeoPost = (num2deg(sqRightTile + 1, sqBottomTile + 1, myZoom)[-1])
sqTopGeoPost = (num2deg(sqLeftTile, sqTopTile, myZoom)[-1])
sqLeftGeoPost = (num2deg(sqLeftTile, sqTopTile, myZoom)[0])
sqRightGeoPost = (num2deg(sqRightTile + 1, sqBottomTile + 1, myZoom)[0])
# shall be exported to sqlite-queue
logging.debug(" baselayer: " + str(myUrlBase))
logging.debug(" seamark: " + str(myUrlSeamark))
logging.debug(" hydro: " + str(myUrlHydro))
logging.debug(" zoom: " + str(myZoom))
logging.debug(" width: " + str(sqWidth) + " tiles")
logging.debug(" height: " + str(sqHeight) + " tiles")
logging.debug(" widthpix: " + str(sqWidthPix) + " pixel")
logging.debug(" heightpix: "+ str(sqHeightPix) + " pixel")
logging.debug(" left tile " + str(sqLeftTile))
logging.debug(" right tile: " + str(sqRightTile))
logging.debug(" top tile: " + str(sqTopTile))
logging.debug(" bottom tile: "+ str(sqBottomTile))
logging.debug(" left: " + str(sqLeftGeoPost))
logging.debug(" right: " + str(sqRightGeoPost))
logging.debug(" top: " + str(sqTopGeoPost))
logging.debug(" bottom: " + str(sqBottomGeoPost))
sqPost = [
'UrlBase="' + str(myUrlBase) + '"',
'UrlSeamark="' + str(myUrlSeamark) + '"',
'UrlHydro="' + str(myUrlHydro) + '"',
'Zoom=' + str(myZoom),
'BottomGeoPost='+ str(sqBottomGeoPost),
'TopGeoPost='+ str(sqTopGeoPost),
'LeftGeoPost='+ str(sqLeftGeoPost),
'RightGeoPost='+ str(sqRightGeoPost),
'status="done"'
]
t = ", "
sqPost = t.join(sqPost)
sqlUpdate = "UPDATE jobs " + "SET " + str(sqPost) + " WHERE Microtime=" + str(sqMicrotime)
logging.debug(" sqlUpdate: " + str(sqlUpdate))
con = sqlite3.connect('queue.sqlite')
with con:
con.row_factory = sqlite3.Row
cur = con.cursor()
cur.execute(sqlUpdate)
con.commit()
logging.debug(" Number of rows updated: " + str(cur.rowcount))
# start imgkap to generate .kap file
import subprocess
myImgkapCall = "./imgkap " + myImageFile + " " + str(sqTopGeoPost) + " " + str(sqLeftGeoPost) + " " + str(sqBottomGeoPost) + " " + str(sqRightGeoPost) + " " + myKapFile
print myImgkapCall
subprocess.call(myImgkapCall, shell=True)
# write textfile
txtPost = [
'These charts cannot replace official charts, use at your own risk!',
'',
'INT Chart Number= ' + sqChart,
'INT Chart Name= ' + sqName,
'',
'Copyright Notices:',
'Base Layer= http://developer.mapquest.com/de/web/products/open/map',
'Seamark Layer= http://www.openseamap.org/',
'DE Hydrography= http://gdiwiki.bsh.de/wiki/index.php/NAUTHIS_Skin_Of_The_Earth',
'NL Hydrography= http://www.rijkswaterstaat.nl/zakelijk/databestanden/online_geografische_gegevens_rijkswaterstaat/',
'',
'Zoom Level= ' + str(myZoom),
'',
'Bottom= '+ str(sqBottomGeoPost),
'Top= '+ str(sqTopGeoPost),
'Left= '+ str(sqLeftGeoPost),
'Right= '+ str(sqRightGeoPost),
'',
'width= ' + str(sqWidth) + ' tiles',
'height= ' + str(sqHeight) + ' tiles',
'width= ' + str(sqWidthPix) + ' pixel',
'height= ' + str(sqHeightPix) + ' pixel',
'',
'left tile number= ' + str(sqLeftTile),
'right tile number= ' + str(sqRightTile),
'top tile number= ' + str(sqTopTile),
'bottom tile number= '+ str(sqBottomTile)
]
t = "\n"
txtPost = t.join(txtPost)
logging.debug(" txtPost: " + txtPost)
with codecs.open(myTextFile, 'w', "utf-8") as f:
f.write(txtPost)
# move and rename files
myPath = "../INT-CHART/INT-"
myNotice = "_MapQuest-OSeaM-BSH"
myNewMbtilesFile = myPath + sqChart + myNotice + ".mbtiles"
myNewImageFile = myPath + sqChart + myNotice + ".jpg"
myNewKapFile = myPath + sqChart + myNotice + ".kap"
myNewTextFile = myPath + sqChart + myNotice + ".txt"
try:
os.rename (myMbtilesFile, myNewMbtilesFile)
os.remove(myMbtilesFile)
except OSError:
pass
try:
os.rename (myImageFile, myNewImageFile)
os.remove(myImageFile)
except OSError:
pass
try:
os.rename (myKapFile, myNewKapFile)
os.remove(myKapFile)
except OSError:
pass
try:
os.rename (myTextFile, myNewTextFile)
os.remove(myTextFile)
except OSError:
pass
php script
<?php
error_reporting(E_ALL);
show_array($_GET);
show_array($_POST);
switch ($_POST['stage'])
{
case 'process':
update();
break;
default :
print_list();
break;
}
function print_form()
{
echo '<form action="'.$_SERVER['PHP_SELF'].'" method="post">';
echo " <input type='submit' name='Text' value='Generate new Chart'/>";
echo " <input type='hidden' name ='stage' value='process' />\n";
echo " <input type='hidden' name ='Chart' value='" . $_GET['Chart'] . "' />\n";
echo " <input type='hidden' name ='Name' value='" . $_GET['Name'] . "' />\n";
echo " <input type='hidden' name ='HO' value='" . $_GET['HO'] . "' />\n";
echo " <input type='hidden' name ='BottomGeoPre' value='" . $_GET['BottomGeoPre'] . "' />\n";
echo " <input type='hidden' name ='TopGeoPre' value='" . $_GET['TopGeoPre'] . "' />\n";
echo " <input type='hidden' name ='LeftGeoPre' value='" . $_GET['LeftGeoPre'] . "' />\n";
echo " <input type='hidden' name ='RightGeoPre' value='" . $_GET['RightGeoPre'] . "' />\n";
echo " <input type='hidden' name ='NumTiles16' value='" . $_GET['NumTiles16'] . "' />\n";
echo "</form>\n";
}
function print_list()
{
if (is_array(glob("../INT-CHART/INT-".$_GET['Chart']."*.*"))) {
foreach (glob("../INT-CHART/INT-".$_GET['Chart']."*.*") as $filename) {
$size = round(filesize($filename)/1024, 2);
$modified = gmdate ( 'd M Y H:i:s', filectime($filename));
echo "<a href='".$filename."'>" .$filename. "</a> - Size: " . $size . " kB - Modified: " . $modified . "</br>";
}
} else {
print_form();
}
}
function update()
{
echo 'Received your request, processing will take some time. Please be patient and check back in an hour ...';
$_POST['Name'] = utf8_encode( $_POST['Name'] );
$db = new SQLite3('../temp/queue.sqlite');
$db->exec("INSERT INTO jobs (
Chart,
Name,
HO,
BottomGeoPre,
TopGeoPre,
LeftGeoPre,
RightGeoPre,
NumTiles16,
status
) VALUES (
'{$_POST['Chart']}',
'{$_POST['Name']}',
'{$_POST['HO']}',
{$_POST['BottomGeoPre']},
{$_POST['TopGeoPre']},
{$_POST['LeftGeoPre']},
{$_POST['RightGeoPre']},
{$_POST['NumTiles16']},
'open'
)");
}
// show content of PHP array, i.e. show_array($_POST)
// this is handy to quickly test what data is in an array
function show_array($x)
{
if (!is_array($x)) return;
reset($x);
echo "<br />";
echo "<b>content of array</b>\n";
if ($x)
while ($i = each($x))
echo "<br />", htmlentities($i[0]), " = ", htmlentities($i[1]), "\n";
echo "<br /><br />\n";
}
?>
Reason for ChartBundler
- Lets talk a bit about amounts here and not low stress or high stress. Using the charts offline on a mobile device. Lets say the display is about 1024x1024 Pixel, then we will need at least 16 tiles per map. If we start on zoom level 8 with the bounding box and continue to zoom level 16 for the harbours, we need around 2 Mio tiles. This 16 tiles on zoom level 8 is about the northern Adria. If we want the Channel just skipping Lands End up to the belgian coast, this would be 6 by 5 on zoom level 8. --Alexej
- Hi Alexej, do not understand this paragraph!? 2 test datasets of mine:
- --Kannix (Diskussion) 16:22, 29. Jan. 2015 (UTC)
