postgresql replication, load balancing and SPOFs

here's article for setting up two pgpools - so no single point of failure (thanks to Nozomi Anazai)
http://www.pgpool.net/pgpool-web/contrib_docs/watchdog/en.html

here's an article how to set up pgpool to handle replication and load balancing of a postgresql cluster (thanks to Tatsuo Ishii)
http://pgpool.projects.postgresql.org/contrib_docs/simple_sr_setting/index.html

And here's a great presentation from Tom Fry @ OrdnanceSurvey on their setup (spatial databases this time) using pgPool. No replication duties in this case I think as it's only handling select duties. Here.

Show databases and tables in postgresql

show databases:
select datname from pg_database;
show tables:
\dt

postgres ssh-copy-id: ERROR: No identities found

I'm working on hot_standby with postgresql on centos. One thing I got stuck on being a noob at linux was how to create ssh keys for the postgres account. I was following the 3dtin blog on postgresql hot_standby and got stuck at the following point; 
su - postgres 
ssh-copy-id 
ssh-copy-id: ERROR: No identities found 

Ok, remove that pesky selinux. I believe that's at fault. Set SELINUX in /etc/sysconfig/selinux to disabled. 
su - postgres; pwd should be /var/lib/pgsql 
mkdir .ssh 
ssh-keygen (hit defaults) 
ssh-copy-id -i ~./ssh/id_rsa.pub 
check with ssh So if you do that on the pg_master and drop the key on the pg_slave, you should then repeat the other way (as the 3dtin blog says).

Cluster your PostGIS

I've been asked to look at clustering work's PostGIS needs. We've got some OSM which is only used for Mapnik to read and render from so I'm going to ignore that for now. I'm more interested in setting up a PostGIS cluster for user-created content.

I've found a great PDF presentation from Steve Bennett at the Ordnance Survery. Also a great, simple note on clustering Postgresql from Satoshi Nagayasu. I'm in the process of knocking up a simple test cluster so I'll report back.

Eggie5's run through of streaming replication looks a brief but solid guide to replication.

Jyro's tutorial here on 3DTin is wonderful. It pushed this fat-arsed noob all the way through so I've got a master/standby setup running. Something to play with and attempt to break - that's the point right? Tutorial here.

Loading PostGIS with ESRI Shapefiles

Note to self:
Use shp2pgsql to create CREATE/INSERT sql;
shp2pgsql -c -D -s 4326 -I <shp file> <table name> > <file output name>.sql
and then follow it up with the lovely;
psql -d <database name> -U <user name> -f <file output name>.sql
This way I've got all that lovely SQL to store somewhere if I need it.

How to get an up-to-date OSM in PostGIS

How to build an OSM database that is up to date. 

Rather than doing a quarterly manual job of downloading a 14Gb+ file, decompressing it (250Gb+) and inserting it into a database it will be a lot easier to download daily (50Mb) OSM change files (.osc) and apply the changes to the existing database. 

The instructions below is a modified version of Martin van Exel's tutorial here - https://docs.google.com/document/pub?id=1paaYsOakgJEYP380R70s4SGYq8ME3ASl-mweVi1DlQ4 

# get osmosis
su cd /opt 
wget http://dev.openstreetmap.org/~bretth/osmosis-build/osmosis-latest.tgz 
tar zxvf osmosis-latest.tgz
rm -rf osmosis-latest.tgz
# in my case osmosis-0.41 

# now pop into PostgreSQL and build a suitable database & user (could be tidier)
adduser osm 
passwd osm 
# and some badly chosen password: osm, osm
su - postgres psql 
createdb osm
createlang plpgsql osm
CREATE USER osm WITH PASSWORD 'osm';
#CREATE DATABASE osm;
GRANT ALL PRIVILEGES ON DATABASE osm to osm;
GRANT ALL PRIVILEGES ON DATABASE osm to postgres;
\q
psql -d osm -U osm -f /usr/pgsql-9.1/share/contrib/postgis-2.0/postgis.sql 
psql -d osm -U osm f /usr/pgsql-9.1/share/contrib/postgis-2.0/spatial_ref_sys.sql
psql -U osm -d osm
CREATE EXTENSION hstore;
\q
psql -U osm -d osm -f /home/<user name>/osm/osmosis-0.41/script/pgsimple_schema_0.6.sql

# make some directories and get some data
su mkdir /tmp/osm 
cd tmp/osm 
mkdir planet
mkdir planet/replication

# now get some OSM data - either get the whole planet.. or get a sample from http://download.geofabrik.de/osm/ 
cd /tmp/osm/planet/
wget http://planet.openstreetmap.org/planet-latest.osm.bz2

# and thump into the database (using the VERY IMPORTANT --slim option, -C is how much memory we're setting aside for this: 4Gb in this case) /opt/osm2pgsql/osm2pgsql -S default.style --slim -d osm -C 4000 /tmp/osm/planet/planet-latest.osm.bz2 

# create the configuration.txt and download.lock 
osmosis --rrii workingDirectory=/tmp/osm/planet/replication 

# create your state.txt file, visit http://toolserver.org/~mazder/replicate-sequences/ and enter details. To check the datetime required examine the contents of the bz2 file. 

bunzip2 -c planet-latest.osm.bz2 | head

Now at this point I've been running osmosis to grab the change file and osm2pgsql to throw it at the database but you can apparently run osmosis on it's own;

# Osmosis on it's own version
/opt/osmosis-0.41/bin/osmosis --rri workingDirectory=/tmp/osm/planet/replication --sc --wpc user="osm" database="osm" password="osm"

# Or the two command route - grab a change file from the queue of necessary changes to perform
/opt/osmosis-0.41/bin/osmosis --rri workingDirectory=/tmp/osm/planet/replication --simplify-change --write-xml-change /tmp/osm/planet/replication/changes.osc.gz

# And append the changes to the database
/opt/osm2pgsql/osm2pgsql --append -S /opt/osm2pgsql/default.style -d osm -U postgres --slim /tmp/osm/planet/replication/changes.osc.gz 

# You can put either set of commands into a script and cron it up, here's the script for the osm2pgsql route

#!/bin/sh
n=`ps -ef | grep -v grep | grep /opt/osmosis-0.41/ | wc -l`
m=`ps -ef | grep -v grep | grep /opt/osm2pgsql/ | wc -l`
let i=n+m
if [ $i -gt 0 ]; then
echo osmosis or osm2pgsql running
else
echo not running
/opt/osmosis-0.41/bin/osmosis --rri workingDirectory=/tmp/osm/planet/replication --simplify-change --write-xml-change /tmp/osm/planet/replication/changes.osc.gz
/opt/osm2pgsql/osm2pgsql --append -S /opt/osm2pgsql/default.style -d osm -U postgres --slim /tmp/osm/planet/replication/changes.osc.gz 
fi

# Now you're probably running mod_tile and mapnik to render and serve up OSM tiles so we need a method to tell mod_tile to re-render all tiles that have updates. We can amend the osm2pgsql call with -e options to produce a list of tiles that need to be expired and thus re-rendered. [See OSM Tile Expire Methods].
# change the osm2pgsql command to read
/opt/osm2pgsql/osm2pgsql --append -S /opt/osm2pgsql/default.style -d osm -U postgres --slim /tmp/osm/planet/replication/changes.osc.gz -e15 -o expire.list

# We now expire all tiles listed in the expire.list. I haven't included the install and setup of Mapnik or mod_tile so i'm going to use some dummy locations for them. I'm expiring tiles from zoom level 6 and downwards
cat expire.list | /opt/mapnik/mod_tile/render_expired --map=osm --min-zoom=6 --touch-from=7 >/dev/null

# The OSM Tile Expire Methods guide also recommends a daily re-render of the lower zoom levels.

Installing PostGIS 2.0 & GeoServer on Centos 6 64bit

Installing PostGIS 2.0 and GeoServer on Centos 6 64bit

as root;
su
exclude=postgres* in base and update sections of /etc/yum.repos.d/CentOS-Base.repo
curl -O http://yum.postgresql.org/9.1/redhat/rhel-6-x86_64/pgdg-centos91-9.1-4.noarch.rpm
rpm -ivh pgdg-centos91-9.1-4.noarch.rpm
curl -O http://mirror.bytemark.co.uk/fedora/epel/6/x86_64/epel-release-6-7.noarch.rpm
rpm -ivh epel-release-6-7.noarch.rpm
yum install postgresql91-server postgis2_91
yum install postgresql91-contrib.x86_64

service postgresql-9.1 initdb

# edit postgres' conf
vi /var/lib/pgsql/9.1/data/postgresql.conf
add listen_addresses = '*'

# allow some connection, make sure the last one matches your local network if you're doing local network stuff...
vi /var/lib/pgsql/9.1/data/pg_hba.conf
local all all trust
host all all 127.0.0.1/32 trust
host all all 192.168.1.0/24 md5

# kick postgres
service postgresql-9.1 start

# run on startup
chkconfig postgresql-9.1 on

# change the password for the postgres account
passwd postgres

su postgres
psql
ALTER USER postgres WITH ENCRYPTED PASSWORD 'postgres';

# extensions time..
CREATE EXTENSION adminpack;

# spatialise your db
CREATE EXTENSION postgis;
CREATE EXTENSION postgis_topology;

\q
exit

# allow incoming connections on 5432 (postgresql standard port)
back to root;
add the following to /etc/sysconfig/iptables above any REJECT rule
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5432 -j ACCEPT

service iptables restart

# back to postgres' bash
sudo -i -u postgres
createdb template_postgis
createlang plpgsql template_postgis
psql -d template_postgis -f /usr/pgsql-9.1/share/contrib/postgis-2.0/postgis.sql
psql -d template_postgis -f /usr/pgsql-9.1/share/contrib/postgis-2.0/spatial_ref_sys.sql
psql -d template_postgis -f /usr/pgsql-9.1/share/contrib/postgis-2.0/postgis_comments.sql
psql
CREATE DATABASE my_spatial_db TEMPLATE template_postgis;
\q
exit

# back to root
# Installing Geoserver
# we need java, java -version says no java
yum install yum-priorities
rpm -Uvh http://apt.sw.be/redhat/el6/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
rpm -Uvh http://mirrors.dotsrc.org/jpackage/6.0/generic/free/RPMS/jpackage-utils-5.0.0-7.jpp6.noarch.rpm
Next we will install Java and Tomcat 6:
yum -y install java
yum -y install tomcat6 tomcat6-webapps tomcat6-admin-webapps
vi /etc/tomcat6/tomcat-users.xml
add a user with roles of admin,manager
chkconfig tomcat6 on
# now to geoserver
cd /home/<user>
wget http://downloads.sourceforge.net/geoserver/geoserver-2.1.4-war.zip
unzip geoserver-2.1.4-war.zip
mv geoserver.war /usr/share/tomcat6/webapps
# start up tomcat6
service tomcat6 startexit
# test geoserver at
http://localhost:8080/geoserver/web/

# and you'll probably want access across your network
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
/sbin/service iptables save
iptables -F

Changing an icon in a treepanel extjs 3

It took me a while to find a solution - probably a slow head day but here's how;
In this case a node represents a web mapping service. If the WMS is alive it should return a list of layers which are used to create a series of child nodes. So if no children change the icon to something relevant.

the css file;
.failedwms-icon {
background: url('resources/deadwms.gif') no-repeat;
height: 16px;
width: 16px;
}

in your js code;
'load': function(node){
// if a node receives a response and no layer child nodes are created we want to change the icon
if (node.childNodes) {
if (node.childNodes)
{
if (node.childNodes.length == 0)
{
node.attributes.iconCls = 'failedwms-icon';
node.getUI().iconNode.className = node.attributes.iconCls;
}
}
}
}

compiling OpenLayers

Grab it from their github: https://github.com/openlayers/openlayers/zipball/master

Drop closure-compiler.jar & compiler.jar into openlayers\tools
Go to \build and ...

build.py -c closure full OpenLayers.js for regular compressed OpenLayers (700kb odd)
buildUncompressed.py full OpenLayers-uncompressed.js for phat Openlayers (2.5Mb odd)

Popularity of WMS InfoFormats

I'm doing some development work on a web map that brings wms layers from a host of UK public wms providers. Part of the work is to provide an info tool and we're using OpenLayers getFeatureInfo control to do this.

The problem with working with so many external wms providers is the lack of consistency in info formats supported. A quick summary (below) shows that html and plain text are the most supported with ogc xml a reasonable third. So we'll be concentrating our support for these formats and hopefully, if time and money permits, to extend support to others. Not good findings for ogc gml as it comes last with only 11% support.

100% support: html & plain
89% support: ogc xml
67% support: xml
19% support: esri raw xml & esri feature info xml
11% support: ogc gml