G2S: The GeoStatistical Server

A free and flexible multiple point (geo)statistics framework including state-of-the-art algorithms:
QuickSampling and Narrow Distribution Selection

View project on GitHub

Final version of QuickSampling available here.

Brief overview

The GeoStatistical Server (G2S) is a framework that allows you to use state-of-the-art Multiple Point Statistics (MPS) algorithms to run stochastic simulations.

G2S is designed to run simulations in a generic way, independently of the code used or language it is written in. For example, it enables to run a C/C++ simulation code using Python, or Python using MATLAB (or any other combination)

Currently, the framework is provided with:

  • QuickSampling (QS) (aka. Quantile Sampling) is a general-purpose pixel-based MPS algorithm, that is designed to be robust, efficient and run in constant time. QS was designed to adapt to your problem, it can be used to do (un)conditional simulation, gap filling or even downscaling, using continuous or categorial variable or a combination of both. The code was developed without restrictions regarding the dimentionality of the data (e.g. 1D, 2D, 3D, nD).
  • A journal article published in Geoscientific Model Development describes the method here.
  • Narrow Distribution Selection (NDS) is an algorithm specifically targeted to simulate spectrally enhanced remote sensed imagery. It requires an external variable (for example, a grayscale image) to control the simulation (of colors)
  • ➔ A paper describing it is available here: Analogue-based colorization of remote sensing images using textural information.

The framework can be easily extended to handle most codes that use gridded data. Currently any compiled code or python code can be handled.

Installation

Because life is short, we designed tools to make it simple. 😉
Except if you plan to run simulations on a remote computer, you need to install both the server and the interfaces on the same machine. Please choose the options below regarding your needs.

Installation of the server

Installation of the server on Ubuntu

Automatic with GCC on Ubuntu

  1. First clone the code from this GitHub.
    git clone https://github.com/GAIA-UNIL/G2S.git
  2. run build/c++-build/install_needs_W_VM.sh

Manual (included Intel C++ Compiler)

  1. First clone the code from this GitHub.
    git clone https://github.com/GAIA-UNIL/G2S.git
  2. Basics for compiling are needed, (e.g. on Ubuntu: build-essential).
  3. The following packages are required: ZMQ, JsonnCpp and zlib for G2S. fftw3 for QS and NDS.
  4. To install them on Ubuntu:sudo apt install build-essential libzmq3-dev libjsoncpp-dev zlib1g-dev libfftw3-dev libcurl4-openssl-dev -y(libcurl4-openssl-dev is optional)
  5. The C++ extension for ZMQ is required too, and can be installed via: wget "https://raw.githubusercontent.com/zeromq/cppzmq/master/zmq.hpp" -O ./include/zmq.hpp
  6. Go to the build subfolder.
  7. Run: make -j, if the Intel C++ compiler is installed, the adaped version will be compiled too. The Intel compiler can be downloaded freely in many cases: here.
    To manualy select between GCC or Intel compiler use make c++ -j or make intel -j, respectively.

Installation of the server on macOS

using MacPort

  1. First clone the code from this GitHub.
    git clone https://github.com/GAIA-UNIL/G2S.git
  2. Install the package manager MacPort (if not already done)
  3. The following packages are required: ZMQ, JsonnCpp and zlib for G2S. fftw3 for QS and NDS.
  4. To install them with macPort:sudo port install zmq-devel jsoncpp-devel zlib cppzmq-devel fftw-3 fftw-3-single curl (curl is optional)
  5. Go to the build subfolder.
  6. Run: make -j, if the Intel C++ compiler is installed the addaped version would be compiled too. The Intel compiler can be downloaded freely in many cases: here.
    To manualy select between GCC or Intel compiler use make c++ -j or make intel -j, respectively.

using Homebrew

  1. First clone the code from this GitHub.
    git clone https://github.com/GAIA-UNIL/G2S.git
  2. Install the package manager Homebrew (if not already done)
  3. The following packages are required: ZMQ, JsonnCpp and zlib for G2S. fftw3 for QS and NDS.
  4. To install them with macPort:brew install fftw zmq jsoncpp curl (curl is optional)
  5. The C++ extension for ZMQ is required too, and can be installed via: curl -o /usr/local/include/zmq.hpp "https://raw.githubusercontent.com/zeromq/cppzmq/master/zmq.hpp"
  6. Go to the build subfolder.
  7. Run: make -j, if the Intel C++ compiler is installed the addaped version would be compiled too. The Intel compiler can be downloaded freely in many cases: here.
    To manualy select between GCC or Intel compiler use make c++ -j or make intel -j, respectively.

Installation of the server on Windows 10

  1. Check that the last updates of Windows are installed
  2. Install WSL follwing instructions here, and select an Linux distribution (we advise Ubuntu for beginners).
  3. Go to the directory build/c++-build
  4. Run install.bat

Installation of interfaces

Installation of the Python interface on Ubuntu

Automatic installation

Use simply pip install G2S

Manual compilation

  1. If needed Python and Numpy: sudo apt install python3-distutils python3-dev python3-numpy -y
  2. (A C++ compiler with c++17 is required)
  3. Go to build/python-build
  4. Run python3 setup.py install --user

Check proper interface installation

Simply run
from g2s import g2s; g2s('--version')

Installation of the MATLAB interface on Ubuntu

  1. (A C++ compiler with c++17 is required)
  2. Open MATLAB
  3. Go to build/build-matlab
  4. Run CompileG2S
  5. Add compiled file in the MATLAB path

Check proper interface installation

Simply run g2s('--version')

Installation of the R interface on Ubuntu

Installation of the Python interface on macOS

Automatic installation

Use simply pip install G2S

Manual compilation

  1. If needed Python and Numpy:sudo port install python37 py37-numpy
  2. Go to build/python-build
  3. Run python3 setup.py install --user

Check proper interface installation

Simply run
from g2s import g2s; g2s('--version')

Installation of the MATLAB interface on macOS

  1. Open MATLAB
  2. Go to build/build-matlab
  3. Run CompileG2S
  4. Add compiled file in the MATLAB path

Check proper interface installation

Simply run g2s('--version')

Installation of the R interface on macOS

Installation of the Python interface on Windows 10

Automatic installation

Use simply pip install G2S

Manual compilation

  1. (A C++ compiler with c++17 is required)
  2. If needed, install python with the option to add it to the Path
  3. Go to build/python-build
  4. Run python3 setup.py install

Check proper interface installation

Simply run
from g2s import g2s; g2s('--version')

Installation of the MATLAB interface on Windows 10

Downlod precompiled interfaces

Download here.
Unzip and add the folder to MATLAB path.

Manual compilation

  1. If needed install python with the option to add it to the Path
  2. Open MATLAB
  3. Install a compiler with c++17, available here (2017 or later)
  4. Go to build/build-matlab
  5. Run CompileG2S
  6. Add compiled file in the MATLAB path

Check proper interface installation

Simply run g2s('--version')

Installation of the R interface on Windows 10

Run the server

⚠️The server generates logs- and data-files in subfolder build/build-*/logs and build/build-*/data that are originally saved for debug purpose, and are currently automatically removed only at the launch of the server or after one day. This parameterization can be changed with -kod and -age

Run the server on Ubuntu

Run ./server in build/c++-build or build/intel-build, respectively for the standard or Intel version

Run the server on macOS

Run ./server in build/c++-build or build/intel-build, respectively for the standard or Intel version

Flag Description
‑d run as daemon
‑To n shutdown the server if there is no activity after n second, 0 for ∞ (default : 0)
‑p select a specific port for the server (default: 8128)
‑kod keep old data, if the flag is not present all files from previous runs are erased at launch
‑age set the time in second after files need to be removed (default: 86400 s = 1d)
‑mT single job at the time (experimental)
‑fM Run as function, without fork (experimental)

Run the server on Windows 10

It is possible to run the server with runServer.bat or runDaemon.bat as daemon available in build/c++-build

Using the interfaces

A call to g2s is needed to launch any computation. Parameters do NOT have a specific order.

Use g2s in Python

from g2s import g2s
data=g2s(...) # it returns an tuple that contains all the output maps and the computing duration

Use g2s in MATLAB

data=g2s(...)		% principal output, the simulation
[data, t]=g2s(...) % the simulation and the computation time
[data, ...,t]=g2s(...) % the simulation, other outputs map and the computation time

Use g2s in R

<

Each call to g2s is composed of parameters of G2S and of the name of the algorithm used.

Flag Description
‑‑version return the version and compilation date of the interface
‑a the algorithm, it can be 'qs', 'nds', 'ds-l'
‑sa address of the computation server (default: localhost, the server is local)
Nice when we have a powerful machine dedicated for computation
‑silent don't display the progression, nice for scripts
‑serverStatus inform if the server is working properly
<1 → error, such as comunication, server shutdown,...)
=0 → indefined
>1 → server is operational
  • 1: standard server working normaly
  • (more is comming soon!)
‑noTO deactivate timeout on ZMQ communication, useful for slow network (e.g., trough internet)
‑TO specify a custom timeout on ZMQ communication( uint32 in millisecond )
‑shutdown shutdown the server, useful at the end of scripts

The following options represent the Async mode, check example in MATLAB and Python directory:

Flag Description
‑submitOnly submit a job
‑statusOnly check progression
‑waitAndDownload download the result
‑kill kill a given task
‑after execute the task after an other on is finished (e.g. '-after',previousTaskId )

Simulation algorithms

G2S is provided with standard state-of-the-art algorithms:

  • QS (aka. QuickSampling)
  • NDS (aka. Narrow Distribution Selection)

Paremeter for QS

Flag Description Mandatory
‑ti Training images (one or more images). If multiple variables, the variables are not automatically normalized to be in the same range. Either normalize them beforehand, or use the kernel to normalize them.
‑di Destination image (one image, aka simulation grid)
‑dt Data type, array of to specify the type of each variable, 0 → continuous and 1 → categorical
‑sp simulation path, image of the position in the path, small values first, -∞ are not simulated
‑ki Image of weighting kernel. Can be used to normalize the variables.
‑k Number of best candidates to consider ∈[1 ∞]
‑f f=1/k equivalent to f of DS with a threshold to 0
‑n N closest neighbors to consider:
- single value for vectors neighbors
- one for each variable
‑s random seed value
‑j to run in parallel (if specified), to use as follows '-j', N1, N2, N3. Use decimal values ∈]0,1[ for fraction of the maximum number of cores (e.g., 0.5 repreent 50% of all available logical cores).
- N1 threads used to parallelize the path (path-level) Default: the maximum number of threads available.
- N2 threads used to parallelize over training images (node-level), work only if many TIs are available. Default: 1
- N3 threads used to parallelize FFTs (path-level). Default: 1
- Favorize N1 and N2 over N3, N1 is usually more efficient than N2, but require more memory.
‑ii index of trining in with to sample from (experimental)
‑cti for a circular/cyclic/periodic training images over each dimention
‑csim for a circular/cyclic/periodic simulation over each dimention
‑adsim augmented dimentionality simulation, allow to do 3D simulation using 2D training image, only for categories (Comming soon!)
‑far fast and risky 😄, like -ii but with a random input (experimental)
‑W_GPU use integrated GPU if available
‑W_CUDA use Nvidia Cuda compatible GPU, specify the device id. (e.g., '‑W_CUDA',0,1 to use the two first GPU)
‑fs full simulation, run the simulation of each pixel independentantly, i.e. dosen't simulate all the vector at the time (experimental)
‑nV no Verbatim (experimental)

Paremeter for NDS

Flag Description Mandatory
‑ti Training images (one or more images). If multiple variables, the variables are not automatically normalized to be in the same range. Either normalize them beforehand, or use the kernel to normalize them.
‑di Destination image (one image, aka simulation grid)
‑dt Data type, array of to specify the type of each variable, 0 → continuous and 1 → categorical
‑ki Image of weighting kernel. Can be used to normalize the variables.
‑k Number of best candidates to consider to compute the narrowness ∈[5 ∞]
‑n N closest neighbors to consider
‑nw narrowness range 0→ max-min, 1 → median, default IQR → 0.5
‑nwv number of variables to consider in the narrowness, (start from the end), default: 1
‑cs chunk size, the number of pixels to simulate at the same time, at each iteration, default: 1
‑uds area to update around each simulated pixel, the M closest pixel default: 10
‑mp partial simulation, 0 → empty, 1 → 100%
‑s seed value
‑j to run in parallel (if specified), to use as follows '-j', N1, N2, N3
- N1 threads used to parallelize the path (path-level) Default: the maximum number of threads available.
- N2 threads used to parallelize over training images (node-level), work only if many TIs are available. Default: 1
- N3 threads used to parallelize FFTs (path-level). Default: 1
- Favorize N1 and N2 over N3, N1 is usually more efficient than N2, but require more memory.
‑W_GPU use integrated GPU if available
‑nV no Verbatim (experimental)

Example

Example of using QS

Unconditional simulation using Python

Unconditional simulation using MATLAB

Conditional simulation using Python

Conditional simulation using MATLAB

Gap filling using Python

Gap filling using MATLAB

Downscaling using Python

Downscaling using MATLAB


Example of using NDS

Spectral ehancemenet: RGB⇒IR using Python

Spectral ehancemenet: RGB⇒IR using MATLAB

Spectral ehancemenet: Pan⇒RGB using Python/h3>

Spectral ehancemenet: Pan⇒RGB using MATLAB

Spectral ehancemenet: Pan⇒IR using Python

Spectral ehancemenet: Pan⇒IR using MATLAB

Benchmarking

This code CAN be used for benchmarking (and I invite you to do so 😉), the code needs to run natively on macOS or on Linux using the Intel Compiler with MKL library. The version needs to be reported, and the time needs to be the time reported by the algorithm (that is the true execution time without taking into account interfaces overhead).

When benchmarking, the code should NOT be used inside a Virtual Machine or trough WSL on Windows 10.

How to cite ?

For QuickSampling

Gravey, M., & Mariethoz, G. (2020). QuickSampling v1.0: a robust and simplified pixel-based multiple-point simulation approach. Geoscientific Model Development, 13(6), 2611–2630. https://doi.org/10.5194/gmd-13-2611-2020

For Narrow Distribution Selection

Gravey, M., Rasera, L. G., & Mariethoz, G. (2019). Analogue-based colorization of remote sensing images using textural information. ISPRS Journal of Photogrammetry and Remote Sensing, 147, 242–254. https://doi.org/10.1016/j.isprsjprs.2018.11.003