Vision
The JRC Big Data Analytics Platform (BDAP) links data, data services, data scientists and thematic experts for generating policy relevant insights and foresight. It will play an instrumental role in advancing JRC to better mobilise and synthesise its collective knowledge and expertise in support to the EC priorities.
Building blocks
BDAP is build around a petabyte-scale storage system coupled with a processing cluster, accessible from anywhere through encrypted protocols and multi-factor authentication. It provides interactive data analysis tools, a remote data science desktop and distributed computing with specialized hardware for machine learning and deep learning tasks. It groups services for data analytics, data visualisation and data dissemination under the same platform.
Main services
All the BDAP services are built on top of the hardware layer, which consists of servers dedicated to storage and servers dedicated to processing. The multi-petabyte storage is managed by EOS, the distributed file system created by CERN, that allows all the services to see the full storage capacity as a single volume. Dedicated hardware for machine learning and deep learning is also available. On top of the hardware layer, three main services are built, as displayed in the following image:
Quick guide
A quick guide to BDAP and its services is available for downloading at the following link:
Documentation
User documentation on authentication methods and how to get your One Time Password and Certificates
Authentication One Time Password New certificate Renew certificateUser guide of the pyjeo library: how to manage geospatial data in JEO-desk and JEO-batch
User guide of the interapro library: how to visualise and analyse geospatial datasets in the JEO-lab Jupyter notebooks
Links to help pages of the main libraries to use to create graphical user interface elements in Jupyter notebooks and Voilà dashboards:
- Widgets GUI elements:
- ipywidgets ipyvuetify vuetifyjs
- Mapping:
- ipyleaflet
- Tabular data:
- qgrid
- Charting:
- bqplot plotly bokeh matplotlib
- Custom drawing:
- ipycanvas
- Hierarchical data display:
- ipytree
- Events management:
- ipyevents
How to access the platform
Various authentication services are used on the BDAP platform, depending on the specific service being accessed. All services require the user to have a BDAP account. Some services require additional security measures in order to be accessed. For those, user certificates and one time passwords are provided to authorised users.
Services
List of services provided by the JRC Big Data Analytics Platform:
Quick access to BDAP servicesThe JEO-lab is a Jupyter notebook environment, intended primarily to interactively analyse and visualize data via a dedicated API. In addition to the usage of the API, JEO-lab is also used for starting up machine learning notebooks or accessing project-specific notebook containers for selected use cases. JEO-lab documentation
The JEO-desk is a desktop terminal service that provides a graphical Linux remote desktop terminal (Xubuntu based) accessible from within a modern web browser with support for HTML5. This virtual terminal is provided from machines located within the JEODPP infrastructure and therefore with fast connection to the JEODPP data. JEO-desk documentation
The JEO-batch service provides computing power to the JRC experiments for tasks such as high scale image processing, data analysis and simulation. It is an advanced, distributed system designed mainly for high-throughput scientific computing. JEO-batch documentation
The JEO-cloud service is based on Nextcloud and is aimed at facilitating collaboration between JEODPP users and making easier the transfer of documents (scripts, project files etc.) and small datasets between the user personal computer and the JEODPP Terminal Service. It also provides some functionalities useful to provide remote support to JEODPP users by the JEODPP team (screen sharing and videoconferencing). JEO-cloud documentation
Voilà is a Jupyter notebook extension to automatically create standalone applications and dashboards. Notebooks are rendered by showing only the output of the cells, while the code is hidden. Suitable for non-technical experts for communicating insights and foresight to a wider audience. Single environment for full data analytics workflows from research and innovation to outreach engaging policy makers and citizens.
As an example of a Voilà dashboard, the CollectionsExplorer is a good starting point to explore the geospatial datasets stored in the platform.
GitLab is the DevOps platform used at BDAP. It helps BDAP team members and users to collaborate on software development and to provide a place where everyone can contribute. Users can add issues in the BDAP GitLab instance to ask for new features, to evaluate new dataset downloading, to request the installation of new software packages and to report bugs or problems in the BDAP services.
Data
Data provided by BDAP:
Web data catalogue for exploring and browsing all the collections stored in BDAP. It follows the SpatioTemporal Asset Catalog (STAC) specifications providing a common language to describe geospatial data.
Voila' application to browse, visualise and compare the main geospatial datasets available at BDAP.
Geospatial data browser for Web mapping applications using WMS standard and served by the Big Data Analytics Platform. Licenses for service data
Platform information
Timeline
BDAP is the successor of the JRC Earth Observation Data and Processing Platform, widening the scope of the platform towards any type of Big Data analytics. The commonly known name "JEODPP" is continued to be used in most documentations and services URL's. Here a timeline of the project progress:
-
1 January 2015Earth Observation and Social Sensing Big Data pilot projectKick-off of EO&SSBD as a pilot project
-
1 March 2016Purchase and installation of the first hardwareJRC Big Data Platform starts its operativity with the first servers installed and configured
-
1 January 2019Big Data Analytics project launched as an institutional projectJRC Big Data Platform ends the pilot phase and enters full institutional
-
12 December 2020Big Data Analytics Platform recognised as an official EC IT platformEC ITC and Cybersecurity Board (ITCB) provide a positive opinion on the evolution of the platform towards a Big Data Analytics Platform as a component of the EC Data Platform
Numbers
Software stack
The JRC Big Data Analytics Platform is mainly built on Open Source Software. Here a partial list of the tools and libraries used:
Publications
The publications listed here correspond to all publications registered in pubsy and containing at least one co-author from the Big Data Analytics Platform. Numerous publications are the result of fruitful collaborations with other JRC projects and external partners.
Books:
P. Soille, S. Loekken, and S. Albani, eds. Proc. of the 2021 Conference on Big Data from Space (BiDS’21). ESA-JRC-SatCen. Online event: Publications Office of the European Union, May 2021. doi: 10.2760/125905 P. Soille, S. Loekken, and S. Albani, eds. Proc. of the 2019 Conference on Big Data from Space (BiDS’19). ESA-JRC-SatCen. Munich, Germany: Publications Office of the European Union, Feb. 2019. doi: 10.2760/848593 P. Soille and P. Marchetti, eds. Proc. of the 2017 Conference on Big Data from Space (BiDS’17). ESA-JRC-SatCen. Toulouse, France: Publications Office of the European Union, Nov. 2017. doi: 10.2760/383579 P. Soille and P. Marchetti, eds. Proc. of the 2016 Conference on Big Data from Space (BiDS’16). ESA-JRC-SatCen. Tenerife, Spain: Publications Office of the European Union, Mar. 2016. doi: 10.2788/854791.Recent journal papers:
P. Kempeneers, O. Pesek, D. De Marchi, and P. Soille. “A Python Package For The Analysis of Geospatial Data”. International Journal of Geo-Information 8.10 (2019). doi: 10.3390/ijgi8100461 V. Syrris, P. Hasenohr, B. Delipetrev, A. Kotsev, and P. Soille. “Evaluation of the potential of convolutional neural networks and random forests for multi-class segmentation of Sentinel-2 imagery”. Remote Sensing 11.8 (2019), p. 4. doi: 10.3390/rs11080907 P. Soille, A. Burger, D. De Marchi, P. Kempeneers, D. Rodriguez, V. Syrris, and V. Vasilev. “A Versatile Data-Intensive Computing Platform for Information Retrieval from Big Geospatial Data”. Future Generation Computer Systems 81.4 (Apr. 2018), pp. 30–40. doi: 10.1016/j.future.2017.11.007. V. Syrris, C. Corbane, M. Pesaresi, and P. Soille. “Mosaicking Copernicus Sentinel-1 data at global scale”. IEEE Transactions on Big Data (2018). doi: 10.1109/TBDATA.2018.2846265. P. Kempeneers and P. Soille. “Optimizing Sentinel-2 image selection in a Big Data context”. Big Earth Data 1.1–2 (2017), pp. 145–158. doi: 10.1080/20964471.2017.1407489.Media gallery
-
CollectionsExplorer dashboard: browse the tree of JEO-lab datasets and easily compare two geospatial datasets -
Presentation of JEODPP activities for RTD EuroGeoss -
Voilà dashboard on Covid-19 spread in italian provinces -
Spread of Covid-19 in the regions of Europe -
California fires as seen from Sentinel-5P -
Voilà dashboard on EU budget expenditures -
Synthetic view of Covid-19 spread in European countries -
2019 Blue Mountains fires as seen from Sentinel-2 images in s2explorer -
Effects of lockdown measures in NO2 drop in Europe -
S2explorer application to monitor agricultural parcels
News
The scope of the JRC Big Data Analytics Platform (BDAP) User-Group Meetings is to: Bring together current and prospective BDAP users and team members; Present latest achievements and on-going developments; Discuss future new developments to better link data, data scientists, thematic and policy experts across different domains for generating policy relevant insights and foresight.
A maintenance of the Big Data Analytics Platform is planned on Thursday 10th of February at 8:00am CET. The intervention will last approximately the whole morning. We will be using this time to perform security updates on several components of the Big Data Analytics Platform and to perform exceptional maintenance operations on the EOS storage. During the maintenance window, the following services will not be available: JEO-batch, GitLab, Nextcloud, RocketChat. In addition, EOS storage access will continue to be available, but smaller interruptions could appear. This needs to be taken into account when using the storage on EOS during the intervention. All the services will be restored by 2:00pm.
All talks including keynote lectures will be available soon for asynchronous viewing with youTube. See Proceedings of BiDS'21 for the papers related to the presentations.
Presentation given at the JupyterCon 2020 on the Voilà dashboards created at the Big Data Analytics Platform for policy support
Contacts
- Email us
- JRC-JEODPP@ec.europa.eu
- Chat with us
- Chat with BDAP group or with other users
- Follow us
- Go to our Connected page
- Ask for support
- Create an issue in BDAP GitLab instance
- Follow issues
- by accessing BDAP GitLab instance or by email (reply to GitLab email notifications)
- Open a use case
- Create a new use case by filling and returning this Excel template
Team
BDAP is made by people who love IT and data science:
- Project leader:
- Pierre Soille
- Statutory staff:
- Armin Burger
- Pieter Kempeneers
- Davide De Marchi
- Paul Hasenohr
- Marco Scavazzon
- Roberto Ugolotti
- Chiara Chiarelli
- Alba Bernini
- Consultants:
- Franck Eyraud
- Pier Valerio Tognoli
- Luca Marletta
- Tomas Kliment
- Guido Notari