Image super-resolution techniques exploiting the stochastic fluctuations of image intensities have become a powerful tool in fluorescence microscopy. Compared to other approaches, these techniques can be applied under standard acquisition settings and do not require special microscopes nor fluorophores. Most of these approaches can be mathematically modeled making use of second-order statistics possibly combined with a priori regularization on the desired solution. In this work, we consider a different paradigm and formulate a physical-inspired data-driven approach based on generative learning. By simulating fluorescence and noise fluctuations by means of a suitable double Poisson-type process, the unknown distribution of the fluctuating sequence of low- resolution and noisy images is approximated via a GAN-type approach (Generative Adversial Network)where both physical and network parameters are optimized, see 2. Compared to this previous work, we use a double Poisson process to simulate fluctuations of fluorophores on one side and noise due to ambiant fluorophores on the other side. We also provide theoretical insights on the choice of the corresponding cost functionals and gradient computations, and assess practical performance on simulated Argolight (see figure 1).

We studied stochastic gradient descent (SGD) algorithm for solving linear inverse problems (e.g., CT image reconstruction) in the Banach space framework of variable exponent Lebesgue spaces ${\ell }^{p_n}\left(ℝ\right)$. These spaces are defined in terms of a point-wise defined variable exponent function and under mild assumptions that the exponent map are Banach spaces, endowed with the so-called Luxembourg norm. Such norm has no closed form expression, it is not additively separable and it is shift variant. Such non-standard spaces have been recently proved to be the appropriate functional framework to enforce pixel-adaptive regularization in signal and image processing applications. Compared to its use in Hilbert settings, however, the application of SGD in the Banach setting requires the use of duality maps and in ${\ell }^{p_n}\left(ℝ\right)$ spaces that is not straightforward, due, in particular to the lack of a closed-form expression and the non-separability property of the underlying norm, which appears in the definition of duality maps. We showed that SGD iterations can effectively be performed using the associated modular function. We compared results obtained by standard approaches and on constant/variable Lebesgue spaces on both simulated and real computed tomography (CT) data (cf Figure 2). Numerical validation show significant improvements in comparison to SGD solutions both in Hilbert and other Banach settings, in particular when non-Gaussian or mixed noise is observed in the data.

The work is published in 22.

Within the framework of inverse problems, modern approaches in fluorescence microscopy reconstruct a super-resolved image from a temporal stack of frames by carefully designing suitable hand-crafted sparsity-promoting regularizers. Numerically, such approaches are solved by proximal gradient-based iterative schemes. Aiming at obtaining a reconstruction more adapted to sample geometries (e.g. thin filaments), we adopt a plug-and-play denoising approach with convergence guarantees and replace the proximity operator associated with the explicit image regularizer with an image denoiser (i.e. a pre-trained denoising network) which, upon appropriate training, mimics the action of an implicit prior. To account for the independence of the fluctuations between molecules, the model relies on second-order statistics. The denoiser is then trained on covariance images coming from data representing sequences of fluctuating fluorescent molecules with filament structure. The method is evaluated on both simulated and real fluorescence microscopy images, showing its ability to correctly reconstruct filament structures with high values of peak signal-to-noise ratio (see Figure 3).

The work is published in 23.

Recent years have seen the development of super-resolution variational optimization in measurement spaces. These so-called "gridless" approaches offer both theoretical (uniqueness, reconstruction guarantees) and numerical results, with very convincing results in biomedical imaging. However, gridless variational optimization is formulated for the reconstruction of point sources, which is not always suitable for biomedical imaging applications: more realistic biological structures such as curves, to represent blood vessels or filaments, would also need to be reconstructed.

In this work, we developed a new approach for gridless reconstruction of curves, understood as the reconstruction of a vector measure $m$ carried by curves from a degraded $y$ acquisition. By introducing a new space of $𝒱$ vector measures with finite divergence, as well as a new CROC functional, we give a result on the extremal points of the unit ball of the $𝒱$ norm; this allows us to reconstruct curves with a gridless variational approach, in an inverse problem framework.

First numerical implementation is illustrated in Figure 4. A new data term for CROC has been proposed linking scalar observed data $y$ with the vector measure $m$.

This work is published in 20, 21.

A code is available at 6.1.5

Growing evidence indicates that only a sparse subset from a pool of sensory neurons is active for the encoding of visual stimuli at any instant in time. Traditionally, to replicate such biological sparsity, generative models have been using the ${\ell }_1$ norm as a penalty due to its convexity, which makes it amenable to fast and simple algorithmic solvers. In this work, we use biological vision as a test-bed and show that the soft thresholding operation associated to the use of the ${\ell }_1$ norm is highly suboptimal compared to other functions suited to approximating ${\ell }_p$ with $0\le p<1$ (including recently proposed continuous exact relaxations), in terms of performance. We show that ${\ell }_1$ sparsity employs a pool with more neurons, i.e. has a higher degree of overcompleteness, in order to maintain the same reconstruction error as the other methods considered. More specifically, at the same sparsity level, the thresholding algorithm using the ${\ell }_1$ norm as a penalty requires a dictionary of ten times more units compared to the proposed approach, where a non-convex continuous relaxation of the ${\ell }_0$ pseudo-norm is used, to reconstruct the external stimulus equally well. At a fixed sparsity level, both ${\ell }_0$- and ${\ell }_1$-based regularization develop units with receptive field (RF) shapes similar to biological neurons in V1 (and a subset of neurons in V2), but ${\ell }_0$-based regularization shows approximately five times better reconstruction of the stimulus. Our results in conjunction with recent metabolic findings indicate that for V1 to operate efficiently it should follow a coding regime which uses a regularization that is closer to the ${\ell }_0$ pseudo-norm rather than the ${\ell }_1$ one, and suggests a similar mode of operation for the sensory cortex in general (see Figure 5).

The results are published in 19.

The code is available at github.com/rentzi.

Plankton organisms are a key component of the Earth bio-sphere. The phytoplankton captures the CO2 of the atmosphere, and the zooplankton, as a phytoplankton predator, aggregates it and stores it on the ocean floor when it dies. This so-called carbon pump is studied by ecologists in order to predict its future efficiency in a climate change era. A modern approach consists in studying the relation between the living environment of the organisms and their functional traits, among which the size stands out. Indeed, a high correlation has been observed between zooplankton size and carbon sequestration. Scientists have developed in situ imaging instruments and built large databases. Taxonomic identification from images is required to estimate the individual volumes of the organisms based on their morphology. The development of automated classification methods has been essential to aid operators in their identification task. The breakthrough of Artificial Neural Network (ANN) made it possible to come up with efficient classifiers. However, the decisions of such classifiers are hard to interpret or explain. We put forward the idea that following the transform-then-classify approach of ANNs using a simple, explicit transform can result in a classifier whose predictions are both interpretable (thus, trustable) and accurate. The proposed transform is defined as a linear combination of per-class target vectors, and the classification is performed, like with ANNs, by a nearest-target decision (see Figure 6). Furthermore, as a main theoretical result, we establish that the proposed transform defines a kernel associated with the Weigthed-Nearest-Neighbors (wNN) classifier. Incidentally, this allows to interpret the wNN classifier as a member of the transform-then-classify family of classifiers.

The detection of cell division and cell death events in live-cell assays has the potential to produce robust metrics of drug pharmacodynamics and return a more comprehensive understanding of tumor cells responses to cancer therapeutic combinations. As cancer drugs may have complex and mixed effects on the biology of the cell, knowing precisely when cellular events occur in a live-cell experiment allows to study the relative contribution of different drug effects (cytotoxic/cell death or cytostatic/cell division) on a cell population. Yet, classical methods require dyes (fluorescent molecules) to measure cell viability as an end-point assay, where the proliferation rates can only be estimated when both viable and dead cells are labeled simultaneously, not to mention that the cell division events are often discarded due to analytical limitations. Live-cell imaging is a promising cell-based assay to determine drug efficacies. However, its main limitation remains the accuracy and depth of the analyses, to acquire automatic measures of the cellular response phenotype, making the understanding of drug action on cell populations difficult. We propose a new algorithmic architecture integrating machine learning, image and signal processing methods to perform dynamic image analyses of single cell events in time-lapse microscopy experiments of drug pharmacological profiling. Our event detection method is based on a pattern detection approach on the local entropy in polarized light frames, making it free of any labeling step and exhibiting two distinct patterns for cell division and death events. Our analysis framework is an open source and adaptable workflow that automatically predicts cellular events (and their times) from each single cell trajectory, along with other classic cellular features of cell image analyses, as a promising solution in pharmacodynamics (see Figure 7). This project is currently supported by two actions studying the possibility of launching a startup company (Young Entrepreneur Program, Labex SignaLife and Emergence et Accompagnement, Canceropôle PACA).

This work was made in collaboration with Fabienne de Graeve, iBV, Nice, France.

Recent technological advances in bioimaging have enabled biologists to investigate dynamic processes in living cells at high spatial and temporal resolutions. Quantitative analysis of these dynamic processes relies on the development of appropriate tracking methods associated with a spatio-temporal analysis. We developed particle tracking and event detection methods to analyze the movements and interactions of Imp-RNP (Ribonucleoprotein) granules in the cell bodies of mushroom neurons in Drosophila brain explants. The individual granule trajectories are considered globally as a graph embedded in a space-time domain. The base granule fusion and splitting events can occur in specific sequences which build particular graph patterns. The proposed analysis consists in defining the relevant patterns, and then searching them in actual graphs (see Figure 8).

This study aimed to identify biomarkers of relapse following surgical intervention by analyzing 17 primary squamous cell carcinoma tumors (9 relapsing, 8 non-relapsing) using multiplex (11 markers) mass spectrometry imaging to investigate the interactions between different cell types and structures. We first have detected cells in the different channels using local thresholding. A k-means clustering has then exihibited different populations and in particular several macrophage subpopulations. An analysis of the cells distances with respect to the tumor and of their density in specific structures (vessels, nerves) has shown some specific localization of these populations (see Figure 9).

These findings have revealed insights into the complex links between immune cells, tumor structures and relapse. The identified B cell influx, as well as the differential behavior of macrophage subpopulations, may have implications for understanding tumor progression and designing targeted therapies to prevent relapse following surgery.

We have developed a features-based supervised algorithm for nuclei classification in histopathological images. Features are composed of color and shape properties of detected nuclei as well as the color inside a crown surrounding the nuclei. We have considered five classes of cells : sane, cancerous, fibroblast, epithelial, lymphocyte. Usign a SVM classifier we have obtained a weighted F1 score of 0.89. We then consider a labelled graph where the node are defined by the nuclei and the edges by adjacent cells in the associated Voronoi tesselation. A regularization step is applied using a local majority vote. However lymphocyte and epithelial cells are excluded from this regularization step due their low prevalence. After regularization, the Voronoi tesselation provides a segmentation of the tissue in seral regions composed of tumour, sane areas and fibrous tissues (see Figure 10). These first results will be improved by considering a regularization at a coarser scale, that is considering the different obtained regions.

This work was carried out in collaboration with Francesco Ponzio from Politechnico di Torino (Italy), Damien Ambrosetti and Giorgio Toni from Nice CHU.

The Banff classification is used by pathologists for the evaluation and monitoring of kidney implants. This classification, in association with other biological and clinical elements, makes it possible to diagnose rejection and estimate the prognosis, through the analysis of different lesions in the 4 compartments (glomeruli, tubules, interstitium and vessels). However, the numerous classification criteria and their quantification, in addition to the very time-consuming nature of their evaluation, can present problems of inter- and intra-individual reproducibility, hence the relevance of automating this evaluation. We propose an analysis methodology for this automatic evaluation. The first step is to identify and separate the different compartments. Secondly, each compartment will be analyzed individually. In this first work, the lumen areas are identified by thresholding followed by mathematical morphology operators. These areas correspond either to tubules or to vessels. To classify them we use nuclear characteristics in the neighborhood of each lumen. We therefore detect the nuclei using a shape dictionary. Each nucleus is associated with a lumen by minimizing the distance. The lumens are then classified as tubule or vessel by an SVM algorithm based on geometric parameters of area, number and shape of nuclei (see Figure 11).

This work was carried out in collaboration with Francesco Ponzio from Politechnico di Torino (Italy), Damien Ambrosetti and Giorgio Toni from Nice CHU.

Type I diabet may lead to kydney pathology consisting of a degradation of glomeruli. In this project we applied two CNN models to detect glomeruli from histopathological images (Faster R-CNN) and grade them into sane, mild, severe or necrosis stages (see Figure 12). We tested the model on two separate datasets: one from Paris and another assembled using data from Nice. The model performed well, with the majority of errors falling within adjacent classes. Although it exhibited comparatively less proficiency on the samples from Paris (as it had not encountered any data from this center during training), it still demonstrated reasonable performance. Notably, most errors were confined to adjacent classes, typically associated with ambiguous samples. We also performed an extensive hyperparameter optimization. This involved experimenting with various architectures, adjusting parameters, and implementing advanced augmentations inspired by the circular shape of glomeruli. These attempts were undertaken in an effort to balance our training dataset and mitigate overfitting.

This work was carried out in collaboration with Francesco Ponzio from Politechnico di Torino (Italy), Damien Ambrosetti from Nice CHU and Nicolas Pote from Bichat hospital.

Reinforcement learning (RL) is one of the machine learning paradigms, alongside supervised and unsupervised learning, where an agent learns from its direct interaction and manipulation of its environment through a set of actions. The agent acts based on observations known as states, and it learns using a reward signal received from the environment. We aimed to derive a RL framework to address histopathological image analysis tasks. This will be exemplified on kidney and lung cancer datasets.

The main challenges present in RL, unlike other paradigms, include the concept of delayed rewards and the necessity for exploration (where the data observed depends on the agent's trajectory) ...Our objective is to train an agent to perform diagnosis on whole slide images, similar to how pathologists do it: by zooming in and out, moving around and select patches/patterns that characterize specific tumor. Throughout the first three months of this project, we conducted an in-depth literature review in the following categories:

The study of Reflectance Confocal Microscopy (RCM) images provides information on epidermal structure, a key to skin health and barrier which changes in each epidermal layer, with age and with certain skin conditions. RCM can also reveal dynamic changes to the epidermis in response to stimuli, as it enables repetitive sampling of the skin without damaging the tissue. Studying RCM images requires manual identification of each cell to derive its geometrical and topological characteristics, which is time-consuming and subject to expert interpretation. More insights could be derived from these data if not for the tediousness of their manual segmentation, highlighting the need for an automated cell identification method. We have previously developed a new unsupervised approach based on multi-task learning to identify cells on RCM images. This approach consists of a dual-task Cycle Generative Adversarial Networks model, where the first task learns RCM images noise and/or texture model from the initial image, while the second task learns the epidermis structure from a Gabor filtering, thus allowing us to denoise RCM images while keeping the position and integrity of the membranes. To our knowledge, this is the first time a Cycle GAN has been embedded in a multi-task learning model. We have started exploring the use of this unsupervised method on images acquired using different imaging modalities and representing different tissues without retraining (histology images, cell culture images, mass spectroscopy images, and fluorescence microscopy images). Indeed, multi-task approaches tend to be less data dependent, and therefore more easily transposable to other types of data not included in the training of the approach. We developed 3 iterations of DermoGAN depending on the type of cell organization (confluent cells, non-confluent cells, and histology images), considering alternatives to the Gabor filter and to the simulated expected results, for example using Canny filter and a Marked Point Process (see Figure 14). We theorize that the performance of DermoGAN is dependent on the cell organization visible in the images used in the training, not on the imaging modality or tissue.

An international patent has been submitted for this work.

The aim of this work is to built a phenotypic map of organoids. We model an organoid as a 3D shape containing several layers of cells composed of 2D point patterns on the sphere. Our goal is to classify organoids with respect to these three components.

We propose an architecture for classifying organoids (clusters of stem cells, cf Figure 15) developed in the presence of endocrine disruptors. First, we proposed a simplified organoid model, from which we generated synthetic datasets composed of point patterns on the sphere defining a graph using the Voronoi representation. We carried out a comparative study between a classification method based on Deep Learning for graphs, and a method based on spatial statistics. We thus showed that the Deep Learning approach was at least as effective and more robust to noise than the other. We then developed an algorithm allowing the nuclei detection of organoids that have been cultured in the laboratory. In order to enable the use of Deep Learning architectures from a limited number of organoids, we used Data Augmentation techniques. We are currently developing two types of architecture (one based on convolution networks, the other on layers dedicated to graphs), to obtain a baseline. With the aim of ultimately proposing a robust and interpretable model, we seek to enrich it with a part based on the 3D analysis of the shape of the organoid, in particular via the decomposition of these shapes into spherical harmonics.

This work is on the detection and characterization of Fibronectin (FN) structures found in the tumour extracellular matrix (ECM). Three structures were identified in our multiplex fluorescent images: aggregates, representing accumulations of insoluble, non-fibrillar FN; FN fibres; and tumour cells expressing cytokeratin (Figure 16).

A greedy algorithm, leveraging the geometric assumptions we have about the aggregates — specifically, that they exhibit an elliptical resemblance — has been developed and tested. This algorithm is grounded in a marked point process model proposed by X. Descombes 4. Additionally, a modified Gabor kernel 31 has been introduced to enhance fibre detection, minimizing the blurring effect. Classical image processing techniques were applied to achieve an accurate segmentation of tumour cells (Figure 17). Ongoing efforts are directed towards characterizing the diverse components of the ECM using this segmentation.

The research activities of this project focus on the analysis of microscopy images of the extracellular matrix (ECM) in the microenvironment of head and neck cancer tumors. This project builds on previous works carried out within the team, through collaborations with clinical partners (CAL, Unicancer, Head and Neck Group) and with Fabienne Anjuère, immunologist at IPMC. This research, carried out as part of the TOPNIVO and CHECK-UP clinical trials, has led to the development of a pipeline for acquiring multiplexed immunofluorescence and HES-stained histological images. In this context, some 6,000 multispectral images and 350 HES (Whole slide images) have been acquired.

The current work focuses on analyzing the topological and geometrical changes in fiber networks of the tumor ECM (Extra-Cellular Matrix) using the developed image analysis pipelines and algorithms. We are also working on characterizing the organization of ECM proteins and finding structural biomarkers to develop predictive models for patient response to immunotherapy, This can be summarized in two main parts (see figure 18):

1- The development of deep learning algorithms for image processing (signal extraction, segmentation, and classification of biological structures of interest), which requires both expert annotation and software development so that these images, which can weigh several gigabytes, can be processed by Deep Learning.

2 - The implementation of quantitative statistics to search for spatial signatures or biomarkers in the images, and more specifically in the extracellular matrix, in order to predict patient response to immunotherapy, as well as to provide insights on the functioning of immunosuppressive functions.

It is admitted that ascidian exhibited a stereotyped development in the first stages of their embryogenesis 30. Moreover, we demonstrated that the developmental speed (characterized as the derivative of the count of cells) is identical from one embryo to the other (up to a scaling factor, the normal developmental speed being linearly correlated to the medium temperature), suggesting that the cell division rate is identical in a population of normally developing embryos (see Figure 19 left and middle).

Thanks to this consistent cell division rate amongst embryos, we can study whether the order of cell divisions is identical across embryos. For this purpose, we ordered the cells with respect to their average division time and estimate the probability that a cell with a late average division divides before a cell with an earlier division (Figure 19 right). This emphasizes the different rounds of divisions that succeed during the embryo development, with still phases at the stages of 64, 76, 112, 184 and 218 cells. It also reveals that the closer the average division times are, the higher the probability that the late one occurs before the early one.

Ascidian exhibited a stereotyped development in the first stages of their embryogenesis 30. Unpublished results demonstrate a strong reproducibility of cell naming with respect to geometrical (cell position), and topological (cell neighborhood) considerations, but have also revealed that some cells exhibit different division orientation patterns.

In this work, we aim at studying the variability of cell division, namely whether a cell division has different orientation patterns across embryos.

We study each cell division individually. First, all embryos are registered against one common reference (one of the embryo at hand), so they share a common geometric referential. Having defined the division orientation as the direction between the centers of mass of the two daughter cells, we collect the direction and compute a kernel-based direction density function (on the 3D sphere), see Figure 20. Extracting the number of modes of this density function together with the number of samples participating to each mode is a powerful means to point out cells whose division is more likely to exhibit different orientation patterns, see Figure 21.