The inverse problem is the task of inferring model parameters from observations -- it is the key to relating data and model. This task is particularly challenging when the model is described by a partial differential equation (PDE) due to computational costs associated to simulation and the inherent ill-posedness of inversion. Solving these kinds of problems can give crucial insight into quantities of interest which are not directly observed, such as material properties, and are highly relevant to numerous application areas in science and engineering. Two key ingredients are required in the problem formulation, a regularizer encoding prior belief, and a forward model such as a numerical solver which maps model parameters to solutions of the PDE in question. In this work, we develop a methodology for inferring the prior from populational data (data originating from a collection of physical systems assumed to belong to the same population), this we call prior calibration [1]. Furthermore, we show how this task can be coupled to learning a fast machine learning based surrogate, side-stepping expensive numerical PDE simulation. Coupling these distinct problems has the key advantage of learning a surrogate model trained on the PDE parameter values which directly relate to the set of available observations. The proposed methodology is formulated as a regularized divergence [2] minimization problem and is coupled to a residual-based neural operator [3] through a bi-level optimization scheme. Furthermore, we show an equivalence to the Bayesian paradigm in the case $N=1$, i.e. where we consider data originating from a single physical system. We test the proposed scheme on 1D and 2D Darcy flows parametrized by a thresholded level-set prior and a log-normal prior. [1] Akyildiz, O. D., Girolami, M., Stuart, A. M., & Vadeboncoeur, A. (2024). Efficient Prior Calibration From Indirect Data. arXiv preprint arXiv:2405.17955. [2] Bonneel, N., Rabin, J., Peyr´e, G., & Pfister, H. (2015). Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51, 22-45. [3] Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., & Anand- kumar, A. (2020). Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895.