Many dynamical systems in physics and other fields possess some form of geometric structure, such as Lagrangian or Hamiltonian structure, symmetries, and conservation laws. Preserving these structures in the course of discretisation typically leads to numerical algorithms with improved stability properties and reduced errors for long-time and strongly nonlinear simulations. Most geometric integrators known and used today are based on linear approximation spaces such as splines or finite elements. When approximating differential equations with a strongly nonlinear solution manifold, these methods tend to require a large number of degrees of freedom. The reason is simple: Even if the intrinsic dimension of the nonlinear solution space is very low, an accurate approximation by a linear space requires a high dimensional approximation space. It thus seems desirable to use nonlinear approximation spaces instead, potentially with drastically reduced dimension. In this talk, we will explore the construction of variational integrators based on nonlinear representations of the solution such as neural networks. We will address some of the challenges such as initialisation and solution of the resulting discrete Euler-Lagrange equations, and show first, promising results.