We introduce SYNFERENCE, a new flexible Python framework for galaxy spectral energy distribution (SED) fitting using simulation-based inference (SBI). SYNFERENCE leverages the SYNTHESIZER package for flexible forward-modelling of galaxy SEDs and integrates the LTU-ILI package to ensure best practices in model training and validation. In this work we demonstrate SYNFERENCE by training a neural posterior estimator on $10^6$ simulated galaxies, based on a flexible eight-parameter physical model, to infer galaxy properties from 14-band Hubble Space Telescope and James Webb Space Telescope (JWST) photometry. We validate this model, demonstrating excellent parameter recovery (e.g. R$^2\gt $0.99 for M$_\star$, R$^2\simeq 0.88$ for age) and accurate posterior calibration against nested sampling results. We apply our trained model to 3088 spectroscopically confirmed galaxies in the JWST Advanced Deep Extragalactic Survey GOODS-South field. The amortized inference is exceptionally fast, having nearly fixed cost per posterior evaluation and processing the entire sample in $\sim$3 min on a single CPU (18 galaxies per CPU per second), a $\sim 1700\times$ speed-up over traditional nested sampling or Markov Chain Monte Carlo techniques. We demonstrate SYNFERENCE's ability to simultaneously infer photometric redshifts and physical parameters, and highlight its utility for rapid Bayesian model comparison by demonstrating systematic stellar mass differences between two commonly used stellar population synthesis models. SYNFERENCE is a powerful scalable tool poised to maximize the scientific return of next-generation galaxy surveys.