We present a method of resolving the geometric structure in unresolved CCD images of the two-component stellar objects with relative separations below the full width at half maximum (FWHM). The practical applicability of this method has been demonstrated on example tests of the newly discovered binary, DENIS-P J1756296-451822, with a relative separation of about 0.15″ (or 0.25 × FWHM). For this purpose, we used unresolved binary images obtained with the VLT/FORS2 camera, which provided precise astrometric positions of the system photocenter. Using the same images, we applied a new, tested method capable of resolving the geometry of the binary by taking into account the difference of the image shape of the binary and of single stars, adopting an effective elliptical point spread function (PSF). In this way, we derived independent additional information on the geometry of the binary system, which allowed us to estimate the mass ratio and improved the overall orbit fit. Also, we used a single series of adaptive optics observations with NACO. Combining these data, we derived a relative flux of the secondary in the I band of 0.66 ± 0.04 and a mass ratio of q ≃ 0.886 ± 0.049, along with dynamical masses of M1 = 63.9−2.1+2.5MJup for the primary and M2 = 56.6−1.9+2.7MJup for the secondary. We note that these values are below the substellar limit. Using theoretical cooling curves for brown dwarfs, we were able to estimate the age of this binary system at between 200 and 350 Myr.