In order to deal with the problems of noise pollution, designing systems and structures that can have targeted control over the propagation of sound waves is of great importance. One of the most common and widely used methods in controlling noise pollution is the use of sandwich panels as acoustic insulation. The distinctive feature of this type of panels is the high variability of their core, so that by changing the material and arrangement of the core, the acoustic and mechanical properties of the panel change. In recent years, nesting structures have been used as the core of sandwich panels. Nesting cores are a good option for designing sound insulation due to their light weight and high stiffness. The various features and characteristics of the nest structure, including cell geometry, the type of structure used, and the type of arrangement and orientation of the cells next to each other, indicate that an optimal structure should be designed to achieve better acoustic behavior. In this research, genetic algorithm has been used to improve the acoustic performance of sandwich panel structure with honeycomb core. This optimization is done for two common types of nesting cell orientations, taking into account that the panel is under in-plane loading. Optimization has been performed on the geometric variables of the nesting cell for a number of materials with different physical and mechanical properties, and the aim was to reduce the sound transmission through this panel in the low frequency range.