This paper introduces voxelized atomic structure (VASt) potentials as a machine learning (ML) framework for developing interatomic potentials. The VASt framework utilizes a voxelized representation of the atomic structure directly as the input to a convolutional neural network (CNN). This allows for high-fidelity representations of highly complex and diverse spatial arrangements of the atomic environments of interest. The CNN implicitly establishes the low-dimensional features needed to correlate each atomic neighborhood to its net atomic force. The selection of the salient features of the atomic structure (i.e., feature engineering) in the VASt framework is implicit, comprehensive, automated, scalable, and highly efficient. The calibrated convolutional layers learn the complex spatial relationships and multibody interactions that govern the physics of atomic systems with remarkable fidelity. We show that VASt potentials predict highly accurate forces on two phases of silicon carbide and the thermal conductivity of silicon over a range of isotropic strain.