AbstractA pipe-in-pipe (PIP) system is an important engineering structure in the petroleum industry. Under the excitation of ocean fluids, the PIP system produces vortex-induced vibration (VIV), which can affect the oil recovery efficiency of large offshore drilling platforms and cause various engineering accidents. In this work, a fluid–structure coupled vibration model of the PIP system excited by a vortex-induced force is established based on an Euler–Bernoulli double-beam model and a wake oscillator model. The harmonic balance method is used to solve the coupled vibration model. Influences of the connection layer stiffness, tension, mass ratio, phase angle, and frequency ratio on displacements of the PIP system are studied. It is shown from numerical results that displacements of inner and outer pipes increase with the slenderness ratio. Displacements of inner and outer pipes decrease with the connection layer stiffness, but the frequency ratio and traveling wave velocity increase with it. In addition, the frequency ratio of the PIP system exhibits a multivalued characteristic, which is the characteristic of a nonlinear system. A small stiffness of the connection layer can induce unstable vibrations. Displacements of inner and outer pipes decrease with the tension, but the frequency ratio increases with it. The collision time of the inner and outer pipes gradually increases with the tension. The pipe displacements decrease with the mass ratio.