Echocardiography is a potential alternative to X-ray fluoroscopy in cardiac catheterization given its richness in soft tissue information and its lack of ionizing radiation. However, its small field of view and acoustic artifacts make direct automatic segmentation of the catheters very challenging. In this study, a fast catheter segmentation framework for echocardiographic imaging guided by the segmentation of corresponding X-ray fluoroscopic imaging is proposed. The complete framework consists of: 1) catheter initialization in the first X-ray frame; 2) catheter tracking in the rest of the X-ray sequence; 3) fast registration of corresponding X-ray and ultrasound frames; and 4) catheter segmentation in ultrasound images guided by the results of both X-ray tracking and fast registration. The main contributions include: 1) a Kalman filter-based growing strategy with more clinical data evalution; 2) a SURF detector applied in a constrained search space for catheter segmentation in ultrasound images; 3) a two layer hierarchical graph model to integrate and smooth catheter fragments into a complete catheter; and 4) the integration of these components into a system for clinical applications. This framework is evaluated on five sequences of porcine data and four sequences of patient data comprising more than 3000 X-ray frames and more than 1000 ultrasound frames. The results show that our algorithm is able to track the catheter in ultrasound images at 1.3 s per frame, with an error of less than 2 mm. However, although this may satisfy the accuracy for visualization purposes and is also fast, the algorithm still needs to be further accelerated for real-time clinical applications.