Earlier detection of small invasive and non-invasive cancers such as ductal carcinoma in-situ could be achieved through systematic interrogation of the mammary duct network. Conventional ductoscopy has failed to penetrate practice because current platforms are unwieldy, inflexible, and cannot safely, swiftly and smoothly navigate down the complex tree-like structure of mammary ducts. Moreover, poor quality views with white light endoscopic visualisation within 1mm ducts hinders cancer exclusion, hence the reported low specificity. Recently, the "MAMMOBOT" has been developed, which is a flexible steerable endoscopic robotic system that can safely navigate mammary ducts. Since the MAMMOBOT requires a navigation system, this paper presents a vision-based navigation framework for robot localisation during ductoscopic examinations. A 3D model of the mammary duct tree was created by a CT scan of a custom-made phantom. Camera localisation is achieved by matching ductoscopic views with virtual views from the CT model. Our framework has been validated on phantom data and the results verify its good performance in the mammary duct tree under challenging conditions such as low video data quality and specular highlights.