Plans for a retirement complex have been dropped amid concerns paths to and from the site would be unsafe and too steep for older residents.