As Kepler slowly continued analyzing Tycho's Mars observations—now available to him in their entirety—and began the slow process of tabulating the Rudolphine Tables, Kepler also picked up the investigation of the laws of optics from his lunar essay of 1600. Both lunar and solar eclipses presented unexplained phenomena, such as unexpected shadow sizes, the red color of a total lunar eclipse, and the reportedly unusual light surrounding a total solar eclipse. Related issues of atmospheric refraction applied to all astronomical observations. Through most of 1603, Kepler paused his other work to focus on optical theory; the resulting manuscript, presented to the emperor on 1 January 1604, was published as Astronomiae Pars Optica (The Optical Part of Astronomy). In it, Kepler described the inverse-square law governing the intensity of light, reflection by flat and curved mirrors, and principles of pinhole cameras, as well as the astronomical implications of optics such as parallax and the apparent sizes of heavenly bodies. He also extended his study of optics to the human eye, and is generally considered by neuroscientists to be the first to recognize that images are projected inverted and reversed by the eye's lens onto the retina. The solution to this dilemma was not of particular importance to Kepler as he did not see it as pertaining to optics, although he did suggest that the image was later corrected "in the hollows of the brain" due to the "activity of the Soul." Today, Astronomiae Pars Optica is generally recognized as the foundation of modern optics (though the law of refraction is conspicuously absent). With respect to the beginnings of projective geometry, Kepler introduced the idea of continuous change of a mathematical entity in this work. He argued that if a focus of a conic section were allowed to move along the line joining the foci, the geometric form would morph or degenerate, one into another. In this way, an ellipse becomes a parabola when a focus moves toward infinity, and when two foci of an ellipse merge into one another, a circle is formed. As the foci of a hyperbola merge into one another, the hyperbola becomes a pair of straight lines. He also assumed that if a straight line is extended to infinity it will meet itself at a single point at infinity, thus having the properties of a large circle.

In October 1604, a bright new evening star (SN 1604) appeared, but Kepler did not believe the rumors until he saw it himself. Kepler began systematically observing the nova. Astrologically, the end of 1603 marked the beginning of a fiery trigon, the start of the about 800-year cycle of great conjunctions; astrologers associated the two previous such periods with the rise of Charlemagne (c. 800 years earlier) and the birth of Christ (c. 1600 years earlier), and thus expected events of great portent, especially regarding the emperor. It was in this context, as the imperial mathematician and astrologer to the emperor, that Kepler described the new star two years later in his De Stella Nova. In it, Kepler addressed the star's astronomical properties while taking a skeptical approach to the many astrological interpretations then circulating. He noted its fading luminosity, speculated about its origin, and used the lack of observed parallax to argue that it was in the sphere of fixed stars, further undermining the doctrine of the immutability of the heavens (the idea accepted since Aristotle that the celestial spheres were perfect and unchanging). The birth of a new star implied the variability of the heavens. In an appendix, Kepler also discussed the recent chronology work of the Polish historian Laurentius Suslyga; he calculated that, if Suslyga was correct that accepted timelines were four years behind, then the Star of Bethlehem—analogous to the present new star—would have coincided with the first great conjunction of the earlier 800-year cycle.

He then set about calculating the entire orbit of Mars, using the geometrical rate law and assuming an egg-shaped ovoid orbit. After approximately 40 failed attempts, in late 1604 he at last hit upon the idea of an ellipse, which he had previously assumed to be too simple a solution for earlier astronomers to have overlooked. Finding that an elliptical orbit fit the Mars data, Kepler immediately concluded that all planets move in ellipses, with the Sun at one focus—his first law of planetary motion. Because he employed no calculating assistants, he did not extend the mathematical analysis beyond Mars. By the end of the year, he completed the manuscript for Astronomia nova, though it would not be published until 1609 due to legal disputes over the use of Tycho's observations, the property of his heirs.