In March 1869 President Ulysses S. Grant offered Stewart the position of Secretary of the Treasury (after Joseph Seligman had declined it), but he was not confirmed by the United States Senate. One source reports that a major impediment to Stewart's appointment was a provision in the act of September 2, 1789, which established the Treasury Department, prohibiting a merchant or importer in active business from heading the Department. Grant requested the two houses of Congress to override the provision, but upon the objection of Charles Sumner, the request was not considered in the Senate. Another source attributes Stewart's rejection to his close association with Judge Henry Hilton, his wife's cousin's husband and a member of the corrupt Tweed Ring.
In 1869 and 1870 A. T. Stewart built the first of the grand Fifth Avenue palaces, on the northwest corner of 34th Street, across from the doyenne of New York society, Caroline Schermerhorn Astor. His architect, as for the store, was John Kellum. When all of Fifth Avenue was of brownstone rowhouses, Stewart's fireproof structure in French Second Empire style was faced with marble.