Taku Yamasaki reportedly made the remarks at a dinner, where he briefed the prime minister on unauthorised meetings he held with Pyongyang officials in China last week, the Kyodo news agency reported.
"I was convinced that North Korea hopes to normalize diplomatic ties by realizing the Pyongyang Declaration on the nuclear, missile, abduction and other issues during the prime minister's tenure," Yamasaki was quoted as saying by guests at the dinner.
Yamasaki, former secretary general of Koizumi's own ruling Liberal Democratic Party (LDP), and another party member were strongly criticized by the prime minister on Friday for making the visits.
Yamasaki and lawmaker Katsuei Hirasawa, also of the LDP, went to meet the North Koreans on Thursday and Friday in Dailian to seek the resumption of stalled talks on Pyongyang's abduction of Japanese nationals.
Hirasawa, a junior minister at the Ministry of Public Management, Home Affairs, Posts and Telecommunications, submitted his resignation from the post.
Koizumi told reporters on Friday that he only learned of the visit from newspapers.
Talks on normalizing ties have been stalled due to public anger in Japan over North Korea's admission that it had kidnapped Japanese nationals to help train spies.
Japan-North Korea bilateral talks "must be conducted by the two governments. If (Hirasawa and Yamasaki) give the wrong message, it would only complicate our relationship," Chief Cabinet Secretary Yasuo Fukuda told reporters last week.