Federal Agency Protection of Privacy Act of 2004

Federal Agency Protection of Privacy Act of 2004 - Requires Federal agencies: (1) when publishing a general notice of proposed rulemaking for any proposed rule or a notice of proposed rulemaking for an interpretative rule involving the internal revenue laws, and such rulemaking pertains to the collection, maintenance, use, or disclosure of personally identifiable information from ten or more individuals, other than agencies, instrumentalities, or Federal employees, to prepare an initial assessment that describes the rule's impact on individual privacy; and (2) when promulgating the final rule, to prepare a final privacy impact assessment that includes a summary of any significant issues raised by and changes made pursuant to public comments on the initial assessment. Instructs agencies to make copies of final assessments publicly available and to publish such assessments or summaries thereof in the Federal Register.

Allows the head of an agency to waive or delay the completion of some or all of: (1) these requirements to the same extent as the agency head may waive or delay the completion of requirements for regulatory flexibility analyses; and (2) certain of the preceding requirements under this Act for national security reasons, or to protect from disclosure classified information, confidential commercial information, or information the disclosures of which may adversely affect a law enforcement effort.

Requires the head of an agency promulgating a rule that may have a significant privacy impact to assure that individuals have been given an opportunity to participate in the rulemaking.

Requires each agency to: (1) carry out a periodic review of promulgated rules that have a signifcant privacy impact to determine whether each such rule can be amended or rescinded in a manner that minimizes such impact while remaining in accordance with applicable statutes; (2) carry out such review in accordance with a plan that provides for the review of each rule every ten years; and (3) annually publish a list of the rules to be reviewed. Requires the list to: (1) include a brief description of each rule and the need for and legal basis of such rule; and (2) invite public comment upon the determination concerning the rule.

Sets forth provisions governing judicial review of agency compliance with this Act.

Requires congressional review of agencies' actions that are relevant to this Act.