LabMD's lawyers argue Congress gave that oversight to the U.S. Department of Health and Human Services, and that the FTC is using the case to expand its data-security powers "at LabMD's expense."

Related: Navigating the cyber-insurance maze

"In this federal agency enforcement action, the FTC overstepped its authority and, in the process, destroyed a small medical testing company," LabMD's lawyers, including Douglas Meal, wrote in Tuesday's court papers.

At the heart of the caseis a July ruling from the FTC that said LabMD failed to adequately protect patients' personal information after a 1,700-page file was exposed on a peer-to-peer file-sharing network.

The 3-0 decision reversed a ruling by the FTC's chief in-house judge, D. Michael Chappell, who earlier said the agency failed to show that LabMD harmed any patients by mistakenly exposing the file.

FTC Chairwoman Edith Ramirez, writing for the commission, said Chappell applied the wrong legal standard in determining the mere exposure of sensitive personal information fell short of causing a substantial injury. Ramirez said lapses in data security could be deemed "unfair" under the Federal Trade Commission Act if the magnitude of the potential harm is high, "even if the likelihood of the injury occurring is low."

The FTC's case against LabMD gained a larger profile as the company's chief executive, Michael Daugherty, railed against the agency's handling of the enforcement action and published a book—"The Devil Inside the Beltway"—that chronicled the investigation.

In the Eleventh Circuit papers, LabMD's defense team said there was "substantial reason to believe" the FTC not only brought the case in retaliation for Daugherty's book but also that the agency "itself had a hand in the very data theft the commission used to justify its action against LabMD."

LabMD has long accused the FTC of having an inappropriate relationship with the data security firm Tiversa, which first discovered the LabMD patient file on the peer-to-peer network LimeWire.

LabMD alleges Tiversa tipped off the FTC to the file's exposure and manufactured evidence that the file was spreading online in retaliation for LabMD refusing to purchase the firm's security remediation services. The FTC and Tiversa have denied any malfeasance.

LabMD's lawyers said in their brief that the company "employed a comprehensive security program that included a compliance program, training, firewalls, network monitoring, password controls, access controls, antivirus, and security-related inspections."