"If people do take action - take the law into their own hands - the rules that apply are sometimes ones that you can't as a government change. There will be consequences and in this case it is very unlikely that Australian law will apply," Roxon told Australian public broadcaster the ABC.
"We are doing all we can diplomatically to ensure that these three Australian men can be released quickly and properly and to make sure that they are being held in conditions that are appropriate."
The West Australian men - Geoff Tuxworth, 47, Simon Peterffy, 44, and Glen Pendlebury, 27 - from the Forest Rescue environmental group boarded the Shonan Maru II in waters off the state's south early Sunday morning.
The daring mission was aimed at forcing the vessel to stop tailing the Sea Shepherd Conservation Society's anti-whaling flagship, the Steve Irwin.
The men climbed past razorwire and spikes to board the ship and deliver a message: "Return us to shore in Australia and then remove yourself from our waters."
But the Shonan Maru No.2 yesterday afternoon was instead persisting with its pursuit of the Steve Irwin, with the Australians still on board.
The men could be taken to Japan to face criminal charges over their actions.
Glenn Inwood, spokesman for the Japanese whaling program at the Institute of Cetacean Research, told The Australian that the incident occurred about 25 miles (40km) off the West Australian coast, outside Australian territorial waters but inside Australia's 200-mile (320km) economic zone.
"They are unhurt and they are being well looked after," Inwood said of the three men.
He told ABC radio, "Not only are they facing [a long time on board the vessel], but they risk being taken to Japan to be tried for trespassing, or for other charges."
Ms Roxon said discussions with the Japanese government were continuing.
"We have had a number of interactions with the Japanese government, particularly through our embassy in Tokyo," she told ABC radio.
She said it was difficult because the incident didn't happen in Australia's territorial waters.
"Because it was only in our exclusive economic zone, that doesn't give us automatic rights to assert Australian law.
"In fact, the clearest advice that we have is that Japanese law would be likely to apply."
"It is likely that these three Australians may be taken back to Japan," she said.
But Sea Shepherd's leader, Paul Watson, said the three men had boarded the Japanese vessel within Australia's 24-mile contiguous zone, in which Australian laws sometimes apply.
"It disturbs me that the Attorney-General says it was outside Australian zone," Watson said. "Both vessels have GPS and I expect that she will be very embarrassed when the truth comes out."
"I think [Roxon] is a coward, it is absolutely incredible she's not doing more to help these Australian citizens."
The Steve Irwin's first mate, Sid Chakravardy, told The (Sydney) Daily Telegraph that the trio had a VHF radio when they boarded the whaling vessel.
"I believe that must have been taken away from them," Chakravardy said. "We've had no contact with them since they boarded the vessel."
Donald Rothwell, an international law expert from Australian National University, said the three activists could face a wide range of charges - and may have even broken Australian law.
Rothwell said unauthorized boarding of a Japanese vessel was an act of trespass "wherever that act may have taken place at sea."
"If the Japanese authorities decide to investigate and bring charges on these grounds, the Australians will find themselves being sent to Japan to face court," he said. "Second, Australian law may have been broken by the unauthorized boarding."
Opposition Leader Tony Abbott urged the government to send a customs vessel to "keep the peace" in the Southern Ocean.
The Federal government must also do everything it can to help three activists being held aboard the Japanese whaling security vessel Shonan Maru No 2, Mr Abbott said.
"The Australian government helps Australians in trouble, including Australians that have got themselves into trouble,'' the Opposition Leader said.
"I think it's just the standard duty of government to do what it can for Australians in trouble overseas and obviously these guys fall into that category.''
Asked if an Australian customs vessel might be sent to the Southern Ocean, she replied: "We are considering all options. Our top priority is to ensure these three Australian men are being treated properly and to arrange for their return to Australia as quickly as possible."
Steps to prevent whaling will continue, Ms Roxon said.
"We don't support the action being taken by Japanese whalers. That's why we have consistently taken steps to prevent whaling, including taking action in the International Court of Justice."
Sea Shepherd head captain Paul Watson urged the government to take swift action to prevent that happening.
"I think the Australian government would be very embarrassed if an armed Japanese vessel can just pick up Australian citizens in Australia and then take them away to Japan," Captain Watson said from the Steve Irwin.
"I'd like the Australian Government to do it's job and uphold international law," Mr Watson told Channel 9.
"They have done nothing.
"The Australian Government should do everything in it's power to get them off that ship."
"I think it's their obligation to protect the interests of their own citizens.
"Japanese vessels have no right to take prisoners in Australian waters."
Opposition environment spokesman Greg Hunt said the government had ignored repeated warnings this sort of incident might happen.
"They should immediately dispatch a Customs vessel," Mr Hunt said.
"They should also seek the immediate transfer of the prisoners from the Japanese ship to Australian authorities."
The Australian Greens called on Prime Minister Julia Gillard to personally intervene by contacting her Japanese counterpart.
"The protesters have acted as a direct result of our own government's failure to take the necessary steps to end illegal whaling in the Southern Ocean," Greens Senator Rachel Siewert said.
She said it would be "unacceptable" for the ship to return to the Southern Ocean with the protesters still on board.