The three attackers were suspected to be Islamic State militants, the security source, who requested not to be named, told Reuters.
The assault also reportedly involved a vehicle bomb detonated outside the building, which was engulfed in smoke and flames in photos taken after the attack. Two managed to get inside and blow themselves up.
At least three members of staff were killed and 21 others were wounded, health ministry spokesman Wedad Abo Al Niran said.
Heavy smoke rose from the building which was surrounded by security forces as wounded people were rushed to hospital.
The assault was carried out by several "terrorist attackers", according to the Libyan unity government's official TV channel, which cited foreign and interior ministry sources.
Foreign Minister Tahar Siala said one of the dead was senior diplomat Ibrahim al-Shaibi who headed a department in his ministry.
Trump insists on building border wall amid government shutdown
Add Donald Trump as an interest to stay up to date on the latest Donald Trump news, video, and analysis from ABC News. Trump spoke to reporters who traveled with him to Iraq. "I can't tell you when the government's going to be open".
Justice Ruth Bader Ginsburg , politics, donald trump, notorious rbg
On Friday, Ginsburg's trainer Bryant Johnson told PEOPLE that she is eager to hit the gym after cancer surgery. Ginsburg, 85, is still at a New York City hospital and no date has been set for her return to Washington .
Trump seeks high ground over Mattis as Syria withdrawal is ordered
Shanahan was travelling away from Washington when Trump tweeted his decision, and a spokesman for Shanahan declined to comment. Mr Trump wrote: "I just had a long and productive call with President @RT_Erdogan of Turkey".
Libya has been torn apart by power struggles and undermined by chronic insecurity since the ouster and assassination of Muammar Gaddafi in 2011.
Two competing administrations, rival militias, tribes and jihadists have been competing for control of territory and the country's vast oil wealth.
IS took advantage of the chaos to gain a foothold in the coastal city of Sirte in 2015.
After eight months of fighting, forces loyal to the UN-backed Government of National Accord regained control of the city. He said the foreign ministry had been targeted because "it is a symbol of sovereignty". The two rival sides met again in May this year - weeks after ISIS suicide attackers killed 14 people at the country's electoral commission - and committed to holding parliamentary and presidential polls in December.
Libya Province, a local arm of Islamic State, claimed credit for the attack in a statement posted to its Amaq news agency and distributed by SITE Intelligence Group, which monitors global terrorist activity. The Tripoli government is recognized by the United Nations.