The Forgotten Capital, which is also known as the Forgotten City depending on which section of the Capital you are in located in, can be found on the northern continent. Reaching it involves going through Bone Village, the Sleeping Forest and Corel Valley.
There are two separate occasions during the story where your team will be required to travel to the Forgotten Capital. The first is after the events that take place at the Temple of the Ancients after Aeris leaves your party and leads you there. The second is after you have returned to Cosmo Canyon after capturing all of the Huge Materia.
You can click on either of the following links to check out the first visit to the Forgotten Capital or the second visit on Disc 2.
There are a number of items that you can pick up in this area including:
Magic Source
Aurora Armlets
Guard Source
Elixir
Enemy Skill Materia
Comet Materia
Viper Halberd
The map below shows the location of the Forgotten Capital on the World Map: