Until 1922, Ayvalık had a large Greek population. Anecdotal evidence indicates that, immediately after the defeat in the naval Battle of Chesma (Çeşme), the Ottoman admiral (later grand vizier) Cezayirli Gazi Hasan Pasha and men from the ships he could extirpate from the disaster, were lodged, on their frantic return to the capital, by a local priest in Ayvalık who did not know who they were. Hasan Pasha did not forget the kindness shown at that hour of crisis and when he became grand vizier, accorded virtual autonomy to the Greeks of Ayvalık, paving the way for its becoming an important cultural center for that community in the Ottoman Empire in the 19th century.

The town was invaded by the Greek Army on 29 May 1919, and taken back three years later by Turkish Armies on 15 September 1922. After the Turkish Independence War, the Greek population in the town was replaced by a Muslim population from Greece under the 1923 agreement for the Exchange of Greek and Turkish Populations. Most of the new population that replaced the former Greek community were Muslim Turks from Mytilene, Crete and Greek Macedonia. One could still hear Greek spoken in the streets till recently. Many of the town's mosques are Greek Orthodox churches that have been converted.

Today, Ayvalık and the numerous islets encircling the bay area are popular holiday resorts. The most important and the biggest of these islets is Cunda Island (Alibey Island) which was connected to the mainland by a bridge in the late 1960s.

Since September 1998, Ayvalık has had an international music academy (AIMA) which gives master classes for violin, viola and cello. It brings together students from all over the world and gives them a precious opportunity to work with distinguished masters of their branch.

Ayvalık also has two of the longest sandy beaches of the whole country which extend as far as the Dikili district of İzmir nearly 30 km in the south. These are Sarımsaklı and Altınova beaches.

In recent years Ayvalık has also become an important point of attraction for scuba divers with its underwater fauna.

Ayvalık and its environs is famous for the highly appreciated quality of olive oil production.